Multi GPU training - inconstant results between virtual GPUs and non Virtual GPUs

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

Modified from the stock example
https://www.tensorflow.org/beta/tutorials/distribute/keras
Working example is attached to this bug report

OS Platform and Distribution

Linux Ubuntu 16.04

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

No

TensorFlow installed from (source or binary):

Binary

TensorFlow version (use command below):

2.0.0-beta0

Python version:

Python 3.5.2

Bazel version (if compiling from source):

N/A (0.23.2)

GCC/Compiler version (if compiling from source):

N/A (5.4.0 20160609)

CUDA/cuDNN version:

10.0.130/7.4.1

GPU model and memory:

4x Titan Xp (12Gb) (Standalong, no SLI connections)

Describe the current behavior
The model accuracy is far off between virtual GPUs and physical GPUs. The default tf.distribute.MirroredStrategy() is used for multi GPU training.
Describe the expected behavior
The trained model should achieve similar level of accuracy regardless of virtual devices (0.9938) or physical devices (0.9698).
Code to reproduce the issue

The script attached is used to train a model using multiple GPUs.

The code is modified from the stock example for quick experiment.

https://www.tensorflow.org/beta/tutorials/distribute/keras

Traing using two VirtualDevices after 10 epoches

when only one GPU is detected, two virtual devices will be created. (init_gpu())
Use the following command to run the script

export CUDA_VISIBLE_DEVICES="0"
python3 mnist_muti_gpus.py

results after 10 epoches

Epoch 10/10
465/469 [============================>.] - ETA: 0s - loss: 0.0250 - accuracy: 0.9937
Learning rate for epoch 10 is 0.00001
469/469 [==============================] - 4s 9ms/step - loss: 0.0249 - accuracy: 0.9938

...

Epoch 20/20
463/469 [============================>.] - ETA: 0s - loss: 0.0240 - accuracy: 0.9939
Learning rate for epoch 20 is 0.00001
469/469 [==============================] - 4s 8ms/step - loss: 0.0239 - accuracy: 0.9940

Traing using two GPUs (non virtual) after 10 epoches

If more than one GPU is visiable, no virtual device will be created. (init_gpu())
Use the following command to run the script

export CUDA_VISIBLE_DEVICES="0,1"
python3 mnist_muti_gpus.py

results after 10 epoches

Epoch 10/10
464/469 [============================>.] - ETA: 0s - loss: 0.1421 - accuracy: 0.9697
Learning rate for epoch 10 is 0.00001
469/469 [==============================] - 4s 8ms/step - loss: 0.1421 - accuracy: 0.9698

...

Epoch 20/20
465/469 [============================>.] - ETA: 0s - loss: 0.1157 - accuracy: 0.9728
Learning rate for epoch 20 is 0.00001
469/469 [==============================] - 4s 9ms/step - loss: 0.1157 - accuracy: 0.9728

Other info / logs
"""
This script is extended from the following stock example:
https://www.tensorflow.org/beta/tutorials/distribute/keras
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals

from tensorflow.keras.callbacks import TensorBoard
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense, Flatten, BatchNormalization
from tensorflow.keras.layers import Conv2D, MaxPool2D
from tensorflow.keras import optimizers
from tensorflow.keras import losses
from tensorflow.keras import initializers
from tensorflow.keras import backend as K
from tensorflow.keras.layers import Activation

import tensorflow as tf
import tensorflow_datasets as tfds
import math
import os

# Initialize GPUs
# This function must be called first
# Setting up GPU memory ussage
# https://www.tensorflow.org/beta/guide/distribute_strategy
# https://www.tensorflow.org/beta/guide/using_gpu
def init_gpus(soft_device_placement=True, log_device_placement=False, create_virtual_devices=False, memory_limit=4096):

    tf.config.set_soft_device_placement(soft_device_placement)
    tf.debugging.set_log_device_placement(log_device_placement)

    gpus = tf.config.experimental.list_physical_devices('GPU')
    if gpus:
        # If there is only one GPU, create two logical virtual devices for developing
        # on a machine with only one GPU installed
        try:
            # Create 2 virtual GPUs on each physical GPU with the given memory_limit GPU memory
            if create_virtual_devices and len(gpus) == 1:
                for gpu in gpus:
                    tf.config.experimental.set_virtual_device_configuration(
                        gpu,
                        [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4096),
                         tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4096)]
                    )

            else:
                # Currently, memory growth needs to be the same across GPUs
                for gpu in gpus:
                    tf.config.experimental.set_memory_growth(gpu, True)

        except RuntimeError as e:
            # Memory growth must be set before GPUs have been initialized
            print(e)

        # print out physical and logical GPUs
        logical_gpus = tf.config.experimental.list_logical_devices('GPU')
        print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")

    else:
        print("No visible GPU is detected...")

def scale(image, label):
    image = tf.cast(image, tf.float32)
    image /= 255
    return image, label

# Callback for printing the LR at the end of each epoch.
class PrintLR(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
        print('\nLearning rate for epoch %d is %.5f' %(epoch + 1, self.model.optimizer.lr.numpy()))

# Function for decaying the learning rate.
# You can define any decay function you need.
def decay(epoch):
    if epoch < 3:
        return 1e-3
    elif epoch >= 3 and epoch < 7:
        return 1e-4
    else:
        return 1e-5

def run():
    ### get data
    init_gpus(
        log_device_placement=False,
        create_virtual_devices=True
    )

    #
    # strategy = tf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.ReductionToOneDevice())
    # strategy = tf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())
    # strategy = tf.distribute.MirroredStrategy(cross_device_ops=tf.distribute.NcclAllReduce())
    strategy = tf.distribute.MirroredStrategy()
    print('Number of devices: {}'.format(strategy.num_replicas_in_sync))

    # Setting with_info to True includes the metadata for the entire dataset,
    # which is being saved here to info. Among other things, this metadata object
    # includes the number of train and test examples.
    datasets, info = tfds.load(
        name='mnist',
        with_info=True,
        as_supervised=True
    )
    mnist_train, mnist_test = datasets['train'], datasets['test']

    # You can also do info.splits.total_num_examples to get the total
    # number of examples in the dataset.
    num_train_examples = info.splits['train'].num_examples
    num_test_examples = info.splits['test'].num_examples

    #
    print("num_train_examples = %d, num_test_examples = %d" %(num_train_examples, num_test_examples))

    BUFFER_SIZE = num_train_examples
    BATCH_SIZE_PER_REPLICA = 64
    BATCH_SIZE = BATCH_SIZE_PER_REPLICA * strategy.num_replicas_in_sync
    EPOCHS = 20

    # train_dataset = mnist_train.map(scale).shuffle(num_train_examples).repeat().batch(BATCH_SIZE)
    train_dataset = mnist_train.shuffle(num_train_examples).repeat().map(scale).batch(BATCH_SIZE)
    eval_dataset = mnist_test.map(scale).batch(BATCH_SIZE)

    with strategy.scope():
        ### compile
        ## Seperate the activation layer so that BatchNormalization can be added later on.
        model = tf.keras.Sequential([
            tf.keras.layers.Conv2D(32, 3, input_shape=(28, 28, 1)),
            tf.keras.layers.Activation(activation='relu'),
            tf.keras.layers.MaxPooling2D(),
            tf.keras.layers.Flatten(),
            tf.keras.layers.Dense(64),
            tf.keras.layers.Activation(activation='relu'),
            tf.keras.layers.Dense(10, activation='softmax')
        ])

        model.compile(
            loss='sparse_categorical_crossentropy',
            optimizer=tf.keras.optimizers.Adam(),
            metrics=['accuracy']
        )
        model.summary()

    # print out physical and logical GPUs
    gpus = tf.config.experimental.list_physical_devices('GPU')
    logical_gpus = tf.config.experimental.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")

    # Define the checkpoint directory to store the checkpoints
    checkpoint_dir = './training_checkpoints'
    # Name of the checkpoint files
    checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt_{epoch}")

    callbacks = [
        tf.keras.callbacks.TensorBoard(log_dir='./logs'),
        tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_prefix, save_weights_only=True),
        tf.keras.callbacks.LearningRateScheduler(decay),
        PrintLR()
    ]

    steps_per_epoch = math.ceil(num_train_examples / BATCH_SIZE)
    model.fit(
        train_dataset,
        steps_per_epoch=steps_per_epoch,
        epochs=EPOCHS,
        callbacks=callbacks
    )

    # validation
    eval_loss, eval_acc = model.evaluate(eval_dataset)

    K.clear_session()
    return

if __name__ == '__main__':
    run()

Gently ping @aaroey

@scotthong, I believed virtual gpu is an experimental feature, it has been tested in inference environment where there are no data dependency between the graph on different virtual GPUs. It has not been tested in training environment.

The problem is that why tf.distribute.MirroredStrategy() does not work on two physical GPUs.
1. the results are similar between one GPU and two Virtual GPUs.     2. the results are very different between one GPU and two physical GPUs.
On Wednesday, August 14, 2019, 03:55:46 PM EDT, Guangda Lai<EMAIL_ADDRESS>wrote:

@scotthong, I believed virtual gpu is an experimental feature, it has been tested in inference environment where there are no data dependency between the graph on different virtual GPUs. It has not been tested in training environment.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
|
|  |  |
|
|
|
|  |
Multi GPU training - inconstant results between virtual GPUs and non Vir...
System information Have I written custom code (as opposed to using a stock example script provided in TensorFlow...
|
|
|
|
|
|
|  |  |
|
|
|
|  |
Build software better, together
GitHub is where people build software. More than 40 million people use GitHub to discover, fork, and contribute ...
|
|
|

@scotthong,
I did not observe much difference in the accuracy on running the code with TensorFlow v2.4.
#with CUDA_VISIBLE_DEVICES="0"
Learning rate for epoch 10 is 0.00001
Epoch 11/20
469/469 [==============================] - 4s 9ms/step - loss: 0.0268 - accuracy: 0.9935
.
.
Learning rate for epoch 20 is 0.00001
2021-01-18 15:38:00.249082: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:454] The `assert_cardinality` transformation is currently not handled by the auto-shard rewrite and will be removed.
79/79 [==============================] - 2s 10ms/step - loss: 0.0434 - accuracy: 0.9848

#with CUDA_VISIBLE_DEVICES="0,1"
Learning rate for epoch 10 is 0.00001
Epoch 11/20
469/469 [==============================] - 4s 9ms/step - loss: 0.0286 - accuracy: 0.9927
.
.
Learning rate for epoch 20 is 0.00001
2021-01-18 15:40:42.211088: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:454] The `assert_cardinality` transformation is currently not handled by the auto-shard rewrite and will be removed.
79/79 [==============================] - 2s 10ms/step - loss: 0.0468 - accuracy: 0.9853

Could you please update TensorFlow to the latest stable version and check if you are still facing the same issue. Thanks!

@scotthong,
I did not observe much difference in the accuracy on running the code with TensorFlow v2.4.
#with CUDA_VISIBLE_DEVICES="0"
Learning rate for epoch 10 is 0.00001
Epoch 11/20
469/469 [==============================] - 4s 9ms/step - loss: 0.0268 - accuracy: 0.9935
.
.
Learning rate for epoch 20 is 0.00001
2021-01-18 15:38:00.249082: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:454] The `assert_cardinality` transformation is currently not handled by the auto-shard rewrite and will be removed.
79/79 [==============================] - 2s 10ms/step - loss: 0.0434 - accuracy: 0.9848

#with CUDA_VISIBLE_DEVICES="0,1"
Learning rate for epoch 10 is 0.00001
Epoch 11/20
469/469 [==============================] - 4s 9ms/step - loss: 0.0286 - accuracy: 0.9927
.
.
Learning rate for epoch 20 is 0.00001
2021-01-18 15:40:42.211088: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:454] The `assert_cardinality` transformation is currently not handled by the auto-shard rewrite and will be removed.
79/79 [==============================] - 2s 10ms/step - loss: 0.0468 - accuracy: 0.9853

Could you please update TensorFlow to the latest stable version and check if you are still facing the same issue. Thanks!


Battery level thinks 50% is 100%

Hello, after a recent update, my waybar status bar was working just fine.
I don't know what version the problem began but it was around mid november
Basically waybar thinks 50% of my battery equals to 100%.
Output of acpi
$acpi
Battery 0: Charging, 92%, 00:19:33 until charged
Battery 1: Unknown, 0%, rate information unavailable

Waybar right now:

Edit: wrong images
Edit2: note that the module on the far right shows 46%.

Fixed with latest commit on master


Add function to perform institution or organization search

Hi,
I have added a function to perform institution search.
Example
"""
.. testcode::
            search_query = scholarly.search_org('ucla')
            print(search_query)

    :Output::

        .. testoutput::

            [{'Organization': 'University of California, Los Angeles',
              'id': '14108176128635076915'},
             {'Organization': 'Universidad Centroccidental Lisandro Alvarado',
              'id': '9670678584336165373'}
            ]

"""

Could you please revise so that it works with the v1.0? This PR seems to have arrived while v1.0 was in development.

i have updated the code to work with v1.0. Let me know if the changes look good enough

Added in v1.0.5

Added in v1.0.5


Add flask-admin and auth for flask-admin

Coverage decreased (-8.07%) to 73.252% when pulling fe04330419d1c15f72d62bc31ee752dccdc128b6 on BSRCrackedThis:master into 3653d851687c876f77329e7e2cb994ddb77afb60 on Stark-Mountain:master.


Better documentation

This is something I've been meaning to get around to. The library itself is pretty handy, but having inadequate documentation will limit its reach. I'd love to get a little mini-site setup with better usage/examples.

i have a problem with Arabic names, when i try to convert it it gives me an error
[function.getimagesize]: failed to open stream
Invalid image: images/4edd5_?????_?????????.jpg
i think i need to add "UTF", but i don't where to add it to fix this error
thank you for this great library

Sounds like a PHP issue. This may help. Hope it works for you!

Hi claviska.
What about special characters as text "inside" the generated image (not the filename)? I have used both utf8_decode/utf8_encode but no success so far.

Make sure the font supports the characters you're trying to use. GD doesn't seem to fallback to other fonts (like the OS) when a glyph is missing.

So true claviska, I forgot to state my solution: like advised, by replacing the font with one that supports special characters, the problem was solved.
Thanks a lot!

Better docs are included in the readme in 3.0.


whats edit?

login id and passward what?
i'm don't engilsh sorry

what edit map data?

what map change google?

The login modal is just a placeholder, it doesn't currently do anything.


Picolibc 1.8.4

Merge in picolibc version 1.8.4.
There is a matching SDK PR pending, https://github.com/zephyrproject-rtos/sdk-ng/pull/705
This offers a new printf variant which cuts out a bunch of format modifier support while allowing us to add long long support to the 'regular' variant while still offering something smaller.
hello-world on cortex-m3 FLASH sizes:
float		13646
float inexact	10934
long long	 8434
integer		 8374
minimal		 7814

For comparison, here are the minimal C lib sizes:
float		11094
long long	 9206
integer		 9170
nano		 8130

With this change, I'm suggesting switching the default configuration to select integer mode with long long support. The SDK PR does that as applications still have the option of selecting 'minimal' mode to save space (the long long option is only available at compile time). Of course, applications may still select among all of the build-time options when using the picolibc module.
Avoiding use of divide in thread initialization (#62900) and disabling malloc reduces the minimal size to 6882 bytes of flash and the full integer version with long long support at 7946 bytes.

This should address issue #62444

I can't remember what the smallest flash/RAM device supported in zephyr is but think it's an stm32 chip with something like 2K RAM and 4K flash, maybe @erwango knows? Can we ensure that builds with this library successfully for at least the minimal zephyr application (using picolibc) and, ideally, hello_world using picolibc too with the default app configuration?

I can't remember what the smallest flash/RAM device supported in zephyr is but think it's an stm32 chip with something like 2K RAM and 4K flash, maybe @erwango knows?

I don't know if that's Zephyr smallest, but current STM32 Zephyr smallest is nucleo_l011k4: 2K RAM, 16K Flash

I don't know if that's Zephyr smallest, but current STM32 Zephyr smallest is nucleo_l011k4: 2K RAM, 16K Flash

$ west build -b nucleo_l011k4 samples/hello_world/
Memory region         Used Size  Region Size  %age Used
           FLASH:       12886 B        16 KB     78.65%
             RAM:        1356 B         2 KB     66.21%
        IDT_LIST:          0 GB         2 KB      0.00%
$ west build -b nucleo_l011k4 samples/hello_world/ -d build-minimal -- -DCONFIG_MINIMAL_LIBC=y
Memory region         Used Size  Region Size  %age Used
           FLASH:       12962 B        16 KB     79.11%
             RAM:        1328 B         2 KB     64.84%
        IDT_LIST:          0 GB         2 KB      0.00%
$ west build -b nucleo_l011k4 samples/hello_world/ -- -DCONFIG_COMMON_LIBC_MALLOC=n -DCONFIG_PICOLIBC_IO_MINIMAL=y
Memory region         Used Size  Region Size  %age Used
           FLASH:       11842 B        16 KB     72.28%
             RAM:        1344 B         2 KB     65.62%
        IDT_LIST:          0 GB         2 KB      0.00%

In the default configuration, it's slightly smaller than the minimal C library version. If you turn off the (unused) malloc code and switch to the smaller printf code, it's even smaller.

This offers a new printf variant which cuts out a bunch of format modifier support while allowing us to add long long support to the 'regular' variant while still offering something smaller.

@keith-packard Could you please try out samples/subsys/logging/dictionary code sample?
I've noticed that it seems to be broken with picolibc (but also newlib, IIRC) and I wonder if this PR would by any chance make the situation better and/or be an opportunity to address other outstanding issues around format modifiers. Thank you!

@keith-packard Could you please try out samples/subsys/logging/dictionary code sample? I've noticed that it seems to be broken with picolibc (but also newlib, IIRC) and I wonder if this PR would by any chance make the situation better and/or be an opportunity to address other outstanding issues around format modifiers. Thank you!

The python log parser code was busted and didn't handle signed char/unsigned char length modifiers for integer specifiers. #63007

As you may have noticed, this PR has changed from pulling in picolibc 1.8.4 to picolibc 1.8.5 -- switching 1.8.4 to enable long long for integer mode increased the size of many applications. 1.8.5 has a new 'long long' variant which allows applications to easily select between those two modes to save space. With this, the SDK will also support separate integer and long long variants.

For this to get merged in 3.5.0, we'll need a new SDK release including picolibc 1.8.5 as well. If that isn't going to happen, then this needs to wait.

For this to get merged in 3.5.0, we'll need a new SDK release including picolibc 1.8.5 as well. If that isn't going to happen, then this needs to wait.

I had a short chat with Anas about this. It's inevitably post 3.5 stuff.

For this to get merged in 3.5.0, we'll need a new SDK release including picolibc 1.8.5 as well. If that isn't going to happen, then this needs to wait.

I had a short chat with Anas about this. It's inevitably post 3.5 stuff.

That's what I expected. No good reason to make this large a change with only a few days to go.

I'm seeing the same failure on main. This test is built with the external C library by default, so I'm not sure what updating picolibc would do anyways...

With SDK 0.16.4 released, this should be ready to merge. I've rebased to re-test against current bits.


how to modify the XML

im getting this message when i import a file onto my final cut pro ,i dont know how to fix it, please see attachement

This means the Motion Template is not installed.
See:
https://brawtoolbox.io/troubleshooting/#the-effect-id-is-invalid
Did you install everything as per these instructions?
https://brawtoolbox.io/installation/

i will follow the links provided , hopefully it works

problem solved...thank you


the PSNR-oriented model with high PSNR performance

Which model does the RRDB_PSNR_x4.pth: the PSNR-oriented model with high PSNR performance. correspond to in the paper?

RRDB_PSNR_x4.pth
https://drive.google.com/open?id=1pJ_T-V1dpb1ewoEra1TGSWl5e6H7M4NN

Hi, Thanks for your reply!  I was able to find this model. What I asked was that: is this model introduced in the paper?
I was wondering if this model is PNSR oriented one used in network interpretation. Just want to make sure.
Also, is the loss function the only difference between RRDB_PSNR_x4.pth and RRDB_ESRGAN_x4.pth?
Appreciate your help and contribution!

as the pre-trained model for ESRGAN. 2) for network interpolation.

Yes, they are trained by different losses.

RRDB_PSNR_x4.pth
https://drive.google.com/open?id=1pJ_T-V1dpb1ewoEra1TGSWl5e6H7M4NN

Hi, why not just put the pretrained models into the dir?


dc.leafletMarkerChart - capture events on markers

My transition from dc.js + leaflet.js to dc.leafletMarkerChart puzzles me on
how to handle events on the markers that are created.

d3.selectAll('.leaflet-marker-icon').on('click', function() {
        console.log("click");
}

does not work.
Is there a function to do that and be able to get id of the markers as well ?
The marker() function seems promissing but unusable (get message: marker is not a function) while doc says:

.marker() - set function (d,map) to build the marker object. Default: standard Leaflet marker is built

You cannot do any normal d3 functionality on leaflet or google maps. Doing something on click sounds like a good feature for the map, so I might add that as another option.
The .marker() function can be used to overwrite what is called when creating the marker. So you could use it to create custom markers.
You could try .markerGroup() to get the layer and then the markers should be inside of that.
If you want greater control then a custom map might be a better fit for you. You have to handle creating the markers and crossfiltering.

Ok I have figure out this problem. No need for extra functions. Use of the marker() is very convenient once understanding its principles.
 Note that I have use L.Marker.extend to host my own id of marker.
Hope that will help (this code is visible from https://github.com/PBrockmann/ClimateProxiesFinder)

  customMarker = L.Marker.extend({
    options: {
      Id: 'Custom data!'
   }
  });

  iconSize = [32,32];
  iconAnchor = [16,32];
  popupAnchor = [0,-20];

  mapChart  = dc.leafletMarkerChart("#chart-map");

  mapChart
      .width(1000)
      .height(300)
      .dimension(mapDim)
      .group(mapGroup)
      .center([45, -19])    // slightly different than zoomHome to have a info updated when triggered
      .zoom(2)
      .tiles(function(map) {			// overwrite default baselayer
	   return L.tileLayer(
                'http://services.arcgisonline.com/ArcGIS/rest/services/Ocean_Basemap/MapServer/tile/{z}/{y}/{x}',
                { attribution: 'LSCE © 2016 | Baselayer © ArcGis' }).addTo(map);
      })
      .mapOptions({maxZoom: mapMaxZoom, zoomControl: false})
      .fitOnRender(false)
      .filterByArea(true)
      .cluster(true)
      .clusterOptions({maxClusterRadius: 50, showCoverageOnHover: false, spiderfyOnMaxZoom: true})
      .title(function() {})
      .popup(function(d,marker) {
		id = d.key[2] -1;
    		return  "Id: " + "" + data[id].Id + ""
    			+ "Position: " + "" + data[id].Longitude.toFixed(2) + "°E, " + data[id].Latitude.toFixed(2) + "°N"
    			+ "Depth (m): " + "" +  data[id].Depth.toFixed(2) + ""
    			+ "Date (ka): " + "" + "from " + data[id].RecentDate.toFixed(2) + " to " + data[id].OldestDate.toFixed(2) + ""
    			+ "Archive: " + "" + data[id].Archive + ""
    			+ "Material: " + "" + data[id].Material + "";
      })
      .popupOnHover(true)
      .marker(function(d,map) {
        	id = d.key[2] -1;
		if (data[id].Archive == "Ice")
			icon = L.icon({ iconSize: iconSize, iconAnchor: iconAnchor, popupAnchor: popupAnchor, iconUrl: 'marker_Ice.png' });
		else if (data[id].Archive == "Lake")
			icon = L.icon({ iconSize: iconSize, iconAnchor: iconAnchor, popupAnchor: popupAnchor, iconUrl: 'marker_Lake.png' });
		else if (data[id].Archive == "Ocean")
			icon = L.icon({ iconSize: iconSize, iconAnchor: iconAnchor, popupAnchor: popupAnchor, iconUrl: 'marker_Ocean.png' });
		else if (data[id].Archive == "Speleothem")
			icon = L.icon({ iconSize: iconSize, iconAnchor: iconAnchor, popupAnchor: popupAnchor, iconUrl: 'marker_Speleothem.png' });
		else if (data[id].Archive == "Tree")
			icon = L.icon({ iconSize: iconSize, iconAnchor: iconAnchor, popupAnchor: popupAnchor, iconUrl: 'marker_Tree.png' });
        	marker = new customMarker([data[id].Latitude, data[id].Longitude], {Id: (id+1).toString(), icon: icon});
                marker.on('mouseover', function(e) {
			iconUrlNew = "highlight_" + e.target.options.icon.options.iconUrl;
			iconNew = L.icon({ iconSize: iconSize, iconAnchor: iconAnchor, popupAnchor: popupAnchor, iconUrl: iconUrlNew });
			e.target.setIcon(iconNew);
			d3.selectAll(".dc-table-column._0")
				.text(function (d, i) {
			     		if (parseInt(d.Id) == e.target.options.Id) {
						this.parentNode.scrollIntoView();
			                 	d3.select(this.parentNode).style("font-weight", "bold");
			               	}
			     		return d.Id;
		        	});
		});
                marker.on('mouseout', function(e) {
			iconUrlNew = e.target.options.icon.options.iconUrl.replace("highlight_", "");
			iconNew = L.icon({ iconSize: iconSize, iconAnchor: iconAnchor, popupAnchor: popupAnchor, iconUrl: iconUrlNew });
			e.target.setIcon(iconNew);
			d3.selectAll(".dc-table-column._0")
				.text(function (d, i) {
			     		if (parseInt(d.Id) == e.target.options.Id) {
			                 	d3.select(this.parentNode).style("font-weight", "normal");
			               	}
			     		return d.Id;
		        	});
		});
                marker.on('click', function(e) {
			console.log("double click");
      			//window.open(...
		});
        	return marker;
      });

That is an awesome website! :+1:


Odinson ide

This branch adds the default parsers (Odin, Brat) inside the TAG class. This helps when trying to import the parsers from outside the project, by using import or require which results in a parse error since the classes are not derequired by browserify in the build process. By setting them as default parsers, they automatically get derequired in the normal build process.

This PR was supposed to go into a forked version a repository. Closing.


Sign BTC Transaction

There needs to be a unit test for signing BTC transactions
Please see this issue:
https://github.com/MelbourneDeveloper/KeepKey.Net/issues/6

The latest unit tests have an example


add kubernetes client auth plugins

Adds support for additional auth providers

@bacongobbler should be good now

thanks!


Remove not necessary argument

The 'enhancer' argument is not necessary,beacuse there is:
return enhancer(createStore)(reducer, preloadedState) at ' createStore.js'
We can see that the function call doesn't pass enhancer as the arguments.
So this is my thought.

Thanks for the PR, but this is a duplicate of #2128 .  Please see my comment at https://github.com/reactjs/redux/pull/2128#issuecomment-263581319 for more info.


Newlines in comments

Fatebook seems to strip newlines in comments. If easy to fix, I'd find newline support helpful! This would be especially helpful for cases where I'm trying to create lists, or want to write multiple paragraphs to share some context.

Thanks for reporting Angelina - fixed!


Create 0000-01-02-cmoreno323.md

Add username's file

Open pull request

Open pull request

Open pull request

Open pull request

Open pull request

Open pull request


Use unidiff

https://pypi.org/project/unidiff/ is a viable replacement for the binary dependency on patch/diffutils, which I am familiar with using via @coala.
If there are bits missing, I am sure we can find help adding the missing functionality.

Nice one. Wasn't aware of a diff implementation in python. Not sure if am able to add it in the next weeks but patches are always welcome.


Example that uses buttons needed

Can you add an example that uses the buttons? For example, the buttons could be used to refresh the data on the screen.

Closing this issue because I am looking into the code and can see that Button 1 does refresh the data. This should be explained somewhere!


Re-design of CLI interface

Re-design of CLI interface
==========================

why?

* too many option stacked below the primary command
* certain flags and options are irrelevant for certain actions
* improve clarity

redesign based on current:
--------------------------

cargo msrv
  --output human xor json
  --no-logging

cargo msrv verify
  Tests a crate against a given msrv
.... file: Cargo.toml key: metadata.msrv
.... env: CARGO_MSRV_VERIFY
.... arg: --msrv {}

  --on-error=(
    exit,
    msrv-run,
    cargo-check,
  )

--strategy (ast, toolchain, (db))

cargo msrv run (?)
  Determine the MSRV for the given crate
  --minimum or CARGO_MSRV_MINIMUM
  --maximum or ...
  --source SOURCE or ...
  --no-edit-cargo-toml or ...
  --beta xor --nightly
  --strategy (ast, toolchain)

cargo msrv install-toolchain
   (Pre) install toolchains for current platform if necessary

longer term:
------------

cargo msrv submit
   Submit (vote) working versions to database

cargo msrv lookup
  Attempts to look up the msrv of dependencies
  Uses Cargo.toml metadata.msrv or/and database
  As obtained sources by cargo

exported from notes

are you also interested in using Clap's declarative syntax for defining the CLI?

are you also interested in using Clap's declarative syntax for defining the CLI?

I don't think that's directly related to the discussion above; so we should make a new issue for this 😄.


fix importing extensions and globalstate

fyi @chrisdias this fixes the issue we saw yesterday with Profiles not applying extensions.


Update CIP-6 to accurately reflect the current state of implementation

Description
CIP-6 was implemented and merged view pr: https://github.com/celestiaorg/celestia-app/pull/3094/files#diff-008f85e59c1e0f88e9ad4be82fd1a15f29261f9b9d1a71a19488c58b94fef363R200
cip-6.md reference implementation should be updated to reflect the changes.

yes the reference implementation section. I'll update the issue to reflect that. @rootulp

Are we also able to move it into Final? I'll take one last pass to make sure everything is covered.

FYI @ninabarbakadze https://github.com/celestiaorg/CIPs/pull/121 merged but didn't close this issue. I think because the hyphen in the PR description.


Add command history

This PR enables common cmdline functionality of navigating past commands using arrow keys.

Thanks so much for doing this!


✨ Avatar image in portrait mode

Add an option to have an avatar image in portrait mode.
 Provide a list of possible shapes, including a shape for portrait mode. Surely interesting for business photos.


getStoredState returns transit format string

Is it possible to have getStoredState return an restored immutable version of the stored state?
At the moment it's just an unwrapped version of the original getStoredState so it returns the raw transit format string.

yes this is a bug, thanks for finding! just released v4.1.0 with fix


$refs do not work with URN #id values

Using this schema:
{
    "$id": "urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d",
    "allOf": [
        { "$ref": "#/definitions/schema" }
    ],
    "definitions": {
        "schema": {}
    }
}

And this JSON document:
{}

With the example program from the README results in the following output:
panic: Reference urn:#/definitions/schema must be canonical

goroutine 1 [running]:
main.main()
        /pdhome/dstokes/dev/f5-appsvcs/goschema.go:15 +0x329
exit status 2

If the $id value is changed from a URN to a URL, then everything works as expected. The JSON Schema specification section 8.2 says $id must be a URI. According to RFC3986, a URI can be a URN or a URL.

I dug into this a little today and it seems Go's implementation of RFC3986 isn't what it should be. I should be able to work around it, but worst case I'll have to reimplement a RFC3986 URI resolution algorithm.

I'm not sure if that's the same thing, but gojsonschema seems to ignore object paths and loads the root object instead.
If you have a reference that looks like this:
{
    "$ref":"somefile.json#/path/to/object"
}

gojsonschema ignores the path to the object resolves such reference as:
{
"path": {
  "to": {
     "object": {
         "foo":"bar"
   }
 }
}

Instead of:
{
  "foo": "bar"
}

Is it related to this mechanism or is it a separate bug?


[Snyk] Upgrade @patternfly/react-core from 5.4.0 to 5.4.8

Snyk has created this PR to upgrade @patternfly/react-core from 5.4.0 to 5.4.8.
:information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project.

The recommended version is 8 versions ahead of your current version.

The recommended version was released on 21 days ago.

Release notes

Package name: @patternfly/react-core

5.4.8 - 2024-10-17

5.4.7 - 2024-10-17

5.4.6 - 2024-10-17

5.4.5 - 2024-10-09

5.4.4 - 2024-10-08

5.4.3 - 2024-10-07

5.4.2 - 2024-10-07

5.4.1 - 2024-09-18

5.4.0 - 2024-08-27

    from @patternfly/react-core GitHub release notes

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.
For more information:

🧐 View latest project report
📜 Customise PR templates
🛠 Adjust upgrade PR settings
🔕 Ignore this dependency or unsubscribe from future upgrade PRs

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 8.27%. Comparing base (292096c) to head (424b23a).

Additional details and impacted files
@@          Coverage Diff           @@
##           master    #839   +/-   ##
======================================
  Coverage    8.27%   8.27%
======================================
  Files          68      68
  Lines        3953    3953
  Branches      864     864
======================================
  Hits          327     327
  Misses       3626    3626

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Mark all button on "Overwrite" objects pop-up

Follow up on #934

Yet another approach would be to use a modal dialogbox container popup. Then you can use the add_function method of cl_salv_table. Something like this.

https://gitlab.com/christianguenter/ABAP_Demo_ALV_Popup/blob/1a1c5207158693d7260b9fd338224420b29eadd5/zcl_test_alv_popup.clas.abap

Has anyone already started working on this? Reason I am asking is that fixing #949 will probably touch the same code and might cause conflict.

#949 is a bug, this one is just nice to have

#954

keeping open until PR is merged


grpc_completion_queue_create_for_pluck(NULL) fail

What version of gRPC and what language are you using?
gRPC-v1.10.0
What operating system (Linux, Windows, …) and version?
2.6.32-042stab127.2
What runtime / compiler are you using (e.g. python version or version of gcc)
gcc
What did you do?
If possible, provide a recipe for reproducing the error. Try being specific and include code snippets if helpful.
What did you expect to see?
What did you see instead?
Segment fault.
This issue is split from https://github.com/grpc/grpc/issues/14601. At the beginning of the program,
grpc_init();
grpc_completion_queue_create_for_pluck(NULL);

will segment fault randomly.
Not all environment can reproduce it. Since the it happens randomly, I guess it's caused by random initial value for some variables.
This bug is introduced after gRPC-v1.7.0.

The way to access the environment is in the comment at the bottom of https://github.com/grpc/grpc/issues/14601

I just verified on that system that it has a faulty sched_getcpu syscall. It ends up returning a value more than the maximum number of cpus which leads to accessing unintended memory. This seems like a linux kernel issue.

gRPC 1.6.0 works fine in such system while 1.7.0 start segment fault randomly. They both has this syscall, is the usage of this number changes starting from 1.7.0?

gRPC 1.6.0 was probably not using sched_getcpu as much as in 1.7.0.

It definitely looks like a case of CPU hotplugging and sysconf(_SC_NPROCESSORS_CONF) not returning the maximum number of possible CPUs.
Created PR #14712 to fix this issue in gRPC. Manually verified that the fix works on the system. Tracking this separately in #14711. Please use gRPC 1.6 till the pull request gets merged.

Thanks @yashykt . This PR solve the grpc_completion_queue_create_for_pluck segment fault in php extension.
Is it possibel that grpc_completion_queue_shutdown(completion_queue); is having the same issue? Because I noticed that the segment fault still there in php extension unless comment this function too. As well as grpc_completion_queue_destroy
Hi @svycka , it is strange that the syscall returns a wrong value. Maybe it will cause other errors not only in gRPC, but also somewhere else. Maybe you can confirm it with your hosting provide to see if they can fix it?

@ZhouyihaiDing The PR makes it returns a default value of 0 for when the system has cpu hot-plugging, and so the same issue should not be occurring in other parts. As far as the hosting goes, I'm not sure what they can do to solve this issue. It's more of providing a way to determine the maximum number of hot-plugged CPUs at grpc initialization.

Maybe grpc_completion_queue_shutdown segment fault is caused by other syscall? Can you please help to test this 3 lines together when you have time? Thank you!
grpc_completion_queue_create_for_pluck
grpc_completion_queue_shutdown
grpc_completion_queue_destroy


Support sending publish options

I have an extension which is published to testers, but not to the general public yet. I'd still like to update it automatically through the Chrome Web Store and continuous integration.
When I perform the publish request through the CLI client I get the following 400 error:
{
  "error": {
    "errors": [
      {
        "domain": "global",
        "reason": "badRequest",
        "message": "Publish condition not met: This item is current published to trusted testers. To publish it to everyone, you must unpublish it first. To keep publishing it to trusted testers, set publish_to_trusted_testers to true in the request."
      }
    ],
    "code": 400,
    "message": "Publish condition not met: This item is current published to trusted testers. To publish it to everyone, you must unpublish it first. To keep publishing it to trusted testers, set publish_to_trusted_testers to true in the request."
  }
}

Would you consider support of passing options to the body of the publish request so this case could be covered? Thanks!

Oops! I went ahead to implement this and when I took a closer look at the tests I've found that the target argument is actually something which supports that. I hope someone finds my issue useful someday 😅

Can you let me know if the argument actually works? In the cli version someone was having trouble with it.

@bfred-it I've created https://github.com/DrewML/chrome-webstore-upload-cli/pull/29 because I couldn't find a way to use it through the CLI. Please give me a hint if there's something I've missed! Thank you! 🙏


Scenario Outline inserts are not parsed in Data Tables

Given the following scenario:
    Scenario Outline: Look up a known username: <Username>
        Given a user exists with details:
            | Username | <Username> |
        When I look up the username '<Username>'
        Then the username does exist

        Examples:
            | Username |
            | known    |

The insert <Username> from the Examples table works correctly in the Scenario name and in the "When" step, but not in the "Given" step. Instead, when parsing the table, I get:
[["Username", "<Username>"]]

Where I would have expected:
[["Username", "known"]]

This should be fixed in a couple of weeks.

I made a bug report which likely is related. I need this ASAP. Is there anything I can do to accelerate this, @bbqsrc ?

See https://github.com/bbqsrc/cucumber-rust/issues/64 for the current state of this.

@bbqsrc this feature is supported in cucumber crate and described in Features book chapter. I think it's should be handled by consumers of this crate like cucumber, rather than expanding right inside parser.
In case you agree, I think this issue can be closed.

I agree with @ilslv about the parser itself doesn't make any substitutions by himself, but rather gives the AST of exactly what has been declared. For cucumber this issue has been resolved.
So I close this for now. Feel free to reopen, if needed.


docs: Add "How to contribute" section in README.md

what's wrong with the documentation?
How to contribute? section is missing from docs.
Add screenshots

Record

[X] I have checked the existing issues
[X] I want to work on this issue

Can I work on this issue since I did before

Can I work on this issue?

Hello

shall i start working on this issue


Explore enforcing conventional commits

In an onsite meeting on "Unification for infra", it was suggested that celestia-app explore ways to enforce conventional commits.

if this is only about the PR title / the commits on main, then this will work: https://github.com/marketplace/actions/conventional-commits-pr
There is probably also one for per commit in a PR.

@MSevey suggested https://github.com/amannn/action-semantic-pull-request


Manually created geoarrow table support in ScatterplotLayer

I was able to load 20 million polygons in lonboard. It was amazing! Now I am trying to figure out how to load 60 million points without having to use GeoPandas but I keep hitting code paths that expect either an interleaved list or paths that go back to numpy or paths that expect a byte like object from C.
Here is roughly what I am trying:
import lonboard
import gzip
import geoarrow.pyarrow as ga
import pyarrow.csv as pv

with gzip.open("/Users/x/data/points_s2_level_4_gzip/397_buildings.csv.gz") as fp:
        table = pv.read_csv(fp)

points = ga.point().from_geobuffers(None, table["latitude"], y=table["longitude"])

geoarrow_schema = pa.schema([pa.field("geometry", points.type, metadata={b"ARROW:extension:name": b"geoarrow.point"})])

point_table = pa.Table.from_arrays([points], schema=geoarrow_schema)
point_table.schema.field("geometry").metadata.get(b"ARROW:extension:name")
map_ = lonboard.ScatterplotLayer(table=point_table)

lonboard is designed to support geoarrow data directly, though so far it's mostly been tested on geopandas input. Can you say what the error is? Everything that expects a numpy array also allows a pyarrow array https://developmentseed.org/lonboard/api/traits/#lonboard.traits.ColorAccessor

I tried a few things and lost that stack trace. Logged the first issue with just going with arrow tables directly upstream:
https://github.com/geoarrow/geoarrow-python/issues/33

import lonboard
import gzip
import geoarrow.pyarrow as ga
import pyarrow.csv as pv
import pyarrow as pa

with gzip.open("/Users/x/data/points_s2_level_4_gzip/397_buildings.csv.gz") as fp:
        table = pv.read_csv(fp)
points = ga.point().from_geobuffers(None, table["latitude"].combine_chunks().to_numpy(), y=table["longitude"].combine_chunks().to_numpy())
geoarrow_schema = pa.schema([pa.field("geometry", points.type, metadata={b"ARROW:extension:name": b"geoarrow.point"})])
point_table = pa.Table.from_arrays([points], schema=geoarrow_schema)

map_ = lonboard.ScatterplotLayer(table=point_table)
map_

File ~/tmp/lib/python3.11/site-packages/lonboard/viewport.py:27, in get_bbox_center(table)
     24 geom_field = table.schema.field(geom_col_idx)
     25 geom_col = table.column(geom_col_idx)
---> 27 bbox = total_bounds(geom_field, geom_col)
     28 centroid = weighted_centroid(geom_field, geom_col)
     30 return bbox, centroid

File ~/tmp/lib/python3.11/site-packages/lonboard/geoarrow/ops/bbox.py:42, in total_bounds(field, column)
     39 extension_type_name = field.metadata[b"ARROW:extension:name"]
     41 if extension_type_name == EXTENSION_NAME.POINT:
---> 42     return _total_bounds_nest_0(column)
     44 if extension_type_name in [EXTENSION_NAME.LINESTRING, EXTENSION_NAME.MULTIPOINT]:
     45     return _total_bounds_nest_1(column)

File ~/tmp/lib/python3.11/site-packages/lonboard/geoarrow/ops/bbox.py:67, in _total_bounds_nest_0(column)
     65 for chunk in column.chunks:
     66     coords = chunk
---> 67     bbox.update(_coords_bbox(coords))
     69 return bbox

File ~/tmp/lib/python3.11/site-packages/lonboard/geoarrow/ops/bbox.py:57, in _coords_bbox(arr)
     56 def _coords_bbox(arr: pa.FixedSizeListArray) -> Bbox:
---> 57     np_arr = arr.flatten().to_numpy().reshape(-1, arr.type.list_size)
     58     min_vals = np.min(np_arr, axis=0)
     59     max_vals = np.max(np_arr, axis=0)

AttributeError: 'PointArray' object has no attribute 'flatten'

Oh well it's integrated with pyarrow, it's not integrated with whatever new types geoarrow-python is making. If you have a chunked array with the storage type of PointArray (i.e. a FixedSizeListArray) it'll work

Thanks for that comment, I grepped for FixedSizeListArray on this repo and found how to implement your recommendation.
Closing the issue and leaving the code below for others:
import gzip
import pyarrow.csv as pv
import pyarrow as pa
import lonboard

with gzip.open("/Users/x/data/points_s2_level_4_gzip/397_buildings.csv.gz") as fp:
        table = pv.read_csv(fp)
coords = np.array([table["latitude"].to_numpy(), table["longitude"].to_numpy()]).T.flatten()
parr = pa.FixedSizeListArray.from_arrays(coords, len(coords))
field = pa.field(
            "geometry",
            parr.type,
            nullable=True,
            metadata={"ARROW:extension:name": "geoarrow.point"},
    )
schema = pa.schema([field])
table = pa.Table.from_arrays([parr], schema=schema)

map_ = lonboard.ScatterplotLayer(table=table)

The ScatterplotLayer is created successfully!

Did it work with 60 million points?

Nope. Got a javascript error, only 20 gb of ram was used (from 128GB), some 100mb of swap was created and javascript errored out. I tried with 20M of those points and it broke too.
My feeling is that the polygon workflow I am using is better vectorized than the point workflow I am following so more tests are due.
This is all so new, so I decided to try the largest and report that instead of reporting what does not work.
New frontiers, lonboard ride down the mountain.


Implement continuous profiling

Description
As an Besu user, I want to have continuous profiling so that we can get profiling data when needed. This can help to troubleshoot some performance issue especially those related to CPU usage.
Expected behavior: [What you expect to happen]
Besu should run in the backhground a profiler (without restarting besu)
Actual behavior: [What actually happens]
There is no continuous profiling, we can still do profiling but we need to restart Besu.

Done with #7006


Saving of low-bit models for later loading?

Dear Ipex Team,
I was wondering if there was a way of saving a model that has been optimised and quantised in its new state for future loading for HF/Pytorch models.
I noticed there was a method in ipex_llm.optimize but this seems to be for a deprecated model loading method as it mentions a missing bigdl-llm config file.
The reason for such a feature is slow loading when having to optimise and quantise upon loading each time, but this is exacerbated when loading a large model that ends up having to utilise paging space due to over-usage of RAM during this process, if this was one-off then it wouldn't be so bad, but for the moment this is the case. It would also enable the sharing of these IPEX-optimised models for other users of Intel-ARC GPUs.
Many thanks.

See the Save and load sub-section under the Code Examples section

My apologies. Thank you!

I seem to be having issues with the save function for the pytorch model
2024-04-10 14:33:21,912 - INFO - intel_extension_for_pytorch auto imported
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [18:09<00:00, 363.25s/it]
2024-04-10 14:58:28,819 - INFO - Converting the current model to sym_int4 format......
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Traceback (most recent call last):
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\torch\serialization.py", line 619, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol, _disable_byteorder_record)
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\torch\serialization.py", line 853, in _save
    zip_file.write_record(name, storage.data_ptr(), num_bytes)
RuntimeError: [enforce fail at inline_container.cc:593] . PytorchStreamWriter failed writing file data/0: file write failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\ellio\miniconda3\envs\llm\Scripts\uvicorn.exe\__main__.py", line 7, in <module>
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\uvicorn\main.py", line 409, in main
    run(
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\uvicorn\main.py", line 575, in run
    server.run()
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\uvicorn\server.py", line 65, in run
    return asyncio.run(self.serve(sockets=sockets))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\asyncio\runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\asyncio\base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\uvicorn\server.py", line 69, in serve
    await self._serve(sockets)
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\uvicorn\server.py", line 76, in _serve
    config.load()
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\uvicorn\config.py", line 433, in load
    self.loaded_app = import_from_string(self.app)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\uvicorn\importer.py", line 19, in import_from_string
    module = importlib.import_module(module_str)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\LLMs\API.py", line 74, in <module>
    function_model.save_low_bit("FunctionCallingModel/low_bit")
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\ipex_llm\transformers\model.py", line 71, in save_low_bit
    self.save_pretrained(*args, **kwargs)
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\transformers\modeling_utils.py", line 2114, in save_pretrained
    save_function(shard, os.path.join(save_directory, shard_file))
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\torch\serialization.py", line 618, in save
    with _open_zipfile_writer(f) as opened_zipfile:
  File "C:\Users\ellio\miniconda3\envs\llm\Lib\site-packages\torch\serialization.py", line 466, in __exit__
    self.file_like.write_end_of_file()
RuntimeError: [enforce fail at inline_container.cc:424] . unexpected pos 45056 vs 44950

It seems save_low_bit is not available for mistral models.

Would you mind providing more regarding your test code, env, etc. so we could reproduce this issue? :)

Would you mind providing more regarding your test code, env, etc. so we could reproduce this issue? :)

Thank you, and sorry for the delay, we must be operating in fairly different timezones. I'm operating on the latest version of Windows 11, have the latest drivers and oneAPI installation, using transformers==4.34.0.
This is the relevant section of code that I am using, roleplay_model saves just fine, as you can see with the error message it is function_model that fails (the pytorch one):
from fastapi import FastAPI, Request, UploadFile, File, Body
import torch
from ipex_llm.transformers import AutoModelForCausalLM, AutoModelForSpeechSeq2Seq
#from transformers import AutoTokenizer, WhisperProcessor, TextGenerationPipeline

from ipex_llm import optimize_model
from ipex_llm.optimize import low_memory_init, load_low_bit
from transformers import AutoTokenizer, WhisperProcessor

from pydantic import BaseModel

app = FastAPI()

do_save_low_bit = True

roleplay_model_path = "text-generation-webui/models/Nexusflow_Starling-LM-7B-beta"
function_model_path = "FunctionCallingModel"
whisper_model_path = "whisper-med-en"

if do_save_low_bit == True:
    # Load Roleplay LLM model to save it (transformers)
    roleplay_model = AutoModelForCausalLM.from_pretrained(roleplay_model_path, load_in_4bit=True, trust_remote_code=True)
    roleplay_model.save_low_bit(roleplay_model_path + "/low_bit")

    # Load Function calling LLM model to save it (pytorch)
    function_model = AutoModelForCausalLM.from_pretrained(function_model_path, trust_remote_code=True)
    function_model = optimize_model(function_model, low_bit='sym_int4')
    function_model.save_low_bit(function_model_path + "/low_bit")

    # Load Whisper model to save it (transformers)
    whisper_model = AutoModelForSpeechSeq2Seq.from_pretrained(whisper_model_path, load_in_4bit=True, trust_remote_code=True)
    whisper_model.save_low_bit(whisper_model_path + "/low_bit")
else:
    # Load Roleplay LLM model (transformers)
    roleplay_model = AutoModelForCausalLM.load_low_bit(roleplay_model_path + "/low_bit", trust_remote_code=True)
    roleplay_tokenizer = AutoTokenizer.from_pretrained(roleplay_model_path, trust_remote_code=True)

    # Load Function calling LLM model (pytorch)
    with low_memory_init():
        function_model = AutoModelForCausalLM.from_pretrained(function_model_path + "/low_bit", torch_dtype="auto", trust_remote_code=True)
    function_model = load_low_bit(function_model, function_model_path + "/low_bit")
    function_tokenizer = AutoTokenizer.from_pretrained(function_model_path, trust_remote_code=True)

    # Load Whisper model (transformers)
    whisper_model = AutoModelForSpeechSeq2Seq.load_low_bit(whisper_model_path + "/low_bit", trust_remote_code=True)
    whisper_model.config.forced_decoder_ids = None
    whisper_processor = WhisperProcessor.from_pretrained(whisper_model_path)
    forced_decoder_ids = whisper_processor.get_decoder_prompt_ids(language='english', task="transcribe")

    # Move models to GPU
    roleplay_model = roleplay_model.to('xpu')
    function_model = function_model.to('xpu')
    whisper_model = whisper_model.to('xpu')

    # Complete an inference on each model to initialize the model
    roleplay_model.generate(roleplay_model.dummy_inputs, max_new_tokens=1)
    function_model.generate(function_model.dummy_inputs, max_new_tokens=1)
    whisper_model.generate(whisper_model.dummy_inputs, max_new_tokens=1)
    ```

![image](https://github.com/intel-analytics/ipex-llm/assets/83019499/a8ccacbc-82a9-47c2-a961-89623ee180fd)

Hi @ElliottDyson,
Could you first verifying that there is sufficient space available on your disk to save the model, and checking whether your model file has been corrupted by another process when saving the model?
If those steps don't resolve the issue, would you mind providing us with your FunctionCallingModel (e.g. huggingface repo id of this model), and run the environment checking scripts to share the results? This will help us better understand and reproduce the exact error.
P.S. If FunctionCallingModel can be loaded with ipex_llm.transformers.AutoModelForCausalLM, it could also be loaded and saved with:
function_model = AutoModelForCausalLM.from_pretrained(function_model_path, load_in_4bit=True, trust_remote_code=True)
function_model.save_low_bit(function_model_path + "/low_bit")


VertexAIGeminiChatGenerator streaming_callback implementation (or streaming response)

Is your feature request related to a problem? Please describe.
Similar to this ticket, the VertexAIGeminiChatGenerator does have streaming_callback implementation, even though the underlying GenerativeModel does support stream=True
https://github.com/deepset-ai/haystack/issues/7836
The underlying model supports an iterable chat response
https://github.com/googleapis/python-aiplatform/blob/main/vertexai/generative_models/_generative_models.py#L522
Describe the solution you'd like
Similar to this
https://github.com/deepset-ai/haystack/pull/8054, by @Amnah199 :)

I think these are pretty much the same but for Google AI generators @julian-risch and @Amnah199 = https://github.com/deepset-ai/haystack-core-integrations/issues/1002


Don't attempt to fetch from remote when using non final versioning strategy

Not exactly sure of the best way to word this but I'd like to be able to have not only developers but our CICD process as well call ./gradle release and have the correct path be taken. For developers I'd like the version generated from the local tag as things work now but NO attempting to talk to the remote repository. I ask how to do this as we, as part of our process, add a dependsOn "publishing" task to the release task. This "publishing" task will publish snapshot versions, which is legal for developers to do, but I don't want any remote checking lest we get a Caused by: org.eclipse.jgit.api.errors.TransportException: <git-url>: not authorized exception.
Any ideas? Am I missing something simple? Posted a snippet of our setup below if that helps at all. Thanks in advance!
    release {
        def credentials = new org.ajoberstar.grgit.Credentials(releaseUsername, releasePassword)
        grgit = org.ajoberstar.grgit.Grgit.open(dir: project.rootDir.absolutePath, creds: credentials)
        versionStrategy org.ajoberstar.gradle.git.release.opinion.Strategies.FINAL
        defaultVersionStrategy = org.ajoberstar.gradle.git.release.opinion.Strategies.SNAPSHOT
        tagStrategy {
            generateMessage = { version -> "Version ${->project.version}" }
        }
    }

I think if you want to skip the fetch you could just exclude the prepare task.
./gradlew release -x prepare

or permanently:
build.gradle
prepare.enabled = false

@ajoberstar that's exactly what I was looking for. Thanks!

@ajoberstar so while that gets us further, meaning the initial call out to the remote repo in the prepare task is "skipped", we still get a failed build at the end when the release task fails to push remotely (even though we're on a SNAPSHOT) version. Any thoughts/ideas?
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':release'.
> Problem pushing to remote.

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':release'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:100)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:70)
        at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:62)
        at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:60)
        at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:97)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:87)
        at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:52)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker$1.run(DefaultTaskGraphExecuter.java:248)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:336)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:328)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:199)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:110)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:241)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:230)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.processTask(DefaultTaskPlanExecutor.java:123)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.access$200(DefaultTaskPlanExecutor.java:79)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:104)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:98)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.execute(DefaultTaskExecutionPlan.java:626)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.executeWithTask(DefaultTaskExecutionPlan.java:581)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.run(DefaultTaskPlanExecutor.java:98)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
Caused by: org.ajoberstar.grgit.exception.GrgitException: Problem pushing to remote.
        at org.ajoberstar.grgit.operation.PushOp.call(PushOp.groovy:117)
        at org.ajoberstar.grgit.operation.PushOp.call(PushOp.groovy)
        at java_util_concurrent_Callable$call.call(Unknown Source)
        at java_util_concurrent_Callable$call.call(Unknown Source)
        at org.ajoberstar.grgit.util.OpSyntaxUtil.tryOp(OpSyntaxUtil.groovy:45)
        at org.ajoberstar.grgit.Grgit.methodMissing(Grgit.groovy:188)
        at org.ajoberstar.gradle.git.release.base.BaseReleasePlugin$_addReleaseTask_closure4_closure6.doCall(BaseReleasePlugin.groovy:97)
        at org.gradle.api.internal.AbstractTask$ClosureTaskAction.execute(AbstractTask.java:718)
        at org.gradle.api.internal.AbstractTask$ClosureTaskAction.execute(AbstractTask.java:691)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$1.run(ExecuteActionsTaskExecuter.java:121)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:336)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:328)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:199)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:110)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:110)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:92)
        ... 29 more
Caused by: org.eclipse.jgit.api.errors.TransportException: https://git.pega.io/scm/pp/gradle-prpc-platform-plugins.git: not authorized
        at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:164)
        at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:80)
        at java_util_concurrent_Callable$call$0.call(Unknown Source)
        at org.ajoberstar.grgit.operation.PushOp.call(PushOp.groovy:114)
        ... 44 more
Caused by: org.eclipse.jgit.errors.TransportException: https://git.pega.io/scm/pp/gradle-prpc-platform-plugins.git: not authorized
        at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:498)
        at org.eclipse.jgit.transport.TransportHttp.openPush(TransportHttp.java:387)
        at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:154)
        at org.eclipse.jgit.transport.Transport.push(Transport.java:1200)
        at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:157)
        ... 47 more

Ah, missed the push in release. It will always try to push the branch you're on if it's not a detached head, even if there's no tag to push. This isn't configurable right now, and given the maintenance status of this project, I'd rather not change it now.
You can probably work around this by not running release and leaving prepare disabled. Neither of them is doing anything you want, given your use case.
Alternatively, my replacement plugin for the release-* plugins (reckon) only interacts with the remote if you explicitly call reckonTagPush. That might work better for your use case.

Thanks. Yeah I ended up getting around by doing something like "if releaseStage == 'final' apply and configure plugin; else read tag from disk and invoke publish task only". It's not pretty IMO but gets the job done.
I'll check out your reckon plugin when I've got a few cycles to spare.


docs: update env vars acording to CLI output

Reverts redhat-developer/app-services-guides#468
Reintroduces changs for quickstarts introduced by @rkpattnaik780

@carlesarnal FYI. This PR would be blocked till we will be able to configure serdes with OAUTH_TOKEN_ENDPOINT_URI value.

#478 PR created to unblock this one

We cannot merge this change due to current version of the registry requiring full API path.
We should first review, merge and release:
https://github.com/redhat-developer/app-services-cli/pull/1539
Making change for the urls in the APP but also change in the guide is needed.

@rkpattnaik780 We can release RHOAS CLI to unblock this PR form merging

A new version of the CLI has been released. Merging!


Could not find com.yandex.money:checkout:<IP_ADDRESS>.

When I tried adding dependency implementation 'com.yandex.money:checkout:<IP_ADDRESS>', I got following error. What could be the reason?

Hi. You forget to add our bintray repository to your repositories section in build.gradle file. Check Readme.md for more information.

Thank you for replying..
I have added bintray repository in project level build.gradle like following screenshot :>

Even I tried running the sample code provided in your repo but issue is same with minor changes in error. Following is the screenshot after compilation of the yandex-checkout-android-sdk-master :>

You should add bintray repository to allprojects.repositories instead of buildscript.repositories

it doesn't help me


Recipe request: dynamically importing a Svelte component

based on @higsch's request: https://twitter.com/h_i_g_s_c_h/status/1353809757602197506
i think it is a mix of:

svelte component template: https://github.com/sveltejs/component-template
svelte:component: https://svelte.dev/tutorial/svelte-component

but am also wondering if the default webpack/rollup loaders would already handle this seamlessly. not sure

I think it is the way @Rich-Harris is importing and embedding components at the NY Times.
window.NYTG.load = function (module) {
  var url = new URL(module, location.href).href;
  return eval("import('" + url + "')");
};

Promise.all([
  NYTG.fetch(
    'https://static01.nyt.com/newsgraphics/2020/03/16/coronavirus-maps/d4b63b1513c5d5f9b232475a9b6ce5a2b255ebc6/data/navigation_links.json'
  ),
]).then(function (data) {
  var target = document.querySelector('[data-component="2"]');
  var url = NYTG.VIRUS_ASSETS + '/build/js/breadcrumbs.js';
  var props = { navigation: data[0] };

  let component;

  NYTG.load(url).then((mod) => {
    const Component = mod.default;

    component = new Component({
      target,
      hydrate: true,
      props,
    });
  });
});

But I have no idea how I would need to compile my component to be working with this pattern.

I think it is the way @Rich-Harris is importing and embedding components at the NY Times.
window.NYTG.load = function (module) {
  var url = new URL(module, location.href).href;
  return eval("import('" + url + "')");
};

Promise.all([
  NYTG.fetch(
    'https://static01.nyt.com/newsgraphics/2020/03/16/coronavirus-maps/d4b63b1513c5d5f9b232475a9b6ce5a2b255ebc6/data/navigation_links.json'
  ),
]).then(function (data) {
  var target = document.querySelector('[data-component="2"]');
  var url = NYTG.VIRUS_ASSETS + '/build/js/breadcrumbs.js';
  var props = { navigation: data[0] };

  let component;

  NYTG.load(url).then((mod) => {
    const Component = mod.default;

    component = new Component({
      target,
      hydrate: true,
      props,
    });
  });
});

But I have no idea how I would need to compile my component to be working with this pattern.

Oh, I think it works with index.mjs from https://github.com/sveltejs/component-template.

Oh, I think it works with index.mjs from https://github.com/sveltejs/component-template.

@higsch i saw from twitter that you managed to solve it right? do you think it's worth writing up a solution? til now i'm not 100% sure what your requirements were heheh

@higsch i saw from twitter that you managed to solve it right? do you think it's worth writing up a solution? til now i'm not 100% sure what your requirements were heheh

Yep, I solved it. It has simply been taking the index.mjs instead of the index.js from the compiled component template. I used it for a project which will be published soon, then I can link this here.

Yep, I solved it. It has simply been taking the index.mjs instead of the index.js from the compiled component template. I used it for a project which will be published soon, then I can link this here.


Add daily configurations for standard builds with -includeSlow.

Add daily configurations for standard builds with -includeSlow. Closes #46.

@dotnet-bot test this please

Replaced by https://github.com/Microsoft/ChakraCore/pull/84


[Visualizer Connector] Add the possibility to send messages from the custom visualizer.

Description
Now, with the visualizer connector, you could get your data to visualize, but we don't have the possibility to send some data back to the model.
It would be good to have both-sided communication.
Expected result:
The visualizer could send data to the chat.
Use case/motivation
No response
Related issues
No response
Confidential information

[X] I confirm that do not share any confidential information

Confirmed with a separate team that the feature works as expected.


#7204 do not rely on bootstrap css

looks good!
small bug: spark progress round corners appear:

also: bring up to date with master.

BeakerX tab in tree view is empty until you click again:

it works right away for me


Bug Report

Environment
Windows build number: Microsoft Windows [Version 10.0.18922.1000]
Windows Terminal version (if applicable): Windows Store Terminal (Preview), v6.2.1
Steps to reproduce
Try to open the app.
Expected behavior
Terminal opens and functions.
Actual behavior
Window looks like it will load, then flashes and disappears.
I have tried:

uninstalling, rebooting, and reinstalling
loading the terminal in another profile
running DISM and SFC, no problems found, no other system weirdness

#1364


Remove stale routing policies

This ensures that routing policies for non-existent load balancers are deleted.
Systems with frequent re-creation of deployments (e.g. CD) causes many load
balancers to be provisioned and deprovisioned. Each provisioned load balancer
gets a new hostname, and thus a new routing policy.
Since routing policies for each and every load balancer was kept, we ended up
with a large number of remove records requests and thus a large
NameServiceQueue.

sdf


Skipping flaky tests for panels and notebooks in Observability

Description
Latest RC for 2.10 release has seen 2 flaky tests in Observability plugin
Issues Resolved
Skips below flaky tests:

6_notebooks.spec.js/Testing paragraphs -- Duplicates paragraphs
4_panels.spec.js/Testing panels table -- Deletes panels

Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Cypress Panels Run:
Running:  plugins/observability-dashboards/4_panels.spec.js                               (1 of 1)
Browserslist: caniuse-lite is outdated. Please run:
npx browserslist@latest --update-db

Why you should do it regularly:
https://github.com/browserslist/browserslist#browsers-data-updating
Couldn't determine Mocha version

  Testing panels table
    ✓ Creates a panel and redirects to the panel (10286ms)
    ✓ Duplicates a panel
    ✓ Deletes panels (5732ms)

  Testing a panel
    ✓ Move to test panel
    ✓ Change date filter of the panel

  Clean up all test data
    - Deletes test panel

  5 passing (24s)
  1 pending

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        6                                                                                │
  │ Passing:      5                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      1                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     24 seconds                                                                       │
  │ Spec Ran:     plugins/observability-dashboards/4_panels.spec.js                                │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

Noteboooks Cypress Run:
 Running:  plugins/observability-dashboards/6_notebooks.spec.js                            (1 of 1)
Browserslist: caniuse-lite is outdated. Please run:
npx browserslist@latest --update-db

Why you should do it regularly:
https://github.com/browserslist/browserslist#browsers-data-updating
Couldn't determine Mocha version

  Adding sample visualization
    ✓ Add sample observability data (12599ms)

  Testing notebooks table
    ✓ Creates a notebook and redirects to the notebook (6354ms)
    ✓ Duplicates and renames a notebook (16858ms)
    ✓ Deletes notebooks (14302ms)

  Testing paragraphs
    ✓ Goes into a notebook and creates paragraphs (16307ms)
    ✓ Renders markdown
    ✓ Shows output message (6970ms)
    - Duplicates paragraphs
    ✓ Adds a dashboards visualization paragraph (15076ms)
    ✓ Adds a SQL query paragraph (18704ms)
    ✓ Adds a PPL query paragraph (18454ms)
    ✓ Runs all paragraphs (12048ms)
    ✓ Deletes paragraphs (12736ms)
    ✓ Cleans up test notebooks (10146ms)

  clean up all test data
    ✓ Delete visualizations from event analytics (12681ms)
    ✓ Deletes test panel (10073ms)

  15 passing (3m)
  1 pending

Creating issue: https://github.com/opensearch-project/opensearch-dashboards-functional-test/issues/892 for tracking fix on this post 2.10 release


Timeout Issue in Entity Framework Core

I have a multi threading application that needs to insert bulk of records in the same table from different places.  After continuously running 2 Hrs my application gets connection time out issue
Stack Trace
System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (258): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
I suspect this may be the cause of deadlock in a particular table. I verified in SQL that table is on deadlock.
I have tried the following solution

using (var scope = new TransactionScope(TransactionScopeOption.Required, new TransactionOptions { IsolationLevel = IsolationLevel.ReadUncommitted }))

I tried with SERIALIZABLE and  SNAPSHOT isolation levels also

When running the multi threading application it says MSDTC Server on unavailble . Note: I started the Distributed Transaction Coordinator service but still i get your environment not support distributed transactions
I tried with SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED but still i am getting the same issue after 2Hrs running continuously
I added this  SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED between the BeginTransaction and Commit transaction in my C# code but no luck.
Could you please someone can help me to overcome from this ?

There don't seem to be any Entity Framework classes in your stack trace, are you using EF Core? Consider also posting this question on stackoverflow.
Either way, you'd have to provide some code sample to help understand what you're doing. The error mentioning MSDTC indicates that your code may be causing distributed transactions (probably unintentionally), which could be the source of your problem - you should definitely not be using distributed transactions unless you specifically need them, as they can cause severe perf degradation and other problems as well.

EF Team Triage: Closing this issue as the requested additional details have not been provided and we have been unable to reproduce it.
BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.

@gopal-k Did you find a solution to this issue? I am facing the same issue.


Use Weblate for translation

It would be great to host our own instance for translation for all uCoin project: uCoin, CuteCoin, ucoin-android-app, RTM with the free Weblate.

Add a link on the README.


ItemToggleComponent is hard coupled to PowerCellDrawComponent

Description

The battery won't even drain if you don't have both components on the entity. what the fuck, man.
why do we make events if we're not going to use them.
make a new component that interfaces between both of them.
christ.

@deltanedas

it should still draw when missing the component, very sus


Issues running the save.py file

Hi I get the following error:
TypeError: Values passed to parameter 'x' has a DataType of int32 not in the list of allowed values: bfloat16, float16, float32, float64
Anyone else faced a similar issue? Any pointers will really help.
Regards,
Harsh

ok sorry to bother, it's a python2 issue -


Add MustOnlyDependOnComposerDependencies

Enhancement description
In a Mono-Repo setup it is common that each module has it's own composer.json file that defines the dependencies. However it is easy to depend on a dependency from another module.
Also dependencies between modules in the same mono-repo should also be defined inside each modules composer.json.
It would be great if we don't have to replicate the whole information from the composer.json, but rather use the composer.json as a source for our architecture testing.
**Suggested approach or solution **
Didn't look at the source yet, however phparch has a similar feature: https://github.com/j6s/phparch#shorthand-for-monorepos-addcomposerbasedcomponent
The phparch solution misses one important thing, as it expects a composer.lock file inside each module, in a mono-repo setup especially for development purposes you mostly just have one lock file in the mono-repo root containing the dependencies of all modules.
https://github.com/symfony/symfony being a good example of such setup

Hi Jonas, thanks for the detailed description 😄
I'm currently working on a refactor (you can check the secondary branch) that affects the way classes are picked to create a rule statement, which will open the possibility to create more versatile selectors and rule types. I expect to finish this refactor during the Christmas holidays, so I guess this can be developed after that. PHPat needs those changes in order to implement something like this.
Johannes (from phparch) is a contributor of this project so I guess his knowledge would be really useful for this feature.
Long story short. This feature will definitely be included but not immediately.
Keep in touch! :)

Just to make sure you dont re-implement something already existing:
https://github.com/maglnet/ComposerRequireChecker

@staabm definitely is a similar feature. I didn't know about that repo!
Seems like ComposerRequireChecker checks that your code is not coupled to second-level vendors that are not explicitly required in your composer.json.
Maybe it might have a place on phpat if you don't want certain modules to be coupled to vendors defined in another composer.json of your mono-repo (or maybe you want to avoid this on critical classes only). I think this rule system could give more versatility to that feature.
BUT: Since right now phpat's validation works like CLASS->RULETYPE->CLASS being those classes necessarily parsed on the AST creation step, it's not something that could be easily implemented right now.

@keulinho I made a phpstan-extension which can do this. Its not perfect but it works. Until this feature  is implemented you could take a look at it: https://github.com/nusje2000/phpstan-monolith.

Surprised to see phparch mentioned here. I agree, having a MustOnlyDependOnComposerDependencies equivalent in phpat would be awesome.

The phparch solution misses one important thing, as it expects a composer.lock file inside each module, in a mono-repo setup especially for development purposes you mostly just have one lock file in the mono-repo root containing the dependencies of all modules.

That statement is actually not quite true: By default it will search for a composer.lock next to the supplied composer.json, but you can pass it your own as a second argument (this is the way I personally use it at work):
$architecture = (new Architecture)
    ->addComposerBasedComponent('packages/subpackage/composer.json', 'project-root/composer.lock');

That being said, don't heavily invest in phparch, phpat is the future.

@j6s thanks for the clarification, i must have missed that one.

That being said, don't heavily invest in phparch, phpat is the future.
That's why i would like to see a similiar feature in phpat :)

Awesome contribution by @j6s released in 0.6.1 🍾
You can see a couple of examples here https://github.com/carlosas/phpat/blob/master/tests/architecture/ComposerTest.php
In a near future I will work on an improvement:

declare the composer files path in the config with an alias (then in the rule you just need to write the alias)
include the composer source/dependency map in the AST so it gets parsed once instead of every call


⚠ WSINGRESOS - GENERICO has degraded performance

In 5b89bd7, WSINGRESOS - GENERICO (https://srvshyweb.yucatan.gob.mx/cgi-bin/wspd_cgi.sh/WService=wsingresos/internet/servicios/wsGenericoPagosWeb.r) experienced degraded performance:

HTTP code: 200
Response time: 2123 ms

Resolved: WSINGRESOS - GENERICO performance has improved in 5f98dd7.


Changing Travis image

The Travis image we were using lost support for HHVM. This should allow us to properly test HHVM again.

Coverage remained the same at 14.878% when pulling 90f712e32a66a2966b4ee6bce6e00b5e7b613d8d on FixTravis into e568bcc0afa5ccaf862cec603ab29e1b06147873 on master.


Add descending issues order

Greetings @OrkoHunter
Issue #51

Hello @chinskiy! Thanks for submitting the PR.

In the file pep8speaks/helpers.py, following are the PEP8 issues :

Line 228:12: E713 test for membership should be 'not in'

This looks good. Thanks for taking this up !


Refactor arguments + environment variables

Implement minimist for cli options (--host, --port, --threads etc.)
Review puppeteer constructor arguments and window.init, use an object instead of argument list
Implement --threads

fix #7
My next step would be to implement some pretty logging that looks just like the miner on https://coin-hive.com/ homepage.
ping @tailot  WDYT?

Looking great! The only thing I would change is that I would set the default THREADS to something like null or -1 or whatever value that makes more sense for you, and when that value is passed to window.init I would do something like:
if (threads != null) {
  miner.setNumThreads(threads);
}

I would do it this way because by default the miner uses navigator.hardwareConcurrency to find the number of CPU cores.

@soyuka Good job!
You could join pieces like these:
  const interval = argv.interval || defaults.INTERVAL || process.env.COINHIVE_INTERVAL;
  const port = argv.port || defaults.PORT || process.env.COINHIVE_PORT;
  const host = argv.host || defaults.HOST || process.env.COINHIVE_HOST;
  const threads = argv.threads || defaults.THREADS || process.env.COINHIVE_THREADS;

  const miner = await CoinHive(siteKey, {interval, port, host, threads});

in
  const options = {
    interval: argv.interval || defaults.INTERVAL || process.env.COINHIVE_INTERVAL,
    port: argv.port || defaults.PORT || process.env.COINHIVE_PORT,
    host: argv.host || defaults.HOST || process.env.COINHIVE_HOST,
    threads: argv.threads || defaults.THREADS || process.env.COINHIVE_THREADS
  }
  const miner = await CoinHive(siteKey, options);

If you add a parameter, make a single change.

Thanks for the review guys, updated.
Interestingly, using setThread was starting the miner. I've changed tests back to what they were but maybe it's worth mentioning?
I've also updated the README  with detailed instructions.


This project needs a cool icon...

...so I can add it to the About page of my site.
:smiley:

And thus started the only issue he could never resolve...

(Though that doesn't mean I'm not going to try! I've managed to convince someone who's good at this stuff to take a stab at it!)

Closing this issue for now. It's not forgotten though.


Add autocompleteOff to MatFormFieldDefaultOptions

Feature Description
Feature Description
Please add an option to MatFormFieldDefaultOptions to disable the autocomplete on the field.
This will allow us to default all form fields with disabled autocomplete of specifying the autocomplete="off"on each one. Add the field to MatFormFieldDefaultOptions and default the value of the autocompleteOff input of MatFormField this this value (I think this is trivial)
Use Case
In our app our UX team wants all autocompletes on all forms to be turned off default, so setting it on each from is overkill. We have to specify autocomplete="off" on every single usage of MatFormField which is tedious and becomes error prone. Add this feature would eliminate this issue for us.
PS: this feature request is based on this feature request: https://github.com/angular/components/issues/16243

I think MatFormField has nothing to do with this. It does not have the reference of the underlying native input which could be a select too. Select does not have autocomplete attribute. This feauture could be implemented in MatInput, however IMO it's easy to create a directive that matches the [matInput]:([autocomplete]) selector which dose this thing for you.

We already tried the approach using a directive, but since it seems to be impossible to register global directives since Angular 2.0, this is not a good approach. We'd need to import a new module (containing the directive) into dozens of other modules :disappointed:

I see how this can be useful, but as @Totati said above, the Material input has nothing to do with the autocomplete so introducing a feature like this would be additional maintenance cost for no benefit. We'll also have to introduce an @Input() autocomplete in case some input fields need to be different from the default. There's also the issue that if we do it for autocomplete, we might as well do it for all the other attributes in the HTML spec.


Add tenant and display_name config options

Hi @Boltgolt ,
I get the following error when I run your commit. It has a problem when parsing the
 toon:
    - Invalid config for [toon]: string value is None for dictionary value @ data['toon']['display_name'].
       Got None. (See /home/homeassistant/.homeassistant/configuration.yaml, line 70).
       Please check the docs at https://home-assistant.io/components/toon/

If I change
CONF_NAME = 'display_name'
DEFAULT_NAME = None

to
CONF_NAME = 'display_name'
DEFAULT_NAME = ''

the error disappear. Is this an error or am I doing something wrong?

@Housebeer this should work fine None but maybe @Boltgolt could have a look at it. I still have not migrated to this newer component and still using the older one with the unofficial api.

@Housebeer thanks for reporting! Turns out that newer hass versions have a more strict policy on conifg types, my last push should fix this.


MGDOBR-975: Error Handling: Check we can update type from Webhook to Kafka (#1017) - Update kustomization images

This Pull request aims to update the kustomization images for the PR MGDOBR-975: Error Handling: Check we can update type from Webhook to Kafka (#1017)

/deploy dev


docs,tools/etcd-dump-metrics: automate metrics documentation

Address https://github.com/coreos/etcd/issues/9438.

/cc @wenjiaswe

Codecov Report

Merging #9984 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #9984      +/-   ##
==========================================
+ Coverage   69.34%   69.35%   +0.01%
==========================================
  Files         386      386
  Lines       35914    35914
==========================================
+ Hits        24905    24909       +4
+ Misses       9212     9210       -2
+ Partials     1797     1795       -2

Impacted Files
Coverage Δ

pkg/transport/timeout_conn.go
80% <0%> (-20%)
:arrow_down:

clientv3/namespace/watch.go
72.72% <0%> (-6.07%)
:arrow_down:

pkg/logutil/zap_grpc.go
47.61% <0%> (-4.77%)
:arrow_down:

pkg/transport/listener.go
58.67% <0%> (-4.09%)
:arrow_down:

clientv3/concurrency/election.go
79.52% <0%> (-2.37%)
:arrow_down:

etcdctl/ctlv3/command/printer_simple.go
72.48% <0%> (-1.35%)
:arrow_down:

pkg/adt/interval_tree.go
90.39% <0%> (-0.91%)
:arrow_down:

etcdserver/api/rafthttp/stream.go
79.39% <0%> (-0.43%)
:arrow_down:

clientv3/watch.go
91.5% <0%> (-0.43%)
:arrow_down:

pkg/proxy/server.go
59.86% <0%> (-0.34%)
:arrow_down:

... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 93be31d...902893f. Read the comment docs.

@gyuho I am sorry I forgot to follow up with #9950. I will do this after future releases. Thanks!

@wenjiaswe That would be great. Thanks!


fix(css/parse): pseudo element and pseudo class spaces

Fix, whitespace is not allowed between

Between the ':'s, or between the ':' and <ident-token> or <function-token>, of a <pseudo-element-selector> or a <pseudo-class-selector>.

From https://www.w3.org/TR/selectors-4/#white-space

In future we will add recovery mode for selectors and all examples will be valid, but just output tokens, now we don't have it, but we will ad it late when we finish selector parser


Login fixes

bypass firebaseUI for anonymous login
login component should appear in a modal window (when clicked on login button)
if using a useAnonymousLogin hook, verify no double account is created when the function is called twice

Need to merge #330 before starting this one


Why do we use 2 databases?

Found Google's Level DB and Berkeley DB.
What do we want from them? Do we really need both?

Ok.
Leveldb is key:value
Berkeleydb provides more sql-like tables.


iterator_to_array($cursor) returns indexed array instead of associative array

when i use iterator_to_array($cursor), how could i get associative array which '_id' as key,like older one
thank you

Duplicate of https://github.com/mongodb/mongo-php-driver/issues/430.