fit_generator crashes though keras.utils.data_utils.Sequence was used

When model.fit_generator is used with workers=0 and subclasses of keras.utils.data_utils.Sequence for both training and validation data, API of Sequence is not recognized inside evaluate_generator, it raises:
  File ".../keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File ".../keras/engine/training.py", line 1415, in fit_generator
    initial_epoch=initial_epoch)
  File ".../keras/engine/training_generator.py", line 230, in fit_generator
    validation_steps,
  File ".../keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File ".../keras/engine/training.py", line 1469, in evaluate_generator
    verbose=verbose)
  File ".../keras/engine/training_generator.py", line 298, in evaluate_generator
    else:
ValueError: `steps=None` is only valid for a generator based on the `keras.utils.Sequence` class. Please specify `steps` or use the `keras.utils.Sequence` class.

Example code:
from keras import Sequential
from keras.layers import Dense
from keras.utils.data_utils import Sequence
import numpy as np

class Dataset(Sequence):
    def __getitem__(self, index):
        return np.random.uniform(size=(16, 8)), np.random.uniform(size=(16, 1))
    def __len__(self):
        return 128

model = Sequential([Dense(4, activation='relu', input_shape=(8,)),
                    Dense(1, activation='sigmoid')])
model.compile(loss='mse', optimizer='adam')
model.fit_generator(generator=Dataset(), validation_data=Dataset(),
                    workers=0)

Issue can be fixed here by replacing:
if isinstance(val_data, Sequence):
    val_enqueuer_gen = iter(val_data)

with
if isinstance(val_data, Sequence):
    val_enqueuer_gen = iter(val_data)
    validation_steps = len(val_data)

Just updated to Keras 2.2.3
Bug still persists, but now it raises:
ValueError                                Traceback (most recent call last)
<ipython-input-3-9dd18d05ed53> in <module>
      3 model.compile(loss='mse', optimizer='adam')
      4 model.fit_generator(generator=Dataset(), validation_data=Dataset(),
----> 5                     workers=0)

...\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
     89                 warnings.warn('Update your `' + object_name + '` call to the ' +
     90                               'Keras 2 API: ' + signature, stacklevel=2)
---> 91             return func(*args, **kwargs)
     92         wrapper._original_function = func
     93         return wrapper

...\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
   1416             use_multiprocessing=use_multiprocessing,
   1417             shuffle=shuffle,
-> 1418             initial_epoch=initial_epoch)
   1419
   1420     @interfaces.legacy_generator_methods_support

c:\users\pmaevskikh\appdata\local\continuum\miniconda3\lib\site-packages\keras\engine\training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
    233                             val_enqueuer_gen,
    234                             validation_steps,
--> 235                             workers=0)
    236                     else:
    237                         # No need for try/except because

...\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
     89                 warnings.warn('Update your `' + object_name + '` call to the ' +
     90                               'Keras 2 API: ' + signature, stacklevel=2)
---> 91             return func(*args, **kwargs)
     92         wrapper._original_function = func
     93         return wrapper

...\keras\engine\training.py in evaluate_generator(self, generator, steps, max_queue_size, workers, use_multiprocessing, verbose)
   1470             workers=workers,
   1471             use_multiprocessing=use_multiprocessing,
-> 1472             verbose=verbose)
   1473
   1474     @interfaces.legacy_generator_methods_support

...\keras\engine\training_generator.py in evaluate_generator(model, generator, steps, max_queue_size, workers, use_multiprocessing, verbose)
    300             steps = len(generator)
    301         else:
--> 302             raise ValueError('`steps=None` is only valid for a generator'
    303                              ' based on the `keras.utils.Sequence` class.'
    304                              ' Please specify `steps` or use the'

ValueError: `steps=None` is only valid for a generator based on the `keras.utils.Sequence` class. Please specify `steps` or use the `keras.utils.Sequence` class.

I reviewed source of training_generator and found that in:
124            elif val_gen:
125                val_data = validation_data
126                if isinstance(val_data, Sequence):
127                    val_enqueuer_gen = iter_sequence_infinite(generator)
128                else:
129                    val_enqueuer_gen = val_data

in line 127 val_enqueuer_gen uses generator which is training data, instead of val_data, and it doesn't set validation_steps though val_data is keras.Sequence

Bug still exists in keras 2.2.4
ValueError: `steps_per_epoch=None` is only valid for a generator based on the `keras.utils.Sequence` class. Please specify `steps_per_epoch` or use the `keras.utils.Sequence` class.

@KIK More detalis, please. Provide code for x and model. Also which version of tensorflow do you use?

Here we go:
class DataGenerator(keras.utils.Sequence):
    'Generates data for Keras'
    def __init__(self, dataframe, features, labels, train_test_factor=0.8, batch_size=32, lstm_memory_size=52*5):
        'Initialization'
        self.dataframe = dataframe
        self.batch_size = batch_size
        self.features = features
        self.labels = labels
        self.train_test_factor = train_test_factor
        self.shuffle = False
        self.on_epoch_end()

    def __len__(self):
        'Denotes the number of batches per epoch'
        return int(np.floor(self.dataframe.shape[0] * self.train_test_factor)) - self.batch_size

    def __getitem__(self, index):
        'Generate one batch of data like [batch_size, lstm_memory_size, features]'

        # features
        features = np.apply_along_axis(np.hstack,
                                       1,
                                       self.dataframe[self.features] \
                                           .iloc[index:index + self.batch_size].values)
        # labels
        labels = np.apply_along_axis(np.hstack,
                                     1,
                                     self.dataframe[self.labels] \
                                         .iloc[index + 1:index + self.batch_size + 1].values)
        return [features], [labels]

    def on_epoch_end(self):
        'Updates indexes after each epoch'
        # later we could track/plot the model weights after each epoch
        if self.shuffle == True:
            print("dadasdasd -----------------------------")
            raise ValueError('Shuffeling not implemented yet')

    def get_dataframe(self):
        return self.dataframe

and
x = DataGenerator(df, ["sigma", "cos_dow"], ["sigma"], 0.8, 2)
x.__getitem__(x.__len__())
isinstance(x, keras.utils.Sequence)  #True

Finally
model.fit_generator(generator=x,
                    validation_data=x,
                    steps_per_epoch=x.__len__(), # https://github.com/keras-team/keras/issues/10855
                    use_multiprocessing=True,
                    workers=6,
                    shuffle=False)

keras version is 2.2.4
tensorflow version is 1.10.1

Hi, i was struggling with the exact same error message today on Windows, Tensorflow 1.12.0.
For me it turned out that I used tensorflow import (import tensorflow.keras.utils.Sequence as sequence) for the sequence generator, but for the model I was training I used the keras import (from keras.layers import..... and from keras.models import....etc.)
changing to tensorflow.keras.layers, tensorflow.keras.models....etc worked for me.

indeed this solves the problem! this was not obvious! thx!

So, it's not Keras issue, but Tensorflow one. That's why it's still closed.
By the way, keras and tensorflow.keras doesn't always match each other, so it's better to use either only keras imports or only tensorflow.keras imports.

Same issue here. no solution?

@ghasemikasra39 Traceback? Sample code?

Same issue here.
Keras                   2.2.4
tensorflow-gpu          1.12.0

@synchro-- traceback and code?

Same issue too.
keras.utils.Sequence class not recognize.

ValueError: steps_per_epoch=None is only valid for a generator based on the keras.utils.Sequence class. Please specify steps_per_epoch or use the keras.utils.Sequence class.

Keras: 2.1.5 and later tensorflow-gpu: 1.10.1

@AlessioA123 Update Keras to 2.2.4, and try again. If it's still fails provide sample code, otherwise I cannot help you

Tested with Keras 2.2.4 and get same error.
from keras.utils import Sequence

class Batchloader(Sequence):

def __init__(...):
def __len__(...):
def __getitem__(...):

A quick test
gen = Batchloader(...)

model.compile(...)

model.fit_generator(generator=gen, verbose=1, epochs=2)

Obtaining previous error.
Debugging i have see in /keras/engine/training.py on line: 2084 and after on line 2092
is_sequence = isinstance(generator, Sequence)

is_sequence is false, but generator (Batchloader) is a keras.utils.Sequence

I don't undestand how can be tensorflow, traceback says:
/lib/python3.6/site-packages/keras/engine/training.py
without no reference to tensowflow.
It's a virtualenv with only tensorflow-gpu (not used inside scripts, only as backend for keras) and keras.
I have do some test but got the same issue with keras 2.1.5, 2.2.3, 2.2.4

@AA12321 run this:
import keras
print(keras.__version__, keras.__file__)

Also, which command have you used to install/update keras?

You right, i don't understand how it's possible but the old venv doesn't work properly. I create a new one and get same error.
Got these
2.1.5 /envKeras/lib/python3.6/site-packages/keras/__init__.py
2.2.4 /envKeras/lib/python3.6/site-packages/keras/__init__.py
Both tensorflow and keras was installed with
pip3 install tensorflow-gpu
pip3 install keras==2.X.X

So you created new virtualenv like:
# in project folder
python3 -m venv kerasenv
source ./kerasenv/bin/activate
python3 -m pip list
python3 -m pip install keras==2.2.4 tensorflow-gpu
python3 ./myscript.py

And it's still fails?

Same issue on colab!

@AA12321 @mirfan899 , You can try this notebook

@AA12321 Try code sample I provided in the first message. If it's also fails, then I cannot help.

@arquolo Tried your sample code and working.
So in keras 2.2.4 with sequential model works, i use a custom model that required keras 2.1.4 to load the weights, so i think that's my problem. Thanks!

The solution which I got is steps_per_epoch=len(x_train)/batch_size or whatever you want the number of steps for each epoch by setting steps_per_epoch in fit_generator

same error here:
Trying to run the cifar 10 cnn example returns:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Program Files\JetBrains\PyCharm 2018.2.4\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2018.2.4\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/Omar/tensorflow_test/examples_keras_new/cifar10_cnn.py", line 118, in <module>
    workers=4)
  File "C:\Users\Omar\AppData\Local\conda\conda\envs\tensor19\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\Omar\AppData\Local\conda\conda\envs\tensor19\lib\site-packages\keras\engine\training.py", line 1418, in fit_generator
    initial_epoch=initial_epoch)
  File "C:\Users\Omar\AppData\Local\conda\conda\envs\tensor19\lib\site-packages\keras\engine\training_generator.py", line 55, in fit_generator
    raise ValueError('`steps_per_epoch=None` is only valid for a'
ValueError: `steps_per_epoch=None` is only valid for a generator based on the `keras.utils.Sequence` class. Please specify `steps_per_epoch` or use the `keras.utils.Sequence` class.

from an anaconda environment with:
Keras                   2.2.4
tensorflow-gpu          1.12.0

clean installation, full list of packages as follows:
(tensor19) C:\Users\Omar>conda list
# packages in environment at C:\Users\Omar\AppData\Local\conda\conda\envs\tensor19:
#
# Name                    Version                   Build  Channel
_tflow_select             2.1.0                       gpu    anaconda
absl-py                   0.6.1                    py36_0    anaconda
astor                     0.7.1                    py36_0    anaconda
blas                      1.0                         mkl    anaconda
ca-certificates           2018.03.07                    0    anaconda
certifi                   2018.10.15               py36_0    anaconda
cudatoolkit               9.0                           1    anaconda
cudnn                     7.1.4                 cuda9.0_0    anaconda
cycler                    0.10.0                    <pip>
gast                      0.2.0                    py36_0    anaconda
grpcio                    1.16.1           py36h351948d_1    anaconda
h5py                      2.8.0            py36h3bdd7fb_2    anaconda
hdf5                      1.10.2               hac2f561_1    anaconda
icc_rt                    2017.0.4             h97af966_0    anaconda
intel-openmp              2019.1                      144    anaconda
Keras                     2.2.4                     <pip>
keras-applications        1.0.6                    py36_0    anaconda
keras-preprocessing       1.0.5                    py36_0    anaconda
kiwisolver                1.0.1                     <pip>
libprotobuf               3.6.1                h7bd577a_0    anaconda
markdown                  3.0.1                    py36_0    anaconda
matplotlib                3.0.2                     <pip>
mkl                       2018.0.3                      1    anaconda
mkl_fft                   1.0.6            py36hdbbee80_0    anaconda
mkl_random                1.0.1            py36h77b88f5_1    anaconda
numpy                     1.15.4           py36ha559c80_0    anaconda
numpy-base                1.15.4           py36h8128ebf_0    anaconda
openssl                   1.1.1                he774522_0    anaconda
pip                       18.1                     py36_0
protobuf                  3.6.1            py36h33f27b4_0    anaconda
pyparsing                 2.3.0                     <pip>
python                    3.6.8                h9f7ef89_0
python-dateutil           2.7.5                     <pip>
PyYAML                    3.13                      <pip>
scikit-learn              0.20.2                    <pip>
scipy                     1.1.0            py36h4f6bf74_1    anaconda
setuptools                40.6.3                   py36_0
six                       1.11.0                   py36_1    anaconda
sklearn                   0.0                       <pip>
sqlite                    3.26.0               he774522_0
tensorboard               1.12.0           py36he025d50_0    anaconda
tensorflow                1.12.0          gpu_py36ha5f9131_0    anaconda
tensorflow-base           1.12.0          gpu_py36h6e53903_0    anaconda
tensorflow-gpu            1.12.0               h0d30ee6_0    anaconda
tensorflow-probability    0.5.0                     <pip>
termcolor                 1.1.0                    py36_1    anaconda
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.15.26706          h3a45250_0
werkzeug                  0.14.1                   py36_0    anaconda
wheel                     0.32.3                   py36_0
wincertstore              0.2              py36h7fe50ca_0
zlib                      1.2.11               h62dcd97_3    anaconda

I have keras 2.2.4 and tensorflow 1.12, when I downgrade to tensorflow 1.8.0, the problem goes away for cifar10_resnet.py (given in keras/examples). Might not be optimal soln, but at least there's a version of tensorflow for which sample code works.

A simple work around would be adding a steps_per_epoch value to keras.models.Sequential().fit_generator(steps_per_epoch = "Constant here")

The solution which I got is steps_per_epoch=len(x_train)/batch_size or whatever you want the number of steps for each epoch by setting steps_per_epoch in fit_generator

This works for me

It's works. Thanks!

The solution which I got is steps_per_epoch=len(x_train)/batch_size or whatever you want the number of steps for each epoch by setting steps_per_epoch in fit_generator

This works for me

It's works. Thanks!

It's really works and helpful to me ,great!

change steps_per_epoch=None to steps_per_epoch=nb_train_samples // batch_size

Using the solution mentioned here (posted by voscar-zhang) I get a new error:
ValueError: Must specify 'validation_steps' to perform validation when doing step-wise training.

Here is my working  codes
model.fit_generator(train_generator, epochs=epochs, steps_per_epoch=nb_train_samples // batch_size,
validation_data=test_generator, nb_val_samples=nb_val_samples //batch_size,
class_weight='auto')

The solution which I got is steps_per_epoch=len(x_train)/batch_size or whatever you want the number of steps for each epoch by setting steps_per_epoch in fit_generator

This works for me

It's works. Thanks!
me too, it works!!!!  thanks
我也可以用了

Using datagen.sample_per_validation//BS  still falls short of 56 samples for batch size 64. Any solution to fill the last batch?

The solution which I got is steps_per_epoch=len(x_train)/batch_size or whatever you want the number of steps for each epoch by setting steps_per_epoch in fit_generator

This works for me

It's works. Thanks!
me too, it works!!!!  thanks
我也可以用了
works for me too, thanks!

Hello I have this issue:
AttributeError: 'ImageDataGenerator' object has no attribute 'fit_generator'
can you please tell me how to solve it?
also, I think I have no problem of library versions or installation requirements.
thanks


Format task doesn't fix new lines to eof

The checkstyle https://checkstyle.sourceforge.io/config_misc.html#NewlineAtEndOfFile NewlineAtEndOfFile item doesn't seem to have a configuration option or accessibility to be applied during the formatKotlin task added by kotlinter. How can I get this working? It's a pretty common ask during pull request code reviews.

Thanks for letting me know. Yeah, that's definitely a common one.
It looks like a rule exists and it's part of the standard set in ktlint 0.33.0:
https://github.com/pinterest/ktlint/blob/master/ktlint-ruleset-standard/src/main/kotlin/com/pinterest/ktlint/ruleset/standard/FinalNewlineRule.kt
It appears to only enforce ending in a newline if editorconfig is present and specifies the value. Kotlinter-gradle isn't currently reading the editorconfig unfortunately.
Some changes to configuration and editorconfig are coming in the next version of ktlint which sounds like it's coming soon. Let's add complete configuration options to kotlinter-gradle at that time!

Should now work out of the box, see https://github.com/jeremymailen/kotlinter-gradle/releases/tag/2.0.0


Add REST API tests to check access rights when changing source/target storage

Motivation and context

How has this been tested?

Checklist

[ ] I submit my changes into the develop branch
[ ] I have created a changelog fragment
[ ] I have updated the documentation accordingly
[ ] I have added tests to cover my changes
[ ] I have linked related issues (see GitHub docs)
[ ] I have increased versions of npm packages if it is necessary
(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)

License

[ ] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.

Summary by CodeRabbit

New Features

Enhanced logic for task updates and linked storage management.
New test methods added for validating project and task associations.

Bug Fixes

Improved date comparison logic for assignee updates.
Refined task selection based on organization filtering.

Documentation

Updated test cases for clarity and maintainability.

Chores

Added new entries in various JSON files to expand datasets and improve functionality.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 74.34%. Comparing base (c9754a9) to head (6fa56b0).
Report is 4 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #8479      +/-   ##
===========================================
- Coverage    74.34%   74.34%   -0.01%
===========================================
  Files          394      394
  Lines        42216    42212       -4
  Branches      3896     3896
===========================================
- Hits         31387    31383       -4
  Misses       10829    10829

Components
Coverage Δ

cvat-ui
78.72% <ø> (-0.03%)
:arrow_down:

cvat-server
70.48% <63.63%> (+0.01%)
:arrow_up:


Move the check that the test data file is present to the start of the setup

This PR fixes the issue that, in the case the test data file was not provided or was put in the wrong folder and/or with the wrong name, the user would have know about it only 20 mins into the setup, and they would have needed to start over.
This PR moves the check at the beginning on the process.

@bjgill I believe I've addressed all your comment. If you are happy, could you please approve?


[Feature]:  Typescript Support

Is your feature request related to a problem? Please describe.
I'm frustrated by the lack of Typescript support
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response

Good news! TypeScript support has been added to the project generator.


[#IOPAY-181] migliorare i testi di errore

With this PR we show in error-modal a Title plus a body text
List of Changes
Created a new enum to map also "body message" for error-modal
Motivation and Context
Better UX with title + text
How Has This Been Tested?
Linter
Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.

this PR also has the changes of PR 50, 51 and 52, so you can only merge this one?


Unable to have "opt in" behaviour of injection since 1.0

Describe the bug
We run our cluster with a sidecar-injector config of policy: disabled, and then "opt in" the pods we want to have istio with the annotation:
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"

Observed behaviour
Pods with this annotation are not injected.
2018-07-31T17:11:22.265486Z     info    AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=bf97a4a8-94e4-11e8-b41a-42010aa4005e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 786d5efc-5615-11e8-807f-42010aa40068 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2018-07-31T17:11:22.265531Z     info    Skipping / due to policy check

Expected behaviour
On 0.8, this resulted in pods being injected if they have the annotation.
From our perspective, this means that istio has gone from "opt in" to "opt out" on our cluster, which is very none-backwards-compatible for us (we have plenty of deployments which don't specify the annotation and don't expect to be injected, but with policy: enabled, they would be).

OK I've worked this out.
There is a new config value in 1.0 .Values.global.proxy.autoInject, this used to be controlled by .Values.global.proxy.policy.
I incorrectly read autoInject as a true/false value.  Whereas actually, autoInject is actually a straight mapping to the old proxy.policy, thus needs to be disabled, not `false.
https://github.com/istio/istio/blob/master/install/kubernetes/helm/istio/templates/sidecar-injector-configmap.yaml#L15
So;

Probably needs a better named config option or the helm template perhaps should transform false to disabled?
Should be some validation somewhere that doesn't accept policy: false as a valid value.

@ostromart can you take a look?

Yes, this is on my plate. Will keep this bug open till the change is in.

https://github.com/istio/istio/pull/7572
This should at least avoid wild goose chases. Long term solution is to use operators for install to validate all that stuff.


fix(examples): add react-hot-loader to deps in webpack3 example

Rel #2559.

Codecov Report

Merging #2567 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2567   +/-   ##
=======================================
  Coverage   99.74%   99.74%
=======================================
  Files         160      160
  Lines        2744     2744
=======================================
  Hits         2737     2737
  Misses          7        7

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e13323...2b7b17d. Read the comment docs.

Released in<EMAIL_ADDRESS>


[Search] New APIs for Search Pop-Out

it took more researching than I thought to find the best approach for getting facet values and counts on type ahead.
The good news is that the Coveo Search API has an endpoint that can be used for this, and it does not count against Queries per month (QPM);
https://platform.cloud.coveo.com/docs?urls.primaryName=Search API#/Search V2/valuesBatchPost
However, as it is not an endpoint used often, there is no Headless/Atomic controller for it at the present time. This means the level of effort would be more important to develop a component for this use case.
Here’s an example of what the request body would look like:
{
"batch": [
{
"field": "@contenttype",
"maximumNumberOfValues": 8,
"queryOverride": "Cell Line Development"
},{
"field": "@processsteps",
"maximumNumberOfValues": 8,
"queryOverride": "Cell Line Development"
},{
"field": "@tools",
"maximumNumberOfValues": 8,
"queryOverride": "Cell Line Development"
}
]
}
Otherwise, a regular Search API request would also return facet values and counts on type ahead, but it would count against QPM.
Please let me know if you have any questions.

Hi, I know that @amlan18041996  is looking into the UI of this, but wasn't sure @rgravitvl , @sharanagoudapatil  or @mhaack  if we needed this task to implement the APIs for Amlan's UI to call.  I have added the info to this.

@davenichols-DHLS I have made a postman collection of api calls for this and have attached it to the Search popout issue assigned to Amlan. I am closing this issue.


Inputs for evaluate.py

Can you give some instructions regarding what input arguments should be provided for evaluate.py (Graphormer/graphormer/evaluate/evaluate.py) ?

@shiyu1994  Thanks!

Hi @shiyu1994
Was the description added as I cannot seem to find it?
Thanks


Persistent Cache for notebooks/text-classification/notebook.ipynb

Hi,
I'm trying to run notebooks/text-classification/notebook.ipynb .  Each time it runs, it recompiles, using a different /tmp/ path each time, e.g.
2023-06-27 17:59:23.000182: INFO ||NCC_WRAPPER||: Compile cache path: /tmp/tmpytdl1fxg
I've never actually completed the compile this way.  It may be progressing but very slowly.  After 30+ minutes it has made little progress (below).  Is this expected?
  0%|         | 1/1500 [00:11<4:41:45, 11.28s/it]2023-06-27 17:59:23.000182
.....................................  ............    ...

To remedy the compile time issue, I tried using Neuron SDK's persistent cache.  With neuron_parallel_compile, I see a cache created and populated at e.g. /var/tmp/neuron-compile-cache/neuronxcc-<IP_ADDRESS>+f7c6cf2a3.  However, for some reason this cache is not leveraged when I run the training.  Furthermore, when I rerun the neuron_parallel_compile,  6-8 graphs are always  recompiled and added into the cache.
Is it possible to use the persistent cache with the notebook?  Or otherwise compile and train in a timely way?

Did you make any changes to the notebook? Where are you running the notebook? On the Hugging Face AMI?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!


Create a module with common attributes

Closes #1558.
Signed-off-by: Igor Pashev<EMAIL_ADDRESS>

This change is

I am thinking about lazy static too:
lazy_static! {
    pub(crate) static ref CONTAINER: Attrs = class![C.relative, C.group];
}

I'd consider using lazy static sparingly, in this case a fn seems just as good and more straightforward.

Yes, it can be done as the global object, but I forsee the need to pass some additional parameters there, and then it will need to be converted to the function back.


gp: define database file format

want to be able to lookup from string to the number of times it's been seen in a representative dataset.
desirable fields:

global count
count among known malware
count among known goodware
section in file

propose starting with JSON because its easy for a human to inspect and explore. we can gzip the json file to keep the size small on disk. its up to floss to decode the entries in the file and index them as it needs.
initial proposal:
array of objects: [ {...}, {...}, {...} ]
each object has the keys:

string (string)
location (string): contains section name, or "overlay" or "header" or "gap"
global count (int)
known malware count (int)
known goodware count (int)

the counts are partitioned/aggregated by the (string, location) key.
like:
{
  "string": "1.1.2",
  "location": ".rdata",
  "global count":<PHONE_NUMBER>2,
  "known malware count": 35325,
  "known goodware count": 89137819,
}

we could use msgspec to quickly decode the entries in the document: https://github.com/jcrist/msgspec

maybe we should use the jsonl format (rather than wrapping the list in [] then each line is implied to be a json object). this way we can easily concatenate the database files together.

proposing
https://github.com/mandiant/flare-floss/blob/4bdf5c93afdf5ebc36b9d30914fd6e6eeb3d0499/floss/qs/db/oss.py#L9-L35
and tests https://github.com/mandiant/flare-floss/blob/quantumstrand/tests/test_oss_db.py
in 4bdf5c9
ok @mr-tz ?

Should we track ASCII vs. UTF16LE type as well?

this may not be obvious at the source code level, so it may introduce some difficulty during extraction.
it sounds maybe nice to have. how would we use this information? to avoid string collisions? what else?

I'm torn on the known goodware/malware count wrt. to:

updating the data
confusing analysts: this string is more often seen in good/bad so the file is good/bad
making it useful, e.g.

what are the base datasets?
would we ever update those?

The absolute numbers don't mean much IMO without a good explanation of the base data which would be hard to do.
Some of these aspects also apply to the general global count, but there we can have a hint like: out of strings data from N binaries (and don't have to qualify them into good/bad).

good arguments. lets remove this data until we have a great idea about how to collect it and explain it.

proposed database in:
https://github.com/mandiant/flare-floss/blob/2ccd4f8/floss/qs/db/gp.py
https://github.com/mandiant/flare-floss/blob/2ccd4f8/tests/test_gp_db.py

ah, sorry, this is is for GP database, which makes this easy to extract. sure, lets capture this, too.

actually, for some databases the encoding may not be available, can we mark it optional (meaning implicit unknown) or make it explicit with types: ascii, utf16le, unknown?

sounds good

looking at this now, I think indexing just by string suffices. we can inspect encoding (if desired) in the indexed objects

updated in e8d80535b938f11841bdd8e89c887aa351aa5a48
also moved dbs to data directory in floss/qs/db in 1baeca0a7236036ad1abec64abb91810bf541f5a


ISC proposal not accepted

Submitting this proposal was easy for me to do because I had a clear understanding of the technical requirements to get the proposal done and I knew that we would do it regardless of funding.
The downside: we will not be able to get there as soon as we thought.
The upside: this might give us the opportunity to work on i18n for lessons.

Dear Zhian N. Kamvar,
Thank you for sharing your proposal "Multi‐language Lessons in The
Carpentries Workbench" with us. Unfortunately, the ISC did not elect
to fund your proposal, as many proposals were submitted to the ISC and
only a limited number could be funded. While we believe your work to
be important, your proposal did not convince the committee that the
work you proposed would impact a sufficiently broad segment of the R
Community.
The ISC appreciates Data Carpentries' commitment to R training, but
the reviewers questions scope of the proposal.  It seems that this
proposal targets a general infrastructure for languages including
Python, SQL, and Bash, rather than something specific to R. If you
choose to re-apply, we suggest a proposal that gives a more narrow
scope focused on R.
All applicants are notified at the same time, so it is okay to share
the ISC's decision at this time.
Regards,
Henrik Bengtsson
Chair, Infrastructure Steering Committee
R Consortium

Q: Why did you not submit this to work on i18n in the first place???
A: It is not a low-risk problem. i18n has aspects of hosting, tooling, and coordination; aka not something I could do on my own. This requires coordination with several stakeholders of the i18n community and when I found out about this proposal deadline, I had one week to submit it.


fix(input): fix chrome autofill style

fixes #3355

@kara My app is starting with the login screen and Chrome autofills the login data if available. I think we also need a fix to update the form state, because even with filled data it is invalid until I click inside the form.
Right after the app starts:

Or do I have to check this on a viewChange detection cycle?

@dahaupt I think that might be some security feature where you have to interact with it before it actually makes the autocomplete data available to the site, you should file a separate issue though if you believe its a real problem. This PR was just about fixing the overlapping placeholder.


Move weight regularization @open sesame 02/02 20:01

Commit 1: [weight/layer] Move weight regularization out of layers
Move weight regularization out of layers to weights
and remove same code from the all the layers.
Loss and grads from weight regularization is done by the
weight itself.
Commit 2: [weight] Move apply gradient to weight
Move apply gradient which will just add the gradient
to the variable multiplied with the gradient.
Commit 3: [optimizer] Update to camelcase
Update apply_gradient(s) to applyGradient(s)
Resolves #762 #774
Signed-off-by: Parichay Kapoor<EMAIL_ADDRESS>

:memo: TAOS-CI Version: 1.4.20191203. Thank you for submitting PR #919. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://nnsuite.mooo.com/.

:memo: TAOS-CI Version: 1.4.20191203. Thank you for submitting PR #919. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://nnsuite.mooo.com/.

:octocat: cibot: @kparichay, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://nnsuite.mooo.com/nntrainer/ci/repo-workers/pr-checker/919-202102022000510.40650391578674-35d6fce5c4245e4ed0dafbe397c2f80a7543d6fa/.

:octocat: cibot: @kparichay, **** does not include doxygen tags such as @file @brief. You must include the doxygen tags in the source code at least.

:octocat: cibot: @kparichay, You wrote code with incorrect doxygen statements. Please check a doxygen rule at http://github.com/nnsuite/TAOS-CI/blob/master/ci/doc/doxygen-documentation.md

:octocat: cibot: @kparichay, One of the module of prebuild group is failed. If you want to get a hint to fix this issue, please go to https://github.com/nnstreamer/nntrainer/wiki/.


ENV Command

Example of the ENV command in a Dockerfile.
For Issue #29

Thank you very much! Glad to finally be part of the open source community!


Move numerical tests to separate Travis build stage

See #631.
The runtests function has changed a bit. The keyword tests has been changed to test_folders, and there are two new keywords:

exclude is an array of specific test files to exclude.
specific_tests is an array of specific test file to include. This overwrites all the other arguments and only tests the given files.

Currently, the only test treated separately is hmc.jl/matrix_support.jl, but this can be quickly updated by changing the numerical_tests variable in runtests.jl.

This should be good to merge once Appveyor passes.

Thanks, @cpfiffer!


FCREPO-3330 - Implement Want-Digest

JIRA Ticket: https://jira.lyrasis.org/browse/FCREPO-3330
What does this Pull Request do?

Implements Want-Digest functionality
Updates to the FixityService interface, removing some unused methods and adding the currently needed signature for getFixity

How should this be tested?
Fedora API Test Suite should now be passing for the 3.2.3 tests.
Interested parties
@fcrepo4/committers

All WANT-DIGEST test suite tests now pass.


Debian jessie base image is now oldstable

The documentation for this image falsely states that it builds on Debian stable, as it instead pulls in the oldstable (jessie) release next to the stable (stretch) release that is used by more recent images. Could the word ‘stable’ be removed from the documentation until it upgrades to stretch?

Some images on Docker Hub have separate Dockerfiles/images tagged with -stretch, if you don't want to break compatibility for people who use your image to do other things than just run OpenLDAP.

With merging pull request #25, this container now runs "stable" again. As there seem to be no compatibility issues, I don't see a point in creating different branches for different base image versions (yet).


fix: Prevent cached bootstrap data from leaking between users w/ same first/last name

SUMMARY

The caching mechanism for bootstrap data was not working as intended in the case where two users share the same first/last name.  The cache key was dependent on the User object passed. When a non-primitive object is used to generate a cache key, flask-caching calls repr() on it to determine the key: https://flask-caching.readthedocs.io/en/latest/index.html#memoization.
The User class comes from flask appbuilder and has the following __repr__:
def get_full_name(self):
    return "{0} {1}".format(self.first_name, self.last_name)

def __repr__(self):
    return self.get_full_name()

Therefore, when two users shared a first and last name, they would generate the same cache key and the cache could leak between the two users.  This was noticed because it would result in affected users seeing menu results they shouldn't see, though it potentially had other subtle effects as well, particularly for anyone adding a user-dependent COMMON_BOOTSTRAP_OVERRIDES_FUNC.
This PR fixes the issue by passing the actual user id to the memoized function so that it's included in the cache key.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

🏷 preset:2023.47


Improve coin configuration

The main `README.md that describes how to configure the LBRY coin didn't work for me. As the coin was not available in the initial coin list, I had to create it. I addition to the instructions given, you have to set the correct RPC port (14390) and switch the settings for

Has getinfo to "false" as lbrycrd-cli does no longer support this command and
Use segwit to "true" as is is required now (seeing Stratum errors like lbry dead lock, exiting... might indicate you missed this one).

I propose to edit the README and I will check whether it makes sense to create the coin using an SQL script that does it during the initial setup phase of the MySQL database.

I have to confirm whether I committed this to the repo or not, but I know that with LBC, the RPC calls being made don't properly include the segwit rules despite being set in the web panel.

Okay, I did: https://github.com/LBRYFoundation/pool/commit/f4276ad835433cfc2819d60b1c9e42c9dcd5cc89

getinfo is now supported again with lbcd. The feature is slated for removal, but at the moment it returns more accurate data than the work around yiimp implements if getinfo isn't supported.


Using fitVids on lazyloaded iframes

We found an issue when combining lazyloaded video iframes with the fitVids script.
We posted a solution in the fitVids' issue tracker:
https://github.com/davatron5000/FitVids.js/issues/258
https://gist.github.com/zitrusblau/47e9dbc58806b32bc6f1e7d869de638e
I wonder if replacing the data-src attribute once the iframe is lazyloaded wouldn't make sense in general?

Using fitvids is a really bad idea.

@aFarkas Thanks for your reply.
Any idea which approach is recommended to achieve responsive embedded videos then?

unfortunately a css-only solution can't take the aspect ratio into account...


Link for download of the version for Linux doesn't work

I'm try install the player in a Linux debian version, but the link in site (googleplaymusicdesktopplayer.com) doesn't work.
I'll be very grateful, if this is fixed.

Another one...
Do people not read the issue template?
#2118 #2117 #2103 #2124 #2125

I have checked that there are no issues with similar or the same content
YOU SHOULD CHECK CLOSED ISSUES ASWELL

^^ I mean, it's right there.... Please guys, the issue template is there so that I don't have to waste time managing duplicate issues.


MFC fickle when building on Phoenix-Slurm

Not really sure what's going on but builds on Phoenix seem somewhat broken.
Here are the modules
atl1-1-02-009-34-0: p-sbryngelson3-0/MFC $ module list

Currently Loaded Modules:
  1) pace-slurm/2022.06               7) bzip2/1.0.8-z5cmka   (H)  13) gettext/<IP_ADDRESS>-yz6qtc       (H)  19) xz/5.2.2-kbeci4       (H)
  2) zlib/1.2.7-s3gked          (H)   8) libmd/1.0.4-wdkbs3   (H)  14) libffi/3.4.2-bvfjil           (H)  20) libxml2/2.9.13-d4fgiv (H)
  3) nvhpc/22.11                      9) libbsd/0.11.5-j4ccxs (H)  15) sqlite/3.38.5-sweldt          (H)  21) cuda/11.6.0-u4jzhg
  4) ncurses/6.2-qhoz4g         (H)  10) expat/2.4.8-kng6xl        16) util-linux-uuid/2.36.2-6u5eni (H)
  5) openssl/1.0.2k-fips-xbtc42 (H)  11) readline/8.1-v3ivmo  (H)  17) python/3.9.12-rkxvr6
  6) cmake/3.23.1-327dbl             12) gdbm/1.19-54ea7n     (H)  18) libiconv/1.16-pbdcxj          (H)

and here is the error
  $ cmake -DMFC_SIMULATION=ON -Wno-dev -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_BUILD_TYPE=Release
-DCMAKE_PREFIX_PATH=/storage/coda1/p-sbryngelson3/0/sbryngelson3/MicroFC/build/install
-DCMAKE_FIND_ROOT_PATH=/storage/coda1/p-sbryngelson3/0/sbryngelson3/MicroFC/build/install
-DCMAKE_INSTALL_PREFIX=/storage/coda1/p-sbryngelson3/0/sbryngelson3/MicroFC/build/install -DMFC_MPI=ON -DMFC_OpenACC=OFF -S
/storage/coda1/p-sbryngelson3/0/sbryngelson3/MicroFC/ -B /storage/coda1/p-sbryngelson3/0/sbryngelson3/MicroFC/build/simulation

-- The C compiler identification is NVHPC 22.11.0
-- The Fortran compiler identification is NVHPC 22.11.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/local/pace-apps/manual/packages/nvhpc/Linux_x86_64/22.11/compilers/bin/nvc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /usr/local/pace-apps/manual/packages/nvhpc/Linux_x86_64/22.11/compilers/bin/nvfortran - skipped
-- Performing Test SUPPORTS_MARCH_NATIVE
-- Performing Test SUPPORTS_MARCH_NATIVE - Success
-- Enabled IPO / LTO
-- Found MPI_Fortran: /storage/pace-apps/manual/packages/nvhpc/Linux_x86_64/22.11/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi_usempif08.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: Fortran
-- Found CUDAToolkit: /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-4.8.5/cuda-11.7.0-7sdye3id7ahz34mzhyzzqbxowjxgxkhu/include (found version "11.7.64")
-- Looking for pthread.h
-- Looking for pthread.h - not found
CMake Error at /storage/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-4.8.5/cmake-3.23.1-327dblnbramviejdezocehqsujhu7yyg/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
  /storage/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-4.8.5/cmake-3.23.1-327dblnbramviejdezocehqsujhu7yyg/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /storage/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-4.8.5/cmake-3.23.1-327dblnbramviejdezocehqsujhu7yyg/share/cmake-3.23/Modules/FindThreads.cmake:238 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  /storage/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-4.8.5/cmake-3.23.1-327dblnbramviejdezocehqsujhu7yyg/share/cmake-3.23/Modules/FindCUDAToolkit.cmake:910 (find_package)
  CMakeLists.txt:259 (find_package)

-- Configuring incomplete, errors occurred!
See also "/storage/home/hcoda1/6/sbryngelson3/p-sbryngelson3-0/MicroFC/build/simulation/CMakeFiles/CMakeOutput.log".
See also "/storage/home/hcoda1/6/sbryngelson3/p-sbryngelson3-0/MicroFC/build/simulation/CMakeFiles/CMakeError.log".

This sometimes occurs for regular MFC, sometimes not.

this seems to be fixed now (though testing is still an issue)


1.1.1 policy, telemetry, and tracing are CrashLoopBackOff

Describe the bug
Three pods crashed after install.
$ kubectl get pods -n istio-system
NAME                                      READY     STATUS             RESTARTS   AGE
grafana-7b46bf6b7c-zjx89                  1/1       Running            0          7m
istio-citadel-5878d994cc-whcf9            1/1       Running            0          7m
istio-egressgateway-976f94bd-tg2ws        0/1       Running            0          7m
istio-galley-7855cc97dc-nlw67             1/1       Running            0          7m
istio-ingressgateway-794cfcf8bc-4b2zb     0/1       Running            0          7m
istio-init-crd-10-4qnxv                   0/1       Completed          0          7m
istio-init-crd-11-snwx6                   0/1       Completed          0          7m
istio-pilot-6dfbf958d5-67bbc              1/2       Running            0          7m
istio-policy-568c97b7c-j25nc              1/2       CrashLoopBackOff   7          7m
istio-sidecar-injector-59fc9d6f7d-2pgll   1/1       Running            0          7m
istio-telemetry-86c8478dcc-qwpkn          1/2       CrashLoopBackOff   7          7m
istio-tracing-75dd89b8b4-ppbxc            0/1       CrashLoopBackOff   5          7m
kiali-5d68f4c676-9b5nl                    1/1       Running            0          7m
prometheus-89bc5668c-bl7ml                1/1       Running            0          7m kubectl get pods -n istio-system
Expected behavior*
All pods are running without crash.
Steps to reproduce the bug
Install istio 1.1.1 by
helm install install/kubernetes/helm/istio --name istio --namespace istio-system
--values install/kubernetes/helm/istio/values-istio-demo.yaml
Version
1.1.1
Installation
helm install install/kubernetes/helm/istio --name istio --namespace istio-system
--values install/kubernetes/helm/istio/values-istio-demo.yaml
Environment
AWS EKS cluster
Cluster state
Both policy and telemetry have the following error.
 Container will be killed and recreated.
Warning  BackOff    4m (x59 over 18m)  kubelet, ip-192-168-85-158.ec2.internal  Back-off restarting failed container
Tracing has the following error.
Events:
Type     Reason     Age                 From                                     Message

Normal   Scheduled  22m                 default-scheduler                        Successfully assigned istio-system/istio-tracing-75dd89b8b4-ppbxc to ip-192-168-114-49.ec2.internal
Normal   Created    21m (x2 over 22m)   kubelet, ip-192-168-114-49.ec2.internal  Created container
Normal   Started    21m (x2 over 22m)   kubelet, ip-192-168-114-49.ec2.internal  Started container
Warning  Unhealthy  20m (x6 over 22m)   kubelet, ip-192-168-114-49.ec2.internal  Liveness probe failed: Get http://<IP_ADDRESS>:16686/: dial tcp <IP_ADDRESS>:16686: connect: connection refused
Warning  Unhealthy  20m (x10 over 22m)  kubelet, ip-192-168-114-49.ec2.internal  Readiness probe failed: Get http://<IP_ADDRESS>:16686/: dial tcp <IP_ADDRESS>:16686: connect: connection refused
Normal   Pulled     20m (x3 over 22m)   kubelet, ip-192-168-114-49.ec2.internal  Container image "docker.io/jaegertracing/all-in-one:1.9" already present on machine
Normal   Killing    17m (x5 over 21m)   kubelet, ip-192-168-114-49.ec2.internal  Killing container with id docker://jaeger:Container failed liveness probe.. Container will be killed and recreated.
Warning  BackOff    2m (x51 over 16m)   kubelet, ip-192-168-114-49.ec2.internal  Back-off restarting failed container

bug.tar.gz

It looks like Mixer is having problems connecting to Galley. Can you add Galley logs as well?

I will repro it later. My 1.1.1 pods were changed to 1.0.6 for stability.

I don't see the similar issues with 1.1.2 pods.

i am facing same issues with 1.1.1/1.1.2 and 1.1.3 , first pilot was not up, now telemetry is not up.  However Policy is up/
***kubectl describe pod -n istio-system istio-telemetry-5f78bf78b7-btcp9 ***
Events:
Type     Reason     Age                     From               Message

Warning  Unhealthy  20m (x145 over 145m)    kubelet, tushar-5  Liveness probe failed: Get http://<IP_ADDRESS>:15014/version: dial tcp <IP_ADDRESS>
:15014: connect: connection refused
Warning  BackOff    4m57s (x608 over 144m)  kubelet, tushar-5  Back-off restarting failed container

I have the same issue here with 1.1.7.. I don't understand why this issue has been closed. Is there a resultion for this issue?
NAME                                      READY   STATUS             RESTARTS   AGE
istio-citadel-66d49b64fc-9stng            1/1     Running            0          12m
istio-galley-676599ffb4-wr8jw             1/1     Running            0          12m
istio-ingressgateway-67bbd58f49-b8wms     0/1     Running            0          11m
istio-ingressgateway-67bbd58f49-kvxh8     0/1     Running            0          12m
istio-init-crd-10-xzh9s                   0/1     Completed          0          12m
istio-init-crd-11-7twdq                   0/1     Completed          0          12m
istio-pilot-6bb4cc6cc6-8nbxn              2/2     Running            0          29d
istio-pilot-85bc7d8b45-dcf58              0/2     Pending            0          12m
istio-policy-65868677c8-bf27z             1/2     CrashLoopBackOff   6          6m55s
istio-policy-65868677c8-cmm2z             1/2     CrashLoopBackOff   7          12m
istio-sidecar-injector-779544894b-7cb55   1/1     Running            0          12m
istio-telemetry-69865bff9f-nlzgn          2/2     Running            0          29d
istio-telemetry-f84b7476d-hv78g           0/2     Pending            0          12m
prometheus-5fffdf8848-gfk2s               1/1     Running            0          12m

When I call describe, I get:
Warning  Unhealthy  6m39s (x7 over 7m19s)  kubelet, ip-10-1-8-121.eu-central-1.compute.internal  Liveness probe failed: Get http://<IP_ADDRESS>:15014/version: dial tcp <IP_ADDRESS>:15014: connect: connection refused

Seeing this as well in 1.1.7.  We have automation to create a number of GKE clusters and install Istio in exactly the same way daily, and most come up without issue.  This has been ongoing in the bad cluster for hours with no other signs of network issues.  If I port-forward to the istio-galley service port 443, I can curl the endpoint, although I get a 404 on /.
NAME                                      READY   STATUS             RESTARTS   AGE
grafana-7f4d444dd5-qdfh4                  1/1     Running            0          3h32m
istio-citadel-7f447d4d4b-l4746            1/1     Running            0          3h32m
istio-cleanup-secrets-1.1.7-5x2vk         0/1     Completed          0          3h32m
istio-egressgateway-745f5c9494-hs5qx      0/1     Running            0          3h32m
istio-galley-84749d54b7-2fs6d             1/1     Running            0          32m
istio-grafana-post-install-1.1.7-j29hz    0/1     Completed          0          3h32m
istio-ingressgateway-6b79f895d6-dgmhm     0/1     Running            0          3h32m
istio-ingressgateway-6b79f895d6-nf6wh     0/1     Running            0          3h32m
istio-init-crd-10-qgx9z                   0/1     Completed          0          3h33m
istio-init-crd-11-c6sw7                   0/1     Completed          0          3h33m
istio-pilot-76899788b6-hg58d              1/2     Running            0          36m
istio-pilot-76899788b6-zc2cw              1/2     Running            0          31m
istio-policy-578bcb878f-bqxtt             1/2     CrashLoopBackOff   17         36m
istio-policy-578bcb878f-lnkfl             1/2     CrashLoopBackOff   15         31m
istio-security-post-install-1.1.7-2tqhw   0/1     Completed          0          3h32m
istio-sidecar-injector-6895997989-d2vnm   1/1     Running            0          3h32m
istio-telemetry-5448cbd995-fvmpl          1/2     CrashLoopBackOff   79         3h32m
istio-telemetry-5448cbd995-zslg7          1/2     CrashLoopBackOff   15         31m
istio-tracing-79db5954f-lvcrx             0/1     CrashLoopBackOff   63         3h32m
kiali-68677d47d7-ltj98                    1/1     Running            0          3h32m
prometheus-5977597c75-mnk58               1/1     Running            0          3h32m

(I killed a bunch of pods in case the original issue had been a transient network issue, as those are unfortunately not that rare in GCE.)
pilot.log.gz
mixer.log.gz
galley.log.gz

Seeing this in 1.1.8.
$ kubectl get pods -n istio-system
NAME                                      READY   STATUS             RESTARTS   AGE
grafana-67c69bb567-c58jb                  1/1     Running            0          42m
istio-citadel-58fc8bfb55-6lc5h            1/1     Running            0          42m
istio-cleanup-secrets-1.1.8-8f8xc         0/1     Completed          0          42m
istio-egressgateway-c57fcdcdd-9kgsm       0/1     Running            0          42m
istio-galley-75bcd59768-cfndt             1/1     Running            0          42m
istio-grafana-post-install-1.1.8-mj4b9    0/1     Completed          0          42m
istio-ingressgateway-784966d549-6j5nh     0/1     Running            0          42m
istio-pilot-754d4dcc94-cwdhw              1/2     Running            0          42m
istio-policy-7d985ff57d-ksf7w             1/2     CrashLoopBackOff   15         42m
istio-security-post-install-1.1.8-r5lqd   0/1     Completed          0          42m
istio-sidecar-injector-6fd47fb7c8-zlksg   1/1     Running            0          42m
istio-telemetry-78dfdd8c4b-dgcfr          1/2     CrashLoopBackOff   15         42m
istio-tracing-5d8f57c8ff-h2drz            1/1     Running            0          42m
kiali-d4d886dd7-7cpgl                     1/1     Running            0          42m
prometheus-d8d46c5b5-tv6q8                1/1     Running            0          42m

From the logs I found this:
2019-06-09T13:32:01.814673Z	info	Using new MCP client sink stack
2019-06-09T13:32:01.814702Z	info	Awaiting for config store sync...
2019-06-09T13:32:01.814790Z	info	ccResolverWrapper: sending new addresses to cc: [{istio-galley.istio-system.svc:9901 0  }]
2019-06-09T13:32:01.814798Z	info	ClientConn switching balancer to "pick_first"
2019-06-09T13:32:01.814822Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420a64970, CONNECTING
2019-06-09T13:32:01.908457Z	info	mcp	(re)trying to establish new MCP sink stream
2019-06-09T13:32:16.817783Z	info	grpc: addrConn.createTransport failed to connect to {istio-galley.istio-system.svc:9901 0  }. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup istio-galley.istio-system.svc on <IP_ADDRESS>:53: read udp <IP_ADDRESS>:56154-><IP_ADDRESS>:53: read: connection refused". Reconnecting...
2019-06-09T13:32:16.817933Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420a64970, TRANSIENT_FAILURE
2019-06-09T13:32:16.818004Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420a64970, CONNECTING
Seems mixer can't connect to Galley ?

@kbruner, it looks like you have attached proxy logs for pilot and mixer (these are the logs from the "istio-proxy" container).  Can you attach logs for the pilot and mixer containers themselves?

@therealmitchconnors I think I dumped the logs for all the containers together for each pod (kubectl logs --all-containers ...).  Sorry, I've long since torn down the broken cluster.

@kbruner I can confirm from the logs you provided that Pilot is unable to reach Kube DNS to lookup Galley.  Did you have any other workloads using Kube DNS at the time?
Here is the offending log line, which is repeated throughout the pilot logs:
transport: Error while dialing dial tcp: lookup istio-galley.istio-system.svc on <IP_ADDRESS>:53: read udp <IP_ADDRESS>:39769-><IP_ADDRESS>:53: i/o timeout

@therealmitchconnors No, the istio deployment was done with automation, the exact same way it's done multiple times a day: same number of nodes, same node size.  No other workloads are deployed to the cluster before Istio.  It does use the current GKE default version, so that can vary depending on day and region.  kube-dns is almost completely unconfigurable in GKE, so there's nothing under our control there.
I wasn't seeing any DNS issues in the cluster when I went to debug half an hour later.  Even if there are timeouts, shouldn't pilot/galley/etc. recover if kube-dns becomes available again?

Yes, it should, though how quickly will depend on how far into the CrashloopBackoff period you are...  I didn't see any evidence that it was ever able to connect to Kube DNS in the logs.  The log line above was repeated over a thousand times over the 20 minutes that the log covers...
It is still possible that the lack of DNS connectivity was caused by the istio-proxy intercepting the DNS queries and not routing them properly, but without a cluster to debug, this is speculative.

Is there any workaround for this issue yet or root cause for "connection refused"?

We are running istio version 1.2.0 on my on-permises cluster.
Cluster is setup using GKE (kubernetes version 1.13.4)

I have the same problem with istio 1.2.2 on K3S in a fresh install
kubectl get pods --all-namespaces
says
NAMESPACE      NAME                                      READY   STATUS             RESTARTS   AGE
istio-system   istio-citadel-68c85b6684-2xcrk            1/1     Running            0          36m
istio-system   istio-galley-77d697957f-rtpsr             1/1     Running            0          36m
istio-system   istio-ingressgateway-8b858ff84-ddhht      0/1     Running            0          36m
istio-system   istio-init-crd-10-4ln7g                   0/1     Completed          0          10h
istio-system   istio-init-crd-11-4v497                   0/1     Completed          0          10h
istio-system   istio-init-crd-12-wl4gs                   0/1     Completed          0          10h
istio-system   istio-pilot-5544b58bb6-nzn58              1/2     Running            0          36m
istio-system   istio-policy-5f9cf6df57-kjltg             1/2     CrashLoopBackOff   15         36m
istio-system   istio-sidecar-injector-66549495d8-4n2cd   1/1     Running            0          36m
istio-system   istio-telemetry-7749c6d54f-xrm8t          1/2     CrashLoopBackOff   14         36m
istio-system   prometheus-776fdf7479-xmqfd               1/1     Running            0          36m
istio-system   svclb-istio-ingressgateway-4hf8l          0/9     Pending            0          36m
istio-system   svclb-istio-ingressgateway-h5jsj          0/9     Pending            0          36m
kube-system    coredns-695688789-42qpk                   1/1     Running            25         12d
kube-system    helm-install-traefik-p8r96                0/1     Completed          0          12d
kube-system    istio-cni-node-2tq6v                      1/1     Running            1          45m
kube-system    istio-cni-node-l855h                      1/1     Running            1          45m
kube-system    kubernetes-dashboard-5f7b999d65-jf2f4     1/1     Running            4          2d
kube-system    svclb-traefik-flbwx                       2/2     Running            44         11d
kube-system    svclb-traefik-md27d                       2/2     Running            50         12d
kube-system    tiller-deploy-57ff77d846-24fhx            1/1     Running            1          10h
kube-system    traefik-55bd9646fc-8z59n                  1/1     Running            25         12d
sysop@hoseplak3s:~/software/istio-1.2.2$

the same problem

I have solved the problem and I found it to be a network problem with my k8s.
[root@k8s-master istio-1.2.2]# kubectl  get pod -n istio-system
NAME                                      READY   STATUS      RESTARTS   AGE
istio-citadel-68c85b6684-hbtcg            1/1     Running     0          32m
istio-cleanup-secrets-1.2.2-tjhln         0/1     Completed   0          32m
istio-galley-77d697957f-s6hps             1/1     Running     0          32m
istio-ingressgateway-8b858ff84-nqn24      1/1     Running     0          32m
istio-init-crd-10-nlssx                   0/1     Completed   0          33m
istio-init-crd-11-9jn4z                   0/1     Completed   0          33m
istio-init-crd-12-5d9bz                   0/1     Completed   0          33m
istio-pilot-5544b58bb6-mnh6w              2/2     Running     0          32m
istio-policy-5f9cf6df57-gpqrw             2/2     Running     4          32m
istio-security-post-install-1.2.2-ztfcr   0/1     Completed   0          32m
istio-sidecar-injector-66549495d8-l2fxw   1/1     Running     0          32m
istio-telemetry-7749c6d54f-kzjbx          2/2     Running     4          32m
prometheus-776fdf7479-6q2wp               1/1     Running     0          32m

Solved.
It was a network problem on the host machine.

@MirtoBusico I'm getting the same issues what kind of networking issues was it?

kubectl logs istio-telemetry-75cffdc74b-4sqql -n istio-system istio-proxy
GC forced
gc 431 @51452.448s 0%: 0.020+82+79 ms clock, 0.020+0/20/61+79 ms cpu, 133->133->124 MB, 248 MB goal, 1 P
scvg342: 0 MB released
scvg342: inuse: 132, idle: 5, sys: 138, released: 1, consumed: 136 (MB)
GC forced
gc 432 @51572.647s 0%: 0.018+75+179 ms clock, 0.018+0/19/55+179 ms cpu, 131->131->124 MB, 248 MB goal, 1 P
2019-08-02T21:41:19.658953Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T21:41:19.661438Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb690, TRANSIENT_FAILURE
2019-08-02T21:41:19.661750Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb690, CONNECTING
2019-08-02T21:41:19.677381Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb690, READY
scvg343: inuse: 133, idle: 5, sys: 138, released: 1, consumed: 136 (MB)
2019-08-02T21:41:26.437712Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T21:41:26.438029Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deaaf0, TRANSIENT_FAILURE
2019-08-02T21:41:26.438041Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deaaf0, CONNECTING
2019-08-02T21:41:26.459487Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deaaf0, READY
2019-08-02T21:41:36.886098Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T21:41:36.886782Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8ac0, TRANSIENT_FAILURE
2019-08-02T21:41:36.886909Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8ac0, CONNECTING
2019-08-02T21:41:36.910656Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8ac0, READY
2019-08-02T21:41:39.934928Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T21:41:39.935705Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb8a0, TRANSIENT_FAILURE
2019-08-02T21:41:39.935884Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb8a0, CONNECTING
2019-08-02T21:41:39.944429Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T21:41:39.945047Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8980, TRANSIENT_FAILURE
2019-08-02T21:41:39.945210Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8980, CONNECTING
2019-08-02T21:41:39.952369Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb8a0, READY
2019-08-02T21:41:39.960708Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T21:41:39.961309Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420dd72a0, TRANSIENT_FAILURE
2019-08-02T21:41:39.961467Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420dd72a0, CONNECTING
2019-08-02T21:41:39.975389Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420dd72a0, READY
2019-08-02T21:41:39.983991Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8980, READY
GC forced
gc 433 @51693.695s 0%: 1.6+187+100 ms clock, 1.6+0.48/58/128+100 ms cpu, 133->133->124 MB, 248 MB goal, 1 P
scvg344: 0 MB released
scvg344: inuse: 133, idle: 4, sys: 138, released: 2, consumed: 136 (MB)
GC forced
gc 434 @51814.047s 0%: 0.016+3794+1997 ms clock, 0.016+0/1697/2084+1997 ms cpu, 131->132->124 MB, 248 MB goal, 1 P
2019-08-02T21:45:19.690138Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T21:45:19.690926Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb690, TRANSIENT_FAILURE
2019-08-02T21:45:19.691171Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb690, CONNECTING
2019-08-02T21:45:19.705383Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb690, READY
2019-08-02T21:45:26.461890Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T21:45:26.462477Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deaaf0, TRANSIENT_FAILURE
2019-08-02T21:45:26.462632Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deaaf0, CONNECTING
2019-08-02T21:45:26.477995Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deaaf0, READY
2019-08-02T21:45:37.013994Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T21:45:37.014400Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8ac0, TRANSIENT_FAILURE
2019-08-02T21:45:37.014415Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8ac0, CONNECTING
2019-08-02T21:45:37.054994Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8ac0, READY
2019-08-02T21:45:39.954681Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T21:45:39.955251Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb8a0, TRANSIENT_FAILURE
2019-08-02T21:45:39.955388Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb8a0, CONNECTING
2019-08-02T21:45:39.968086Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb8a0, READY
2019-08-02T21:45:39.976182Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T21:45:39.976800Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420dd72a0, TRANSIENT_FAILURE
2019-08-02T21:45:39.976953Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420dd72a0, CONNECTING
2019-08-02T21:45:39.990113Z	info	transport: loopyWriter.run returning.   Using the inClusterConfig.  This might not work.
2019-08-02T07:20:56.609657Z	info	Built new config.Snapshot: id='0'
2019-08-02T07:20:56.609880Z	info	Cleaning up handler table, with config ID:-1
gc 1 @0.044s 24%: 0.013+17+10 ms clock, 0.013+1.9/5.0/0.31+10 ms cpu, 4->5->3 MB, 5 MB goal, 1 P
2019-08-02T07:20:56.644895Z	warn	Failed to discover kinds: [dogstatsd cloudwatch], start retry in background
2019-08-02T07:20:56.645031Z	info	Built new config.Snapshot: id='1'
2019-08-02T07:20:56.645079Z	info	Cleaning up handler table, with config ID:0
2019-08-02T07:20:56.645085Z	info	Awaiting for config store sync...
gc 2 @0.113s 33%: 0.015+32+25 ms clock, 0.015+7.7/6.5/9.4+25 ms cpu, 5->5->4 MB, 6 MB goal, 1 P
2019-08-02T07:20:57.679336Z	info	Publishing 3 events
2019-08-02T07:20:57.679890Z	error	failed to evaluate expression for field 'TraceId': unknown attribute request.headers
2019-08-02T07:20:57.680063Z	error	error evaluating AttributeBinding expression for attribute key 'destination.owner': unknown attribute destination.owner
2019-08-02T07:20:57.680270Z	info	Built new config.Snapshot: id='2'
2019-08-02T07:20:57.680496Z	info	Cleaning up handler table, with config ID:1
2019-08-02T07:20:58.850815Z	info	Publishing 4 events
2019-08-02T07:20:58.852607Z	info	Built new config.Snapshot: id='3'
2019-08-02T07:20:58.852924Z	info	Cleaning up handler table, with config ID:2
2019-08-02T07:21:01.263098Z	info	Publishing 10 events
2019-08-02T07:21:02.779506Z	error	Instance not found: instance='server-request-count.metric'
2019-08-02T07:21:02.786930Z	error	Instance not found: instance='server-request-bytes.metric'
2019-08-02T07:21:02.787518Z	error	Instance not found: instance='server-response-bytes.metric'
2019-08-02T07:21:02.787979Z	error	Instance not found: instance='server-response-latencies.metric'
2019-08-02T07:21:02.817703Z	error	No valid instances found
2019-08-02T07:21:02.818822Z	error	No valid actions found in rule
2019-08-02T07:21:02.819242Z	error	Instance not found: instance='server-received-bytes-count.metric'
2019-08-02T07:21:02.819338Z	error	Instance not found: instance='server-sent-bytes-count.metric'
2019-08-02T07:21:02.819441Z	error	No valid instances found
2019-08-02T07:21:02.820303Z	error	No valid actions found in rule
2019-08-02T07:21:02.820462Z	error	Instance not found: instance='client-request-count.metric'
2019-08-02T07:21:02.820610Z	error	Instance not found: instance='client-request-bytes.metric'
2019-08-02T07:21:02.820697Z	error	Instance not found: instance='client-response-bytes.metric'
2019-08-02T07:21:02.820792Z	error	Instance not found: instance='client-roundtrip-latencies.metric'
2019-08-02T07:21:02.820895Z	error	No valid instances found
2019-08-02T07:21:02.820970Z	error	No valid actions found in rule
2019-08-02T07:21:02.821111Z	error	Instance not found: instance='mixer-request-latency.metric'
2019-08-02T07:21:02.821200Z	error	Instance not found: instance='mixer-request-count.metric'
2019-08-02T07:21:02.821281Z	error	No valid instances found
2019-08-02T07:21:02.821370Z	error	No valid actions found in rule
2019-08-02T07:21:02.821733Z	error	Instance not found: instance='client-received-bytes-count.metric'
2019-08-02T07:21:02.822421Z	error	Instance not found: instance='client-sent-bytes-count.metric'
2019-08-02T07:21:02.822514Z	error	No valid instances found
2019-08-02T07:21:02.822586Z	error	No valid actions found in rule
2019-08-02T07:21:02.822675Z	error	Handler not found: handler='handler.kubernetesenv'
2019-08-02T07:21:02.822793Z	error	No valid actions found in rule
2019-08-02T07:21:02.822881Z	error	Handler not found: handler='handler.kubernetesenv'
2019-08-02T07:21:02.822959Z	error	No valid actions found in rule
2019-08-02T07:21:02.823003Z	info	Built new config.Snapshot: id='4'
gc 3 @6.237s 2%: 0.010+62+35 ms clock, 0.010+25/10/0+35 ms cpu, 7->8->5 MB, 8 MB goal, 1 P
gc 4 @6.415s 5%: 0.008+616+114 ms clock, 0.008+19/110/0+114 ms cpu, 74->121->120 MB, 75 MB goal, 1 P
2019-08-02T07:21:03.704450Z	info	Starting monitor server...
Istio Mixer<EMAIL_ADDRESS>Starting gRPC server on port 9091
2019-08-02T07:21:03.707740Z	info	ControlZ available at <IP_ADDRESS>:9876
2019-08-02T07:21:03.712957Z	info	parsed scheme: ""
2019-08-02T07:21:03.713139Z	info	scheme "" not registered, fallback to default scheme
2019-08-02T07:21:03.713496Z	info	ccResolverWrapper: sending new addresses to cc: [{monitoring.googleapis.com:443 0  <nil>}]
2019-08-02T07:21:03.713603Z	info	ClientConn switching balancer to "pick_first"
2019-08-02T07:21:03.713796Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc4204266e0, CONNECTING
2019-08-02T07:21:03.715093Z	info	parsed scheme: ""
2019-08-02T07:21:03.715256Z	info	scheme "" not registered, fallback to default scheme
2019-08-02T07:21:03.715400Z	info	parsed scheme: ""
2019-08-02T07:21:03.715478Z	info	scheme "" not registered, fallback to default scheme
2019-08-02T07:21:03.715706Z	info	ccResolverWrapper: sending new addresses to cc: [{logging.googleapis.com:443 0  <nil>}]
2019-08-02T07:21:03.715886Z	info	ClientConn switching balancer to "pick_first"
2019-08-02T07:21:03.715982Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc4204276c0, CONNECTING
2019-08-02T07:21:03.716252Z	info	ccResolverWrapper: sending new addresses to cc: [{logging.googleapis.com:443 0  <nil>}]
2019-08-02T07:21:03.716340Z	info	ClientConn switching balancer to "pick_first"
2019-08-02T07:21:03.716505Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420427850, CONNECTING
2019-08-02T07:21:03.721403Z	info	parsed scheme: ""
2019-08-02T07:21:03.721575Z	info	scheme "" not registered, fallback to default scheme
2019-08-02T07:21:03.721861Z	info	parsed scheme: ""
2019-08-02T07:21:03.722005Z	info	scheme "" not registered, fallback to default scheme
2019-08-02T07:21:03.722236Z	info	parsed scheme: ""
2019-08-02T07:21:03.722336Z	info	scheme "" not registered, fallback to default scheme
2019-08-02T07:21:03.725200Z	info	Cleaning up handler table, with config ID:3
2019-08-02T07:21:03.727529Z	info	ccResolverWrapper: sending new addresses to cc: [{monitoring.googleapis.com:443 0  <nil>}]
2019-08-02T07:21:03.727687Z	info	ClientConn switching balancer to "pick_first"
2019-08-02T07:21:03.728019Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8980, CONNECTING
2019-08-02T07:21:03.728431Z	info	ccResolverWrapper: sending new addresses to cc: [{cloudtrace.googleapis.com:443 0  <nil>}]
2019-08-02T07:21:03.728514Z	info	ClientConn switching balancer to "pick_first"
2019-08-02T07:21:03.728693Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8ac0, CONNECTING
2019-08-02T07:21:03.729900Z	info	ccResolverWrapper: sending new addresses to cc: [{contextgraph.googleapis.com:443 0  <nil>}]
2019-08-02T07:21:03.730027Z	info	ClientConn switching balancer to "pick_first"
2019-08-02T07:21:03.730340Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8c60, CONNECTING
2019-08-02T07:21:03.829461Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc4204276c0, READY
2019-08-02T07:21:03.831815Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8c60, READY
2019-08-02T07:21:03.832443Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc4204266e0, READY
2019-08-02T07:21:03.833721Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8980, READY
2019-08-02T07:21:03.834086Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420427850, READY
2019-08-02T07:21:03.838988Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8ac0, READY
gc 5 @7.163s 6%: 0.009+145+60 ms clock, 0.009+7.9/38/21+60 ms cpu, 121->128->127 MB, 240 MB goal, 1 P
2019-08-02T07:21:05.975959Z	info	Publishing 15 events
2019-08-02T07:21:05.985738Z	info	Built new config.Snapshot: id='5'
2019-08-02T07:21:05.992830Z	info	parsed scheme: ""
2019-08-02T07:21:05.993052Z	info	scheme "" not registered, fallback to default scheme
2019-08-02T07:21:05.993423Z	info	ccResolverWrapper: sending new addresses to cc: [{monitoring.googleapis.com:443 0  <nil>}]
2019-08-02T07:21:05.993555Z	info	ClientConn switching balancer to "pick_first"
2019-08-02T07:21:05.993708Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deaaf0, CONNECTING
2019-08-02T07:21:05.995282Z	info	parsed scheme: ""
2019-08-02T07:21:05.995422Z	info	scheme "" not registered, fallback to default scheme
2019-08-02T07:21:05.995533Z	info	parsed scheme: ""
2019-08-02T07:21:05.995601Z	info	scheme "" not registered, fallback to default scheme
2019-08-02T07:21:05.995883Z	info	ccResolverWrapper: sending new addresses to cc: [{logging.googleapis.com:443 0  <nil>}]
2019-08-02T07:21:05.997654Z	info	ClientConn switching balancer to "pick_first"
2019-08-02T07:21:05.997861Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb690, CONNECTING
2019-08-02T07:21:05.998178Z	info	ccResolverWrapper: sending new addresses to cc: [{logging.googleapis.com:443 0  <nil>}]
2019-08-02T07:21:05.998289Z	info	ClientConn switching balancer to "pick_first"
2019-08-02T07:21:05.998377Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb8a0, CONNECTING
2019-08-02T07:21:06.000319Z	info	parsed scheme: ""
2019-08-02T07:21:06.000425Z	info	scheme "" not registered, fallback to default scheme
2019-08-02T07:21:06.000650Z	info	adapters	getting kubeconfig from: ""	{"adapter": "handler.kubernetesenv.istio-system"}
2019-08-02T07:21:06.000723Z	warn	Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2019-08-02T07:21:06.002466Z	info	adapters	Waiting for kubernetes cache sync...	{"adapter": "handler.kubernetesenv.istio-system"}
2019-08-02T07:21:06.002651Z	info	ccResolverWrapper: sending new addresses to cc: [{contextgraph.googleapis.com:443 0  <nil>}]
2019-08-02T07:21:06.002727Z	info	ClientConn switching balancer to "pick_first"
2019-08-02T07:21:06.002832Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420dd72a0, CONNECTING
2019-08-02T07:21:06.116190Z	info	adapters	Cache sync successful.	{"adapter": "handler.kubernetesenv.istio-system"}
2019-08-02T07:21:06.134228Z	info	Cleaning up handler table, with config ID:4
2019-08-02T07:21:06.134295Z	info	adapters	Sending last data before shutting down	{"adapter": "handler.stackdriver.istio-system"}
2019-08-02T07:21:06.137588Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T07:21:06.137701Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T07:21:06.137999Z	error	adapters	adapter did not close all the scheduled daemons	{"adapter": "handler.stackdriver.istio-system"}
2019-08-02T07:21:06.138086Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T07:21:06.138159Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T07:21:06.138284Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deaaf0, READY
2019-08-02T07:21:06.140575Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420dd72a0, READY
2019-08-02T07:21:06.142049Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb8a0, READY
2019-08-02T07:21:06.142383Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb690, READY
GC forced
gc 6 @127.447s 1%: 0.84+2169+99 ms clock, 0.84+114/1332/308+99 ms cpu, 139->139->124 MB, 244 MB goal, 1 P
scvg0: inuse: 131, idle: 8, sys: 140, released: 0, consumed: 140 (MB)
2019-08-02T07:25:03.861299Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T07:25:03.862052Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8980, TRANSIENT_FAILURE
2019-08-02T07:25:03.862210Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8980, CONNECTING
2019-08-02T07:25:03.878405Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8980, READY
2019-08-02T07:25:03.888804Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T07:25:03.889483Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8ac0, TRANSIENT_FAILURE
2019-08-02T07:25:03.889644Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8ac0, CONNECTING
2019-08-02T07:25:03.904648Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420ba8ac0, READY
2019-08-02T07:25:06.141794Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T07:25:06.142703Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420dd72a0, TRANSIENT_FAILURE
2019-08-02T07:25:06.143433Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420dd72a0, CONNECTING
2019-08-02T07:25:06.145633Z	info	transport: loopyWriter.run returning. Err: connection error: desc = "transport is closing"
2019-08-02T07:25:06.146147Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb8a0, TRANSIENT_FAILURE
2019-08-02T07:25:06.146325Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb8a0, CONNECTING
2019-08-02T07:25:06.161945Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420dd72a0, READY
2019-08-02T07:25:06.162414Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc420deb8a0, READY
GC forced
gc 7 @249.748s 0%: 0.021+196+91 ms clock, 0.021+0/51/144+91 ms cpu, 133->133->124 MB, 248 MB goal, 1 P
scvg1: inuse: 132, idle: 7, sys: 140, released: 0, consumed: 140 (MB)

@MirtoBusico I'm getting the same issues what kind of networking issues was it?

Well it was a wrong hostname.
in /etc/hosts I put the external host address with the name "k3s-master"
But in some commands I used the name "k3s-server" which don't have any addrress.
Verifying that

hostname gives the name you expected (eg myhost)
/etc/hosts contains a "x.x.x.x myhost" line
/etc/hosts DO NOT contain any line "<IP_ADDRESS> myhost"
in every command and configuration file the host is indicated by "myhost"

the installation was successful

I have solved the problem and I found it to be a network problem with my k8s.
[root@k8s-master istio-1.2.2]# kubectl  get pod -n istio-system
NAME                                      READY   STATUS      RESTARTS   AGE
istio-citadel-68c85b6684-hbtcg            1/1     Running     0          32m
istio-cleanup-secrets-1.2.2-tjhln         0/1     Completed   0          32m
istio-galley-77d697957f-s6hps             1/1     Running     0          32m
istio-ingressgateway-8b858ff84-nqn24      1/1     Running     0          32m
istio-init-crd-10-nlssx                   0/1     Completed   0          33m
istio-init-crd-11-9jn4z                   0/1     Completed   0          33m
istio-init-crd-12-5d9bz                   0/1     Completed   0          33m
istio-pilot-5544b58bb6-mnh6w              2/2     Running     0          32m
istio-policy-5f9cf6df57-gpqrw             2/2     Running     4          32m
istio-security-post-install-1.2.2-ztfcr   0/1     Completed   0          32m
istio-sidecar-injector-66549495d8-l2fxw   1/1     Running     0          32m
istio-telemetry-7749c6d54f-kzjbx          2/2     Running     4          32m
prometheus-776fdf7479-6q2wp               1/1     Running     0          32m

I'm having the same issue. How did you solve it?

I am seeing a similar problem running in Kind.  The policy and telemetry pods are in CrashLoopBackoff

@jarvisuser90 @jmjava would you be willing to open separate issues to track the behavior you are seeing? Also, it would be great to have logs from the policy and telemetry pods, as well as the context of what platform you are running on, etc.

@therealmitchconnors My issue was related to my flannel pod network not being set up correctly.

I encountered the same issue and it was caused by dns problem. kube-dns service was deleted (I didnot know how), the issue was resolved after I recreated the service.
[root@admin ~]# kubectl get po -n istio-system
NAME                                      READY   STATUS    RESTARTS   AGE
grafana-7f4d444dd5-q6kb7                  1/1     Running   1          3h2m
istio-citadel-79875c6f4f-stcf5            1/1     Running   1          3h2m
istio-egressgateway-84756c5c5b-zrjdw      1/1     Running   1          3h2m
istio-galley-684d78bff8-bg78m             1/1     Running   1          3h2m
istio-ingressgateway-56f7bb484c-vknbn     1/1     Running   1          3h2m
istio-pilot-5bc798bf4b-ksdr5              2/2     Running   2          3h2m
istio-pilot-5bc798bf4b-zjddf              2/2     Running   0          4m14s
istio-policy-7fbf6fb5c6-9s22m             2/2     Running   16         39m
istio-sidecar-injector-57d99ff88b-msx4q   1/1     Running   1          3h2m
istio-telemetry-56d45b77fd-kqj2m          2/2     Running   18         41m
istio-tracing-79db5954f-hjp62             1/1     Running   55         3h2m
kiali-68677d47d7-xgmq6                    1/1     Running   1          3h2m
prometheus-5977597c75-pvt4p               1/1     Running   1          3h2m

Guy, I had similar issue and root cause was CoreDNS issue. So DNS issues are causing failed pods.

Seems like this can be closed?  The issue is DNS related, as I understand it.

Yes this has been resolved for me, confirmed DNS issue


Language-specific guides request

I'd like to request PHP, specifically Laravel. Thanks.

Thanks for requesting; let me close this as duplicate of  https://github.com/docker/docker.github.io/issues/12719


RC npm: install scripts are common attack vectors

It should probably be noted in the doc that install scripts are often dangerous as they can allow for the running of arbitrary code at installation time. This has been used in the past to try and steal credentials (mostly related to crypto wallets and the like.)
Should suggest in most cases applying: npm --ignore-scripts when possible.

Well, when possible. There are valid use cases but I also think in general running unknown arbitrary scripts as part of installation should be discouraged, unless there's a way to sandbox it.
I would be also curious to know if there's any effort from npm side to support certain common use cases just in a safer way. Are there any specific use cases that come to mind where you would 100% want to run arbitrary scripts as part of installation?

I don't think anyone would dispute that install scripts "should be" discouraged.
That, however, is irrelevant - they're currently the only way to accomplish a bunch of valid use cases. As such, until there's a better alternative for these, it would be harmful to recommend ignoring scripts.
Valid use cases include "compile something after install", "download an artifact tailored to the system on install", etc.

I'm tending towards security by default here and with recommending the use of various methods in which --ignore-scripts comes into play, whether that's via command line, or through the .npmrc config.

So we all agree that scripts are a potential source of attacks.
Given that a lot of things will be breaking if users use the --ignore-scripts, how about the following changes:

In the "Dependency management > Intake" section, we could suggest doing a code review, and explicitly mention the scripts as part of the "code". (most uses won't code review, but if they do they will take a look at the script as well)

We can add a note in the same section that there exists a --ignore-scripts for users who want to impose a policy on scripts; warn that because there are genuine use cases it is likely to break; and conclude that for this reason so we do not recommend it unless users verify themselves that none of their dependencies need it.

Wdut?

Do we have any data as supporting evidence for the statement that "given that a lot of things will be breaking if users use --ignore-scripts" ?

I personally don't; but I suppose listing all the packages that use scripts (possible with a combination of dependency graph desp.dev) could provide a less hanwavvy answer.
@oliverchang @calebbrown is this data we could pull out fr the package analysis project?

Anything that compiles on install, or downloads assets on install, won't work when ignoring scripts - which includes electron, sass, and a bunch of others.

@lirantal is this convincing enough? Note that we're not ruling out, we're just saying developers may use it but at their own risk. If there's a better way to express it, please let us know.

Just a small note that this and #20 are very similar so maybe we consolidate this discussion on one of them. @laurentsimon with regards to the above, if we're saying developers may use at their own risk then why even both here with best practices at all? please see my input here
As to the above examples:

Electron isn't something that end users install regularly, is it?
node-sass I accept and am familiar with and I am happy to see it is showing a declining downloads trend

“Best practices” doesn’t mean “be as paranoid as possible”, it means “have the best balance between security and usability as possible”. Many potential security measures don’t meet that bar - including ignore-scripts.

So, I would argue it's not paranoia given that a common attack is to use pre/post scripts to run malicious code like stealing crypto wallet keys. Certain other languages don't allow arbitrary pre/post scripts or have built in mechanisms for the package manager to sandbox the installation.
I haven't been around npm much in the past several years, but if we can't suggest ignore scripts, are there suggestions we can make around best practices for sandboxing or avoiding the common attacks?

"paranoia" doesn't necessarily imply "unjustified" :-)
That's the problem. There is nothing that can be done short of ignoring all scripts, and there are no alternatives for package authors to avoid using scripts for the many relevant use cases.
Until npm provides alternatives - or more granular protection mechanisms - the status quo (virtually nobody ignores scripts, and those who do tend to run into problems) remains unchanged.

Closing with same reasoning as #20

@ljharb
issues with --ignore-scripts have been solved by @lavamoat/allow-scripts
It covers:

allowing selected scripts
setting up the repository and adding the flag
avoiding running an install script from a package that spoofed its name (eg. A bundled package that calls itself bcrypt won't get run if you allowed the real bcrypt)
stopping the installation early if the ignore-scripts flag was somehow accidentally removed or forgotten

No functionality depending on install scripts is broken.
There is no other way to defend against well crafted malicious packages using this technique. And they're becoming more and more common.
Full talk on this topic exactly:
https://m.youtube.com/watch?v=y2p1e7UmGYY

Replied in #20.


MeshView v0.8 - new enhancements

I would like to request two new enhancements:

The option to view point clouds (with activated slice function) without meshes.
The option to view one side of the brain and hide the other side of the brain (see attached image).

@marovst can you validate?

@Majpuc These versions have been validated and work great. Thank you for the help.


Important delay while pushing notifications and other operations from server to device

Hi WSO2 Team,
I am working on your WSO2EMM solution and more specifically the Android agent.
What I noticed when dealing with this part is some delay while sending notifications and other operations from the server to the device. Here I need to mention that I am using "Local Polling" and not "GCM".
Is there any recommendation to resolve this issue please guys ?

While using "Local Polling" the delay which appears when the operations getting applied to the device is due to polling interval. In Android platform configurations under Platform configurations, You can find Polling Interval configuration. You can change that to the minimum level of 1 second.

Yes that's what I did but the problem persists ! Operations are sent from the server and they are not even received by the device in most of the cases.
On the operations log page it keeps on showing "wating" status.