Deep neural network with stacked autoencoder on MNIST

Helo,
firts of all sorry for my english, it's not my native language (I'm french)
As the tittle said, I'm trying to train deep neural network with stack autoencoder but I'm stuck ...
thanks to fchollet's exemple I managed to implement a simple deep neural network that is work thinks to ReLU activation function (Xavier Glorot thesis).
But now I want to compar the result I have with this simple deep neural network to a deep network with stack auto encoder pre training.
I start with this code but I don't know how I can continue ... and everytime I try to add code I have an error ... so this is my valid code :
from future import absolute_import
from future import print_function
import numpy as np
np.random.seed(1337) # for reproducibility
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, AutoEncoder, Layer
from keras.optimizers import SGD, Adam, RMSprop, Adagrad, Adadelta
from keras.utils import np_utils
from keras.utils.dot_utils import Grapher
from keras.callbacks import ModelCheckpoint
batch_size = 10000
nb_classes = 10
nb_epoch = 1
adg = Adagrad()
sgd = SGD()
rms = RMSprop()
the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)
X_train = X_train.astype("float64")
X_test = X_test.astype("float64")
X_train /= 255
X_test /= 255
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')
convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)
model = Sequential()
creating the autoencoder
first hidden layer
model.add(AutoEncoder(encoder=Dense(784, 700),
decoder=Dense(700, 784),
output_reconstruction=False, tie_weights=True))
model.add(Activation('tanh'))
model.compile(loss='mean_squared_error', optimizer=rms)
model.fit(X_train, X_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=False, verbose=1, validation_data=None)
for layer in model.layers:
config=layer.get_config()
weight=layer.get_weights()

print(config)
print(weight)
second hidden layer
model.add(AutoEncoder(encoder=Dense(700, 600),
decoder=Dense(600, 700),
output_reconstruction=False, tie_weights=True))
model.add(Activation('tanh'))
model.compile(loss='mean_squared_error', optimizer=rms)
model.fit(X_train, X_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=False, verbose=1, validation_data=None)
Pre-training
fine-tuning
autoencoder evaluation

Not sure if this is what you are looking for but the following works.
(which is direct use of example in documentation http://keras.io/layers/core/#autoencoder)
from __future__ import absolute_import
from __future__ import print_function
import numpy as np
np.random.seed(1337) # for reproducibility

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import containers
from keras.layers.core import Dense, Dropout, Activation, AutoEncoder, Layer
from keras.optimizers import RMSprop
from keras.utils import np_utils
from keras.utils.dot_utils import Grapher
from keras.callbacks import ModelCheckpoint

batch_size = 10000
nb_classes = 10
nb_epoch = 30

# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)
X_train = X_train.astype("float32")
X_test = X_test.astype("float32")
X_train /= 255
X_test /= 255
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

#creating the autoencoder
ae = Sequential()
encoder = containers.Sequential([Dense(784, 700), Dense(700, 600)])
decoder = containers.Sequential([Dense(600, 700), Dense(700, 784)])
ae.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=False, tie_weights=True))

ae.compile(loss='mean_squared_error', optimizer=RMSprop())
ae.fit(X_train, X_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=False, verbose=1, validation_data=None)

The output
60000 train samples
10000 test samples
Epoch 0
60000/60000 [==============================] - 11s - loss: 0.1098
Epoch 1
60000/60000 [==============================] - 11s - loss: 0.1008
Epoch 2
60000/60000 [==============================] - 12s - loss: 0.0875
Epoch 3
60000/60000 [==============================] - 12s - loss: 0.0724
Epoch 4
60000/60000 [==============================] - 12s - loss: 0.0649
Epoch 5
20000/60000 [=========>....................] - ETA: 8s - loss: 0.0631

Updated the code to show how to use validation_data. output_reconstruction must be enabled.

thank you very much for your fast reply it's very apreciable. I can't test this code right now cause I haven't my laptop with me but I'll try it to night.
If I don't misunderstood the method for training Deep neural network with autoencoder, the first step is to train one by one each autoencoder to encode and decode their input. After the pre training is done, I can set the weights of my DNN with the weights of all encoder. Then I can apply a simple SGD.
Is my understand right ?
So if i right, my goal is to train a second autoencoder with inputs of the firs autoencoder. And that's what I don't find the way to do it. But perhaps with your code I'm going to succeed.

What you say sounds correct.
If you need to do layer-by-layer pre-training, then I think you need to write similar scripts for each stage, save the trained weight with save_weight function and load it at the next stage with load_weight function.
This however might not work, since the documentation says that when you load saved weight with load_weight function, the architecture of model must be identical. But looking at the source code this might not be the case and you can simply use the weight from the previous stage.
If your goal is to do experiment with pre-trainng, you are doing it right. But if your goal is to train a network, then keep in mind that by applying glorot initialization (which is default initialization scheme in Keras), you don't need to do pre-training. You can normally directly start training the network.

To do layer by layer pretraining you will currently need to run fit() (or train) on each model and then couple them later after training is done using output_reconstruction=True

It looks like, I didn't put activation function. (Sorry, I have not used Keras' AE before.)
This may be more correct architecture. It needs to be checked though.
encoder = containers.Sequential([Dense(784, 700, activation='sigmoid'),
                                 Dense(700, 600, activation='sigmoid')])
decoder = containers.Sequential([Dense(600, 700, activation='sigmoid'),
                                 Dense(700, 784, activation='sigmoid')])

As a matter of fact, it certainly changes the output
Without activation
Epoch 0
60000/60000 [==============================] - 37s - loss: 0.0371 - val_loss: 0.0229
Epoch 1
60000/60000 [==============================] - 36s - loss: 0.0191 - val_loss: 0.0157
Epoch 2
60000/60000 [==============================] - 36s - loss: 0.0144 - val_loss: 0.0126
Epoch 3
 3136/60000 [>.............................] - ETA: 36s - loss: 0.0130

With activation
60000/60000 [==============================] - 41s - loss: 0.0719 - val_loss: 0.0677
Epoch 1
60000/60000 [==============================] - 42s - loss: 0.0674 - val_loss: 0.0676
Epoch 2
60000/60000 [==============================] - 42s - loss: 0.0673 - val_loss: 0.0675
Epoch 3
 1216/60000 [..............................] - ETA: 42s - loss: 0.0669

@mthrok : yes you can stack the layers like that, but it is not doing greedy layerwise training. Just so you are aware.

I try do something like that to do greedy layerwise but it's not working...
from future import absolute_import
from future import print_function
import numpy as np
np.random.seed(1337) # for reproducibility
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, AutoEncoder, Layer
from keras.optimizers import SGD, Adam, RMSprop, Adagrad, Adadelta
from keras.utils import np_utils
from keras.utils.dot_utils import Grapher
from keras.callbacks import ModelCheckpoint
from keras.layers import containers
batch_size = 10000
nb_classes = 10
nb_epoch = 1
adg = Adagrad()
sgd = SGD()
rms = RMSprop()
the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)
X_train = X_train.astype("float64")
X_test = X_test.astype("float64")
X_train /= 255
X_test /= 255
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')
convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)
#creating the autoencoder
#first autoencoder
AE1_output_reconstruction = True
ae1 = Sequential()
encoder1 = containers.Sequential([Dense(784, 700, activation='tanh'), Dense(700, 600, activation='tanh')])
decoder1 = containers.Sequential([Dense(600, 700, activation='tanh'), Dense(700, 784, activation='tanh')])
ae1.add(AutoEncoder(encoder=encoder1, decoder=decoder1,
output_reconstruction=AE1_output_reconstruction, tie_weights=True))
#training the first autoencoder
ae1.compile(loss='mean_squared_error', optimizer=RMSprop())
ae1.fit(X_train, X_train, batch_size=batch_size, nb_epoch=nb_epoch,
show_accuracy=False, verbose=1, validation_data=[X_test, X_test])
#getting output of the first autoencoder to connect to the input of the
#second autoencoder
AE1_output_reconstruction = False #set output_reconstruction at False to get the hidden layer of autoencoder
FirstAeOutput = ae1.predict(X_train)
#second autoencoder
AE2_output_reconstruction = True
ae2 = Sequential()
encoder2 = containers.Sequential([Dense(600, 500, activation='tanh'), Dense(500, 400, activation='tanh')])
decoder2 = containers.Sequential([Dense(400, 500, activation='tanh'), Dense(500, 600, activation='tanh')])
ae2.add(AutoEncoder(encoder=encoder2, decoder=decoder2,
output_reconstruction=AE2_output_reconstruction, tie_weights=True))
#training the second autoencoder
ae2.compile(loss='mean_squared_error', optimizer=RMSprop())
ae2.fit(FirstAeOutput, FirstAeOutput, batch_size=batch_size, nb_epoch=nb_epoch,
show_accuracy=False, verbose=1)
#getting output of the second autoencoder to connect to the input of the
#third autoencoder
AE2_output_reconstruction = False
SecondAeOutput = ae2.predict(FirstAeOutput)
#third autoencoder
AE3_output_reconstruction = True
ae3 = Sequential()
encoder3 = containers.Sequential([Dense(400, 300, activation='tanh'), Dense(300, 200, activation='tanh')])
decoder3 = containers.Sequential([Dense(200, 300, activation='tanh'), Dense(300, 400, activation='tanh')])
ae3.add(AutoEncoder(encoder=encoder3, decoder=decoder3,
output_reconstruction=AE3_output_reconstruction, tie_weights=True))
#training the third autoencoder
ae2.compile(loss='mean_squared_error', optimizer=RMSprop())
ae2.fit(SecondAeOutput, SecondAeOutput, batch_size=batch_size, nb_epoch=nb_epoch,
show_accuracy=False, verbose=1)
#creating the Deep neural network with all encoder of each autoencoder trained before
model = Sequential()
model.add(ae1[0].encoder)
model.add(ae2[0].encoder)
model.add(ae3[0].encoder)
model.add(Dense(200, 10))
model.add(Activation('softmax'))
model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=True, verbose=2, validation_data=(X_test, Y_test))
score = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

Here is a layer-by-layer example.
But one thing I am not sure is if I am reusing encoder weight correctly, because the output before fine turning is almost same as no training at all. Maybe I need to do get_weight and set_weight manually.
I will look into it later.
from __future__ import absolute_import
from __future__ import print_function
import numpy as np
np.random.seed(1337) # for reproducibility

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import containers
from keras.layers.core import Dense, AutoEncoder
from keras.activations import sigmoid
from keras.utils import np_utils

batch_size = 64
nb_classes = 10
nb_epoch = 1
nb_hidden_layers = [784, 600, 500, 400]

# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(-1, 784)
X_test = X_test.reshape(-1, 784)
X_train = X_train.astype("float32") / 255.0
X_test = X_test.astype("float32") / 255.0
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

# Layer-wise pretraining
encoders = []
nb_hidden_layers = [784, 600, 500, 400]
X_train_tmp = np.copy(X_train)
for i, (n_in, n_out) in enumerate(zip(nb_hidden_layers[:-1], nb_hidden_layers[1:]), start=1):
    print('Training the layer {}: Input {} -> Output {}'.format(i, n_in, n_out))
    # Create AE and training
    ae = Sequential()
    encoder = containers.Sequential([Dense(n_in, n_out, activation='sigmoid')])
    decoder = containers.Sequential([Dense(n_out, n_in, activation='sigmoid')])
    ae.add(AutoEncoder(encoder=encoder, decoder=decoder,
                       output_reconstruction=False, tie_weights=True))
    ae.compile(loss='mean_squared_error', optimizer='rmsprop')
    ae.fit(X_train_tmp, X_train_tmp, batch_size=batch_size, nb_epoch=nb_epoch)
    # Store trainined weight and update training data
    encoders.append(ae.layers[0].encoder)
    X_train_tmp = ae.predict(X_train_tmp)

# Fine-turning
model = Sequential()
for encoder in encoders:
    model.add(encoder)
model.add(Dense(nb_hidden_layers[-1], nb_classes, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
score = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
print('Test score before fine turning:', score[0])
print('Test accuracy after fine turning:', score[1])
model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
          show_accuracy=True, validation_data=(X_test, Y_test))
score = model.evaluate(X_test, Y_test, show_accuracy=True, verbose=0)
print('Test score after fine turning:', score[0])
print('Test accuracy after fine turning:', score[1])

Outputs
60000 train samples
10000 test samples
Training the layer 1: Input 784 -> Output 600
Epoch 0
60000/60000 [==============================] - 18s - loss: 0.0733
Training the layer 2: Input 600 -> Output 500
Epoch 0
60000/60000 [==============================] - 11s - loss: 0.0022
Training the layer 3: Input 500 -> Output 400
Epoch 0
60000/60000 [==============================] - 8s - loss: 0.0003
Test score before fine turning: 2.36135076835
Test accuracy after fine turning: 0.1028
Train on 60000 samples, validate on 10000 samples
Epoch 0
60000/60000 [==============================] - 18s - loss: 1.3828 - acc: 0.4964 - val_loss: 0.6930 - val_acc: 0.7838
Test score before fine turning: 0.692979552291
Test accuracy after fine turning: 0.7838

I'm reading an article (thesis of LISA labs) about different method to train deep neural networks.
so this thesis explain that to train a network with autoencoder we should use crossentropy for eache autoencoder.

Cross entropy is for classification (ie you need classes). Autoencoders are purely MSE based.

But when i use parameter tie_weights
ae.add(AutoEncoder(encoder=encoder, decoder=decoder,
output_reconstruction=False, tie_weights=True))
it gives the error:
TypeError: init() got an unexpected keyword argument 'tie_weights'
How can i fix it, does it means my keras is in older version? and the document also has no tie_weights parameter for autoencoder :http://keras.io/layers/core/#autoencoder
Thanks.

This is because weight tying has been removed

thanks, so what can we do if i want to use tie_weights?

Traceback (most recent call last):
File "/home/nidhi/Documents/project/SAE.py", line 18, in
(X_train,y_train),(X_test,y_test)=mnist.load_data()
File "/usr/local/lib/python2.7/dist-packages/keras/datasets/mnist.py", line 17, in load_data
data = six.moves.cPickle.load(f)
File "/usr/lib/python2.7/gzip.py", line 455, in readline
c = self.read(readsize)
File "/usr/lib/python2.7/gzip.py", line 261, in read
self._read(readsize)
File "/usr/lib/python2.7/gzip.py", line 308, in _read
self._read_eof()
File "/usr/lib/python2.7/gzip.py", line 347, in _read_eof
hex(self.crc)))
IOError: CRC check failed 0x7603be46 != 0x4bbebed3L

@Nidhi1211 : This is unrelated. Your error is clearly in your data load....
@xypan1232 : You will have to write your own extend Layer and write your own autoencoder.

Traceback (most recent call last):
File "/home/nidhi/Documents/project/SAE.py", line 40, in
ae.add(AutoEncoder(encoder=encoder,decoder=decoder,output_reconstruction=False,tie_weights=True))
TypeError: init() got an unexpected keyword argument 'tie_weights'

@Nidhi1211 : I suggest you learn how to read stack traces.
The first stack trace is clearly not the same as the second.
The second is also mentioned above if you spend a few seconds to read the context.

Hi,
I'm having trouble to understand how properly configure AutoEncoder for non MNIST dataset. Let's say I have a dataset with N rows and M features and I'm trying to perform unsupervised learning to extract features from it. For that I setup simple autoencoder code following keras documentation example (http://keras.io/layers/core/#autoencoder). Here it is:
    # input_dim, output_dim, h1_dim below are
    # input, output and hidden layer dimensions, respectively
    encoder = containers.Sequential([Dense(output_dim=h1_dim, input_dim=input_dim)])
    decoder = containers.Sequential([Dense(output_dim=input_dim, input_dim=h1_dim)])
    model = Sequential()
    model.add(AutoEncoder(encoder=encoder, decoder=decoder,
                            output_reconstruction=True))

    # optimizer
    optimizer = RMSprop(lr=0.001, rho=0.9, epsilon=1e-06)

    # compile model
    model.compile(loss='mean_squared_error', optimizer=optimizer)

    print "model", pprint.pformat(model.to_json())
    model.fit(X, X, nb_epoch = epochs, batch_size = batch_size,
            validation_split=0.2,
            verbose = 2, shuffle=shuffle, show_accuracy = False)

Running this code with output_reconstructions=True flag in a model I'm able to fit the data X and I can predict a new set of values. But their dimension is the same as my input one. What I wanted is to extract the hidden layer values. The keras documentation says:
output_reconstruction: If this is False,
the output of the autoencoder is the output of the deepest hidden layer.
Otherwise, the output of the final decoder layer is returned.

So I though I'll use output_reconstructions=False and then I'll be able to extract
hidden layer. But I got the following error when I used this option in my model:
Epoch 1/50
Traceback (most recent call last):
  File "./autoenc.py", line 186, in <module>
    main()
  File "./autoenc.py", line 183, in main
    float(opts.split_size))
  File "./autoenc.py", line 151, in neuralnet
    verbose = 2, shuffle=shuffle, show_accuracy = False)
  File "build/bdist.linux-x86_64/egg/keras/models.py", line 581, in fit
  File "build/bdist.linux-x86_64/egg/keras/models.py", line 254, in _fit
  File "build/bdist.linux-x86_64/egg/keras/models.py", line 308, in _test_loop
  File "build/bdist.linux-x86_64/egg/keras/backend/theano_backend.py", line 365, in __call__
  File "/nfs/cor/user/vk/DataAnalysis/CUDA/homesite/src/theano/theano/compile/function_module.py", line 618, in __call__
    storage_map=self.fn.storage_map)
  File "/nfs/cor/user/vk/DataAnalysis/CUDA/homesite/src/theano/theano/gof/link.py", line 297, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/nfs/cor/user/vk/DataAnalysis/CUDA/homesite/src/theano/theano/compile/function_module.py", line 607, in __call__
    outputs = self.fn()
ValueError: dimension mismatch in args to gemm (10000,301)x(301,100)->(10000,301)
Apply node that caused the error: GpuGemm{inplace}(GpuFromHost.0, TensorConstant{1.0}, GpuFromHost.0, <CudaNdarrayType(float32, matrix)>, TensorConstant{-1.0})
Toposort index: 8
Inputs types: [CudaNdarrayType(float32, matrix), TensorType(float32, scalar), CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, matrix), TensorType(float32, scalar)]
Inputs shapes: [(10000, 301), (), (10000, 301), (301, 100), ()]
Inputs strides: [(301, 1), (), (301, 1), (100, 1), ()]
Inputs values: ['not shown', array(1.0, dtype=float32), 'not shown', 'not shown', array(-1.0, dtype=float32)]
Outputs clients: [[GpuElemwise{Add}[(0, 1)](GpuDimShuffle{x,0}.0, GpuGemm{inplace}.0)]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Please note, that my data X is a dataset without labels, I used 10000 as a batch size and my dataset has 301 features. I used hidden layer with 100 neurons and run keras version 0.3.0 on GPU.
I have several questions:

why using output_reconstruction=True flags works and False value does not?
even though this ticket and most examples use standard dataset like MNIST, I don't see any difference between MNIST and any other dataset, therefore I presume the code should work out of the box. Am I wrong on this statement, if so can someone explain the reason.
why using output_reconstructions=False gives dimension mismatch?
if I'll use activation='tanh' I got slightly different error:
ValueError: GpuElemwise. Input dimension mis-match. Input 2 (indices start at 0) has shape[1] == 301, but the output's size on that axis is 100. This is particular strange since it says that my output size is a size of my hidden layer, but this is what docs tell would happen.

I would appreciate any suggestions and explanations even using some dummy example.
Thanks,
Valentin.

why using output_reconstruction=True flags works and False value does not?
It is because you ask the "fit" function to do validation as well. The following would work even for output_reconstruction - False

model.fit(X, X, nb_epoch = epochs, batch_size = batch_size,
validation_data=None,
verbose = 2, shuffle=shuffle, show_accuracy = False)

even though this ticket and most examples use standard dataset like MNIST, I don't see any difference between MNIST and any other dataset, therefore I presume the code should work out of the box. Am I wrong on this statement, if so can someone explain the reason.

No difference between MNIST and any other dataset.

why using output_reconstructions=False gives dimension mismatch
It is mainly because of the "fit" function

if I'll use activation='tanh' I got slightly different error: ValueError: GpuElemwise. Input dimension mis-match. Input 2 (indices start at 0) has shape[1] == 301, but the output's size on that axis is 100. This is particular strange since it says that my output size is a size of my hidden layer, but this is what docs tell would happen.

Cannot understand why. Any more detailed explanation?

Hi all,
I'm trying to stack some auto encoders, but without success. Here is my code:
# from https://github.com/fchollet/keras/issues/358

from __future__ import absolute_import
from __future__ import print_function

import numpy as np
from keras import models
from keras import optimizers
from keras.datasets import mnist
from keras.layers import containers
from keras.layers.core import Dense, AutoEncoder

np.random.seed(1337)
batch_size = 100
nb_classes = 10
nb_epoch = 1

(X_train, _), (X_test, _) = mnist.load_data()
X_train = X_train.reshape(-1, 784)
X_test = X_test.reshape(-1, 784)
X_train = X_train.astype("float32") / 255.0
X_test = X_test.astype("float32") / 255.0
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

# Autoencoder 1
ae1 = models.Sequential()
ae1.add(AutoEncoder(encoder=containers.Sequential([Dense(500, input_dim=784)]),
                    decoder=containers.Sequential([Dense(784, input_dim=500)]),
                    output_reconstruction=True))
ae1.compile(loss='mse', optimizer=optimizers.RMSprop())
ae1.fit(X_train, X_train, batch_size=batch_size, nb_epoch=nb_epoch,
        show_accuracy=True, verbose=1, validation_data=[X_test, X_test])
X_train_tmp = ae1.predict(X_train)
print("Autoencoder data format: {0} - should be (60000, 500)".format(X_train_tmp.shape))

# Autoencoder 2
ae2 = models.Sequential()
ae2.add(AutoEncoder(encoder=containers.Sequential([Dense(400, input_dim=500)]),
                    decoder=containers.Sequential([Dense(500, input_dim=400)]),
                    output_reconstruction=True))
ae2.compile(loss='mse', optimizer=optimizers.RMSprop())
ae2.fit(X_train_tmp, X_train_tmp, batch_size=batch_size, nb_epoch=nb_epoch)
X_train_tmp = ae1.predict(X_train_tmp)

The output is:
Using Theano backend.
60000 train samples
10000 test samples
Train on 60000 samples, validate on 10000 samples
Epoch 1/1
60000/60000 [==============================] - 6s - loss: 0.0442 - acc: 0.0139 - val_loss: 0.0245 - val_acc: 0.0127
Autoencoder data format: (60000, 784) - should be (60000, 500)
Epoch 1/1
Traceback (most recent call last):
  File "/Users/sallesd/PycharmProjects/keras_tutorials/autoencoder.py", line 43, in <module>
    ae2.fit(X_train_tmp, X_train_tmp, batch_size=batch_size, nb_epoch=nb_epoch)
  File "/Users/sallesd/anaconda/envs/keras/lib/python2.7/site-packages/keras/models.py", line 646, in fit
    shuffle=shuffle, metrics=metrics)
  File "/Users/sallesd/anaconda/envs/keras/lib/python2.7/site-packages/keras/models.py", line 280, in _fit
    outs = f(ins_batch)
  File "/Users/sallesd/anaconda/envs/keras/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 384, in __call__
    return self.function(*inputs)
  File "/Users/sallesd/anaconda/envs/keras/lib/python2.7/site-packages/theano/compile/function_module.py", line 606, in __call__
    storage_map=self.fn.storage_map)
  File "/Users/sallesd/anaconda/envs/keras/lib/python2.7/site-packages/theano/compile/function_module.py", line 595, in __call__
    outputs = self.fn()
ValueError: Shape mismatch: x has 784 cols (and 100 rows) but y has 500 rows (and 400 cols)
Apply node that caused the error: Dot22(<TensorType(float32, matrix)>, <TensorType(float32, matrix)>)
Inputs types: [TensorType(float32, matrix), TensorType(float32, matrix)]
Inputs shapes: [(100, 784), (500, 400)]
Inputs strides: [(3136, 4), (1600, 4)]
Inputs values: ['not shown', 'not shown']

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Thanks,
Daniel

@dchevitarese you are trying to fit your second autoencoder with an input with size 784, while it expects one of 500.
if I get it right, you want to sneak on the innermost layer, so take care of what data are you dealing with. here is some hint:
`# ...
Autoencoder 1
realAE = AutoEncoder(encoder=containers.Sequential([Dense(500, input_dim=784)]),
decoder=containers.Sequential([Dense(784, input_dim=500)]),
output_reconstruction=True))
ae1 = models.Sequential([realAE])
ae1.compile(loss='mse', optimizer=optimizers.RMSprop())
ae1.fit(X_train, X_train, batch_size=batch_size, nb_epoch=nb_epoch,
show_accuracy=True, verbose=1, validation_data=[X_test, X_test])
realAE.output_reconstruction = False                     # grab inner representation ...
ae1.compile(loss='mse', optimizer=optimizers.RMSprop())  # ... but you have to recompile, don't re-train, weights are ketp (someone can confirm?)
X_train_tmp = ae1.predict(X_train)                       # X_train_tmp is now sized at 500
print("Autoencoder data format: {0} - should be (60000, 500)".format(X_train_tmp.shape))
... the same applies for AE 2
`
I hope it helps.

Hi @dibenedetto, I didn't know that I would have to recompile, but it did the trick. In the end, I got ~91% of accuracy. I could use a CNN to do the same job, but I am investigating this AE's to pre-train layers - and this also explains my next question: What do you mean with "take care of what data are you dealing with"?
Thank you

@mthrok Thanks for your help and your code!
Does anyone have any sample code to visualize the layers and output please?
Thanks in advance!

Hi @isalirezag, you can get all configuration by using model.get_config() that will give you something like this:
{'layers': [{'decoder_config': {'layers': [{'W_constraint': None,       'W_regularizer': None,       'activation': 'sigmoid',       'activity_regularizer': None,       'b_constraint': None,       'b_regularizer': None,       'cache_enabled': True,       'custom_name': 'dense',       'init': 'glorot_uniform',       'input_dim': None,       'input_shape': (860,),       'name': 'Dense',       'output_dim': 784,       'trainable': True}],     'name': 'Sequential'},    'encoder_config': {'layers': [{'W_constraint': None,       'W_regularizer': None,       'activation': 'sigmoid',       'activity_regularizer': None,       'b_constraint': None,       'b_regularizer': None,       'cache_enabled': True,       'custom_name': 'dense',       'init': 'glorot_uniform',       'input_dim': None,       'input_shape': (784,),       'name': 'Dense',       'output_dim': 860,       'trainable': True}],     'name': 'Sequential'},    'name': 'AutoEncoder',    'output_reconstruction': True}],  'loss': 'binary_crossentropy',  'name': 'Sequential',  'optimizer': {'epsilon': 1e-06,   'lr': 0.0010000000474974513,   'name': 'RMSprop',   'rho': 0.8999999761581421},  'sample_weight_mode': None}
Unfortunately, I don't think keras has a good visualization functionality. The only thing you get is a very simple graphviz plot, which is not helpful.

It seems the code can only work on keras==0.3.0

I just wanna know if the AutoEncoder has been removed from the newest Keras?
I have no idea why I cannot import AutoEncoder and containers...

It has been removed. You can easily accomplish it using the functional api.
Check it the blog for an example.
On Jun 9, 2016 2:56 AM, "lurker"<EMAIL_ADDRESS>wrote:

I just wanna know if the AutoEncoder has been removed from the newest
Keras?
I have no idea why I cannot import AutoEncoder and containers...
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/358#issuecomment-224815001, or mute
the thread
https://github.com/notifications/unsubscribe/AFHcNR8-Avd6cXVOPkKFAm4-EXoE5FQUks5qJ7kjgaJpZM4FT7x6
.

Hey, guys, I am also working on how to layer-by-layer train AE, and I'm new to Keras.
Will such code work ?
`from keras.layers import Input, Dense
from keras.models import Model
import numpy as np
import matplotlib.pyplot as plt
encoding_dim = 64
input_img = Input(shape=(867,))
encoded_l1 = Dense(512, activation='relu')(input_img)
encoded_l2 = Dense(256, activation='relu')(encoded_l1)
encoded_l3 = Dense(128, activation='relu')(encoded_l2)
encoded = Dense(64, activation='relu')(encoded_l3)
decoded = Dense(64, activation='relu')(encoded)
decoded_l1 = Dense(128, activation='relu')(decoded)
decoded_l2 = Dense(256, activation='relu')(decoded_l1)
decoded_l3 = Dense(512, activation='relu')(decoded_l2)
decoded_l4 = Dense(867, activation='sigmoid')(decoded_l3)
Layerwise Trainning
encoder_l1 = Model(input=input_img, output=Dense(867, activation='sigmoid')(encoded_l1))
encoder_l1.compile(optimizer='Adadelta', loss='binary_crossentropy')
encoder_l1.fit(x_train, x_train, nb_epoch=50, batch_size=256, shuffle=True, validation_data=(x_test, x_test))
encoder_l2 = Model(input=input_img, output=Dense(867, activation='sigmoid')(encoded_l2))
encoder_l2.compile(optimizer='Adadelta', loss='binary_crossentropy')
encoder_l2.fit(x_train, x_train, nb_epoch=50, batch_size=256, shuffle=True, validation_data=(x_test, x_test))
encoder_l3 = Model(input=input_img, output=Dense(867, activation='sigmoid')(encoded_l3))
encoder_l3.compile(optimizer='Adadelta', loss='binary_crossentropy')
encoder_l3.fit(x_train, x_train, nb_epoch=50, batch_size=256, shuffle=True, validation_data=(x_test, x_test))
encoder_l4 = Model(input=input_img, output=Dense(867, activation='sigmoid')(encoded))
encoder_l4.compile(optimizer='Adadelta', loss='binary_crossentropy')
encoder_l4.fit(x_train, x_train, nb_epoch=50, batch_size=256, shuffle=True, validation_data=(x_test, x_test))
autoencoder = Model(input=input_img, output=decoded_l4)
encoder = Model(input=input_img, output=encoded)
autoencoder.compile(optimizer='Adadelta', loss='binary_crossentropy')
autoencoder.fit(x_train, x_train,
nb_epoch=100,
batch_size=256,
shuffle=True,
validation_data=(x_test, x_test))`


file_put_contents(../../../../config/credentials.php.enc): failed to open stream: No such file or directory

so simple ,i just
mkdir ./config


affect

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: cf54bf5d-1748-46bf-a503-5e978f56126c
Version Independent ID: 5976a1df-52e9-eb19-fd41-c7b04fce74f8
Content: How different technologies effect Microsoft Teams sign-on, including restricting sign-on, and sign-in behaviours. - Microsoft Teams
Content Source: Teams/sign-in-teams.md
Service: msteams
GitHub Login: @MSFTTracyP
Microsoft Alias: tracyp

@scanum Why was this closed? I believe @nahloe was calling attention to the fact that this page’s title incorrectly uses “effect” instead of “affect.” That should be corrected.


Invalid type Google::Ads::GoogleAds::V1::Common::PolicyViolationKey to assign to submessage field ''.

Hi,
I m migrating 2.0.0 code to "3.2.0".
I am specifically trying to add a policy exemption to the payload of a keyword. Here is the code:
factories = Google::Ads::GoogleAds::Factories
keyword = factories.version_alternate_for(:resources).ad_group_criterion
operation = factories.version_alternate_for(:operations).create_resource.ad_group_criterion(keyword)
policy_violation_key = Google::Ads::GoogleAds::V1::Common::PolicyViolationKey.new(policy_name: "pharma", violating_text: "drug")
operation.exempt_policy_violation_keys.push(policy_violation_key)
# => *** Google::Protobuf::TypeError Exception: Invalid type Google::Ads::GoogleAds::V1::Common::PolicyViolationKey to assign to submessage field ''.

It worked on v2.0.0, what must I do now please? there is no relevant stacktrace and the error message is a little confusing.
Thanks in advance.

oh boy, sorry, I had to update my factory to Google::Ads::GoogleAds::V2::Common::PolicyViolationKey, of course :)


Missing reference to self when setting cache_dir for file cache when dimensions is supplied

@weskamm It looks like this line was meant to set self.cache_dir but is missing self.
https://github.com/mapproxy/mapproxy/blob/4315ecab0ab42031de4928c751b04995603a5b1b/mapproxy/cache/file.py#L54C13-L54C23

Ah, I see you stumbled upon that line as well :)
I think it should be used instead of self.cache_dir in the following line

No problem, I'll fix this in my PR fixing #796.


Sending empty list to parallel model causes segmentation fault

all latest package, with pytorch-1.1.0 and cuda9.0
snippet:
import torch_geometric.nn as tg_nn
net = tg_nn.GCNConv(1,1).cuda()
net = tg_nn.DataParallel(net)
net([])

I will look into it. Thanks.

Fixed in master. Thank you.


Expose preset temps for presets in HVAC mode

Is your feature request related to a problem? Please describe.
It is rather a missing feature as the values are not exposed by the plugin in AC mode...
Describe the solution you'd like
I'm using temp sensors (Danfoss) that can help set a target temperature and I'm coupling that with the schedule plugin...
Describe alternatives you've considered
I'd like to set the comfort target as the sensor set temp but I can't do that when heat pump is in AC mode as the preset temp is not exposed by your plugin...
Additional context
Could look like:
service: versatile_thermostat.set_preset_temperature
data:
    preset: boost
    temperature_hvac: 17.8
    temperature_away_hvac: 15
target:
    entity_id: climate.my_thermostat

Hello,
I think you can change the ac preset value with the following service call:
service: versatile_thermostat.set_preset_temperature
        data:
            preset: boost_ac            ####  notice the _ac suffix here
            temperature: 17.8
            temperature_away: 15
        target:
            entity_id: climate.thermostat_2

I will try this and I let you know.

Ok this was not accepted because list of preset was limited to eco, boost, comfort.
This is now fixed in 3.4.


Add key name lookup

For the next version, add code to look up the human-readable "key name" for a key.  This will be platform-specific, and it will also depend on the user's current key mapping.  For example, KEY_A will have the name "A" on a computer with a US keyboard layout and it will have the name "Q" if the computer uses a French keyboard layout.

How about standard for naming those keys? Right now everybody invents the names. Like LEFT in SDL, ArrowLeft in USB HID and LeftArrow somewhere else. I'd prefer CamelCase SDL convention.

That's a very good point.
If anything, I would prefer something derived from the USB HID names since they're part of a standard, but not all USB HID names are valid identifiers (e.g. *) and the full HID names are a bit verbose ("Keyboard a and A").  Now that I'm looking, the library source uses one convention for the enum (KEY_$x and KP_$x) and a slightly different convention for the id strings ($x and KP$x), which is less than optimal.
On the other hand, short names are kind of nice.

I guess USB HID didn't think about key names consistency even though they are most likely to be used in macros for different programs. So making some work beforehand and then submitting to USB HID may be a good option.
For example, Far Manager on Windows records keypresses as Keys("AltE R E A D M E F4 PgDn PgDn Enter"), which what I would type to search for README file, open it in editor and insert newline after two pages. I would stick to this convention as it is already there. Actually, it looks like Far Manager macros record keys pressed in specific and maintains a mapping of layout symbols to keys to know that recorded Keys("Alt F9 щ у") in Russian layout is the same as Keys("Alt F9 o e") if the letter is being used to navigate menus etc. But that's a tricky use case, although a practical one to draft specification from.
So, in the story above canonical key names match for keys that are not letters/digits. For letters/digits I'd like to see separation between KeyA and typed A character, which can be also a or Ф or ф depending on layout and Shift state. Maybe introducing a registry of "reverse layout mapping" could be an additional feature, to make sure that you can find KeyA even if user hits a key labelled Ф on his layout and tries to record such shortcut in settings.

To clarify, there would be two types of names for keys, but they're separate (so you can use the same name in both for different keys).

The Key ID identifies the location of the key, independent of keyboard layout.  This should be taken from the USB HID tables exclusively.  This is never localized.  This should be stored internally for key bindings, e.g., in configuration files.

The Key Label is specific to the keyboard layout, and is localized.  This should be displayed to the user.

For example, let's say the leftmost key on the home row is bound to "move left" and the user has a French keyboard layout.  The config file would have a line like A: MoveLeft, and the game would print a message to the screen like "Press Q to move left".  This information has to be gleaned at runtime from the appropriate APIs, some of which are fairly messy, and all of which are completely incompatible with each other.


Fix remaining manager being removed from member list by --force-new-cluster

When loading a snapshot while forcing a new cluster, we need to avoid removing ourselves from the member list. This caused the restored swarm to show zero managers.
Add a test.
Also, fix a few small items:

Remove unnecessary cast
Change map[uint64]bool to map[uint64]struct{}
Remove trailing space in log message
Active members listed in the snapshot should be added to the removed list, not simply ignored.
Use assert.Len in raft tests.

This is a 1.13-rc regression caused by #1469.
cc @LK4D4 @cyli

LGTM

Current coverage is 54.46% (diff: 76.92%)

Merging #1852 into master will increase coverage by 0.03%

@@             master      #1852   diff @@
==========================================
  Files           103        103
  Lines         17219      17221     +2
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           9372       9379     +7
+ Misses         6706       6701     -5
  Partials       1141       1141

Powered by Codecov. Last update 387bc93...e58133c

LGTM.


MISSINGNO pokemon deleted

Simply removed the Pokemon called "MISSINGNO" in the list of Pokemon that you don't want to transfer. Also, fixed a typo (""We will not store you(r) login information, the file is generated on client side.").

you rewrote the whole file.. look at #6


program not working on x86 win 10

Is program compatible with x86 CPU?

I don't have one, but it should be


figfam member list: show only CDSs

The figfam member list page, the one that is linked from the figfam id on the genome overview page,
currently lists both CDSs and pseudogenes. Please change it to show only CDSs.
-Maulik

#395


Make sure hmac isn't given a unicode string

So I ran into a problem with this today where the headers were unicode which isn't supported for python2 hmac.  Converting to a bytearray seems like a reasonable guard against this and might help others.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Were you able to verify this locally such that it fixes your issue and didn't cause any other obvious issues? I'm not using Python on any projects and am unable to verify at this time?

Yes...
Starting the code thusly-
AWS_CLIENT_SECRET_KEY=foobar python app.py
and sending in a request-
curl --request POST \   --url http://localhost:5000/s3/sign \   --header 'content-type: application/json' \   --data '{"headers":"😎"}'
The un-patched code throws a 500 error while the patch-1 code returns-
{   "signature": "lifmnNdZDK4yrGeX875UyCdFEkM=" }
As a sanity check if I instead send in an innocuous header text both versions come up with the same signature-
{ "headers": "some text to sign" }
{ "signature": "Tb2/i33y+AoKgKr+8BakjLWuMjE=" }

Cool. Thanks for finding & fixing this!

No problem.  Thanks for providing the examples!


Feature Request: Bulk Editor Additions

I think I may be becoming a bother, but I just wonder if it would be possible to add studio and country to the bulk editor for TV Shows?

we've migrated v3 to GitLab (https://gitlab.com/tinyMediaManager/tinyMediaManager) along with all issues. The issues for v2 at GitHub will be closed - we will respond to the corresponding issue in GitLab in the future
br
Manuel


Can I show the network structure at the scope level

I want to see the structure of my model, which is two tower of ALBERT, but it's too large and so many nodes.
Can I show the network structure at the scope level?

Duplicate of #68


Track - Tracked expense appears on Self-DM chat in LHN when share it offline

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: v9.0.49-0
Reproducible in staging?: Y
Reproducible in production?: Y
Email or phone of affected tester (no customers<EMAIL_ADDRESS>Issue reported by: Applause Internal Team
Action Performed:

Go offline
Go to LHN> Track expense> Complete the flow
Select Share with my accountant> Complete the flow
Go online
Take a look at Self DM chat at LHN

Expected Result:
Tracked expense should not appear since it is already shared
Actual Result:
Shared expense appears on Self-DM chat in LHN when user come back online
Workaround:
Unknown
Platforms:

[ ] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/user-attachments/assets/30eb893e-1fe8-4182-a6db-e49dd7e67ee2

View all open jobs on GitHub

@lschurr  FYI I haven't added the External label as I wasn't 100% sure about this issue. Please take a look and add the label if you agree it's a bug and can be handled by external contributors

We've added the Hot Pick label + project - need to wait for an engineer to pick this up.

Hey @luacmartins - just using the auto-assigner since this is a BE issue. Are you able to investigate and work on this one?

My plate is quite full atm. @thienlnam do you have bandwidth to look into this one since it seems related to tracked expenses?

My plate is full for a while as well, so won't be able to pick this up either. Doesn't look like this bug impacts functionality or anything perhaps in a few weeks or something

Going to unassign myself since this is Internal. Please add me back if we end up needing BZ for payment!

Unassigning engineers who are maxed out and will call out as hot pick

@muttmuure @FitseTLT I'm experimenting with improving how internal issues are reported and worked on. Could you please open a new issue for the backend fix and use this template? It's even OK if you assign it to me. Basically, it's just not clear to me what backend fix is required for this issue, and I see that in a lot of issues. I'm attempting to make the reporting of such problem more standard so it's easier and faster for me to pick it up and work on it.

Are you requesting me to open the issue?

Issue not reproducible during KI retests. (First week)

Since this is no longer reproducible, let's close


Planning and Time Management

https://github.com/nesterone/sf-eng-init-gl-spring-2016/blob/master/00-bootcamp/02-planing-and-time-management/planing-and-time-management.md#planning-and-time-management


checking for details instead of action menu

Bug fix ('fix', 'repair', 'bug')
Non breaking change ('update', 'change')

This ended up becoming unnecessary for now.


CMU Access needed to use resource

The resource "Math Fundamentals for Robotics by Michael Erdmann" in Mathematics/Robotics only links to the landing page for the course to access the actual study material on these pages you need to be physically present at CMU Campus or have a @andrew.cmu.edu id.
So, this resource is actually quite useless for the general public.

Resolved, Thanks


Dependabot

This will help you to keep repo up to date.
Btw, I recommend to get rid of variables. With this pull request, it's obsolete

With the exception of Spokestack Tray, the repositories that contain Android apps represent sample code, and I'm not particularly keen on getting pinged for each of them every time Jetbrains or Google releases a new version of a support library. I do want to know if there's a security vulnerability, so Dependabot alerts/PRs are enabled for those. I also want the repos to build with the latest version of Spokestack, so I have a separate action that runs to update it when new versions are released.
I agree with you about the variables. This type of organization was an example I'd picked up from another project, but it doesn't make much sense for self-contained repositories like this. I'll remove those in a separate PR.

Closing for now based on what I said above, but I appreciate the input and will keep it around for future consideration.

You will stuck on outdated libraries.


Should I use the projector weights when evaluating pretrained model?

Thanks for this great repo.
I was able to load the model and run eval following this command
python -m llava.eval.run_llava \
    --model-name llava_models/13b \
    --image-file "https://llava-vl.github.io/static/images/view.jpg" \
    --query "What are the things I should be cautious about when I visit here?"

However, I notice that the weights for the projector layers in this model is different from the projector weights (as well as embedding weights) downloaded here https://github.com/haotian-liu/LLaVA#llava-pretrained-projector-weights. Are the separate projection weights needed for this model? Why are they different from the weights currently in the model?

Hi, thank you for your interest in our work.
You do not need to load the pretrained MLP layer for running our chat model.  The current released checkpoint uses full-model finetuning (except for CLIP visual encoder), thus the projection layer is also updated -- this accounts for the weight difference.
We are also trying with ablation on freezing the linear projection layer or not, and will update the comparison and analysis in our paper. Thanks.


example code for adding a custom adjuster to zipkin-sparkstreaming job

zipkin-sparkstreaming custom adjuster example
This example shows how to add a custom adjuster for zipkin-sparkstreaming.
We'll create a jar with a simple adjuster that trims the length of binary annotation values.
This adjuster will be applied by placing it in the class path of the spark job.
Steps for creating an adjuster jar

Create a java source file that..

extends zipkin.sparkstreaming.Adjuster
has an annotation org.springframework.context.annotation.Configuration

Create a resource file META-INF/spring.factories that contains

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
your.package.FooAdjuster

Put compiled class and META-INF/spring.factories into a jar.

In maven, the following structure would accomplish this.
src/main/java/your/package/FooAdjuster.java
src/main/resources/META-INF/spring.factories

Building example
mvn clean package

Running adjuster jar with the job
Download spark-streaming jar
wget -O zipkin-sparkstreaming-job.jar 'https://search.maven.org/remote_content?g=io.zipkin.sparkstreaming&a=zipkin-sparkstreaming-job&v=LATEST'

Run the job by adding adjuster jar to the classpath.
Note We'll can't run the job with -jar flag. If we use this flag, -cp option is ignored.
java -cp "zipkin-sparkstreaming-job.jar:zipkin-sparkstreaming-example-*.jar" \
  zipkin.sparkstreaming.job.ZipkinSparkStreamingJob \
  --zipkin.storage.type=elasticsearch \
  --zipkin.storage.elasticsearch.hosts=http://<IP_ADDRESS>:9200 \
  --zipkin.sparkstreaming.stream.kafka.bootstrap-servers=<IP_ADDRESS>:9092

Why this should work..
The spring factories thing allows spring boot to pick up and load the class you made. This is called auto-configuration... like java service loader, but better. Read more here.

network broke earlier...
so the key takeaway is let's put detailed documentation in the upstream project, then link to that if folks need or want to learn more.
This README could probably do better to talk about the value of what this is doing. For example, some intro like.

Adjusters are used to clean or change data that goes into zipkin, such that it is more usable. For the sake of example, we assume you have an application adding very large tags in trace data.  This is an example adjuster for the spark streaming job, which truncates tags to a configured length.

then, the build/run instructions you have already.
what's left after that is how to actually run the example.
For example, how would you know if this works? Maybe passing an arg that limits to 10 characters to the job then making a test trace. If this part isn't ready due to missing "testing mode" for upstream, it can be a TODO.

Sure, we can take it out. But why we shouldn't use BYTES in zipkin?
The only supported binary annotation type is String. That's the only one
queryable, and the agreed upon next model drops other types. There have
been numerous issues on people making mistakes notably with BYTES and i64.
I can search through the issues list, but the advice to only use String is
at least a year old.

You can merge this when you are ready. I will be displaced for a while


⚠ Samsung has degraded performance

In ced0765, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2008 ms

Resolved: Samsung performance has improved in bb97cf1.


Transfer  base16-sioyek?

Hi, I maintain base16-sioyek, would you be interested in transferring it to this org? Or should I open a PR to add it to the template list in the readme?

Hey @loiccoyle, I've just sent you an invite to the tinted-theming org under the team @tinted-theming/sioyek. Once you've accepted the invite, you should be able to transfer the repo across, once that is done I'll set the tinted-theming/sioyek team to maintain that repository, add the bot token for building and things should be mostly set up. There is more info about the process here: https://github.com/tinted-theming/home/blob/main/CONTRIBUTING.md

Great, I transferred the repo https://github.com/tinted-theming/base16-sioyek

Is there anything that I need to do or can we close this?

All done :smile:


PBR not working on IOS after compiling app to binaries

[X] Review the documentation: https://docs.viromedia.com/
[X] Search for existing issues: https://github.com/viromedia/viro
[x] Use the latest ViroReact release: https://docs.viromedia.com/docs/releases

Environment
Please provide the following information about your environment:

Development OS:
Mac
Device OS & Version:
IOS 12
Version: ViroReact version and React Native version
Viro React 2.11.0
React Native 0.55.1
Device(s): What device(s) are you are seeing the issue on (i.e. iPhone X, Samsung Note 8, etc)
IPhone 6S

Description
When I run the development app in XCode, PBR works fine and I get a shiny texture on my model, but after archiving the app and uploading it to TestFlight, the PBR goes away and the model has a matte texture on it.
Reproducible Demo

<ViroARSceneNavigator
            apiKey='xxxxxxxxxxxxx'
            initialScene={{scene: myScene}}
            style={{flex: 1}}
            debug={true}
            hdrEnabled={true}
            pbrEnabled={true}
          />

...

ViroMaterials.createMaterials({
  rocks: {
    lightingModel: 'PBR'
  }
})

...

<ViroDirectionalLight position={[0, 0.5, 0]} color="#ffffff" direction={[0, -.01, 0]} attenuationStartDistance={10} attenuationEndDistance={20} castsShadow={true} shadowMapSize={2048} shadowNearZ={2} shadowFarZ={5} shadowOpacity={.7} intensity={500}/>
<ViroAmbientLight color="#ffffff" intensity={1000}/>
<ViroOmniLight
  intensity={13000}
  position={[0, 0.3, 0]}
  color={"#fff8c2"}
  attenuationStartDistance={20}
  attenuationEndDistance={30}
  />

<ViroSpotLight
  color="#ffffff"
  attenuationStartDistance={5}
  attenuationEndDistance={10}
  position={[0, 0, 0]}
  direction={[0, -1, 0]}
  innerAngle={0}
  outerAngle={35}
  intensity={12000}
  />

<Viro3DObject source={require('../assets/models/rockstorm9.vrx')}
    position={[-0.03, -1.8, 0.0]}
    scale={[0.0011, 0.0011, 0.0011]}
    materials={['rocks']}
    type="VRX"
    animation={{name:'OSG_Scene', run: true, loop: true}}/>

I'm having the exact same issue here. Same versions...

It only happens when I run the release build/archive the app

Hi @Tino-F,
I took a look at your code and there are 2 issues:

the materials={['rocks']} property is unnecessary for FBX models as the lighting model is specified within the FBX object format.

this is working in debug but not release/archive because of how React Native works. When you converted your .fbx to .vrx it spit out a few other png/jpg that the file uses as textures. In debug mode w/ a package server, you didn't need to require() them because they're treated as url files. However, to make them available in release, you need to require() the files in order for React Native to know that it needs to bundle the assets. This is where the resources=[] property comes in handy.

So to get your object working, it should look something like this:
            <Viro3DObject
              source={require('./res/wheels.vrx')}
              type='VRX'
              resources={[
                require('./res/wheels_Base_Color.jpg'),
                require('./res/wheels_Metallic.jpg'),
                require('./res/wheels_Roughness.jpg'),
                require('./res/wheels_Normal_OpenGL.jpg'),
              ]}/>

Requiring the textures resolved the issue. Thank you!


Explain how to integrate with emacs

ReasonProject mentions IDE integration and has links to Atom and Vim. Ideally it could also mention emacs and provide a link to reason-mode as well as specific instructions.
If reason-mode is setup (assuming the user also does npm -g install git://github.com/reasonml/reason-cli.git as the documentation asks), does the user also need to run npm run env -- emacs? On mac that's more likely to be npm run env -- open -a Emacs for users that have installed pure emacs from here.
I'll assume that the right way is to use npm run env -- open -a Emacs. I'll submit a pull request with this assumption.

Thank you! I will look at this soon. If I don't merge in about a day, please feel free to ping me again.

I'm not so sure the advice on emacs is correct in the readme. The emacs integration docs now say to install the global tools, and that's worked for me. However, installing global tools goes against the entire narrative around non-invasive use of Reason...
It looks like our choices are:
(1) Ask the user to install global tools for emacs. This makes emacs a special case in the readme, but it aligns with the emacs integration doc.
(2) Leave the doc as-is, but my experience was that it's suboptimal for typical emacs users, who live inside emacs all day and don't want to start it per-environment. Plus running emacs from the command line on Mac is unlikely to do what you want.
Let me know which way you'd like this to go, I'd be happy to take a turn at the doc either way.


Fix JSON enconding issue

I'll explain tomorrow... it's insane.

Enhance doesn't mutate the object, it creates a new request.


added settings compatibility for contextMenu. is it usable or not

This PR adds settings mode to the tree component. It is just contextMenu right now.

@Taifuru I did the same stuff :) But also did it in a recursive way with the ability to override settings for children nodes. This PR should cover your use case as well.
https://github.com/valor-software/ng2-tree/pull/32

@rychkog yes, if it's works, it's fine with me :) good work by the way.

@Taifuru thanks for participation! :)


quiche: Add quic_pcc_sender_impl.h

Description:
The intention of this PR is to add quic_pcc_sender_impl.h, but since it depends on quic_types.h, in the end the majority of this PR are

A fix to make (Set|Get)QuicFlag work. (A new QUICHE tarball is pulled in because of this)
Refactor to make all platform tests to build.

Risk Level: none, code not used yet.
Testing:
bazel test --test_output=all test/extensions/quic_listeners/quiche/platform:all @com_googlesource_quiche//:all
Docs Changes: none
Release Notes: none

/assign @danzh2010

/assign @alyssawilk
/assign @mattklein123


Basic support for vue-styled-components

Can we get support for vue-styled-components? It would be great to render the components in the documentation. Rendering props would be nice, but is not necessary.
Right now, an error is thrown in the console
⠸ Compiling...Warning: Cannot parse src/components/buttons/StyledButton.vue: Error: No suitable component definition found

It usually means that vue-docgen-api does not understand your source code or when using third-party libraries, try to file an issue here:
https://github.com/vue-styleguidist/vue-docgen-api/issues

The component looks like this:
<script>
import styled from 'vue-styled-components';
const StyledButton = styled('button')`
    padding: 0 10px;
    margin: 0;
    box-shadow: 1px 2px 3px rgba(0, 0, 0, .1);
    overflow: hidden;
`;

export default StyledButton;
</script>

Is there currently any way to use vue styleguidist with styled commponents?

Hey guys !! Since we have a workaround I will close this one.
Please have a looksie at the cookbook.
https://vue-styleguidist.github.io/Cookbook.html
Thank you @jessicasachs


LuckPerms shows log messages twice (bungee)

Hey,
So i've installed LuckPerms on both the proxy and the spigot servers, however logs show up twice now. Did I do something wrong?
http://prntscr.com/or32nf

Are you using the pluginmsg messaging service?

Yes I am. It shows it once for me when I make a change in luckperms, but when someone else does they see the logs twice.
http://prntscr.com/oul0yt

13:42:09 [INFO] [LP] LOG > (Console) [U] (ni20ck05)
13:42:09 [INFO] [LP] LOG > parent set villager
13:42:09 [INFO] [LuckPerms] [Messaging] Received user update ping for 'ni20ck05' with id: afd53817-802c-4e33-9cc8-a65bcb84db29
13:42:13 [INFO] [/<IP_ADDRESS>:62155] <-> InitialHandler has connected
13:42:15 [INFO] [LP] LOG > (Console) [U] (ni20ck05)
13:42:15 [INFO] [LP] LOG > parent set novice
13:42:15 [INFO] [LuckPerms] [Messaging] Received user update ping for 'ni20ck05' with id: 1b54aa6b-3248-4041-b126-fa3d0664a87b

https://github.com/lucko/LuckPerms/blob/c86f6c0f368610152112ece73c898aa0954cfcc9/bungee/src/main/resources/config.yml#L227-L232
:)))

@lucko I have that set to true on the proxy and false on the spigot servers, yet it still shows duplicates.
http://prntscr.com/ouqa1i

Do you have multiple proxies?

I only have one proxy.
Bungee config: https://pastebin.com/86JSjj00 (database leftout, they're all on the same though)
Spigot config: https://pastebin.com/GthK6jYX

Any idea?

@lucko Any news? :P

I disabled it on the proxy and enabled it on all backend servers, that seems to work.

Actually no, now the permissions don't sync.

That setting only controls whether log entries received from other servers via the messaging service are broadcasted to online players.
It doesn't change any other property of the system, and has no side effects.

Yeah, that comment was invalid.
However, setting it to true on the backend servers and false on the proxy stopped the double log messages.
However, it no longer seems to broadcast the logs among all servers. Really not sure what I do wrong.

I'm unable to reproduce this now on the latest version. :)

I'm facing the exact same problem here using LuckPerms 5.2.35. All messages a broadcasted twice on the server where the command was executed.

broadcast-received-log-entries is set to false on all bukkit servers and true on proxy.
Also using the pluginmsg channel.


Using Hoeffding Tree as the base learner for Bagging

Expected behavior
It should be possible to use the Hoeffding Tree classifier as the base learner for the Bagging method.
Precisely, I've tried executing Bagging using the Hoeffding Tree as the base learner and it failed.
Observed behavior
No classification results were generated. An exception was raised during execution.
Exception:
17/12/09 23:17:06 ERROR Executor: Exception in task 1.0 in stage 10.0 (TID 21)
java.lang.ClassCastException: org.apache.spark.streamdm.classifiers.trees.HoeffdingTreeModel cannot be cast to org.apache.spark.streamdm.core.ClassificationModel
	at org.apache.spark.streamdm.classifiers.meta.Bagging$$anonfun$ensemblePredict$1.apply$mcVI$sp(Bagging.scala:114)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
	at org.apache.spark.streamdm.classifiers.meta.Bagging.ensemblePredict(Bagging.scala:113)
	at org.apache.spark.streamdm.classifiers.meta.Bagging$$anonfun$predict$1.apply(Bagging.scala:97)
	at org.apache.spark.streamdm.classifiers.meta.Bagging$$anonfun$predict$1.apply(Bagging.scala:97)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193)
	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Steps to reproduce the issue
Used 2 different datasets (electricity and covertype).
Command line
./spark.sh "200 EvaluatePrequential -l (meta.Bagging -l trees.HoeffdingTree) -s (FileReader -f ../data/elecNormNew.arff -k 4532 -d 10 -i 45312) -e (BasicClassificationEvaluator -c -m) -h" 1> result_elec.txt 2> log_elec.log
./spark.sh "200 EvaluatePrequential -l (meta.Bagging -l trees.HoeffdingTree)-s (FileReader -f ../data/covtypeNorm.arff -k 5810 -d 10 -i 581012) -e (BasicClassificationEvaluator -c -m) -h" 1> result_covt.txt 2> log_covt.log

Data source
elecNormNew.arff and covtypeNorm.arff
Infrastructure details

Java Version: 1.8.0_152
Scala Version: 2.10
Spark version: 2.2.0
OS version: macOS Sierra (10.12.4)
Cluster mode or local mode? Both

Issue addressed by #85


Jupyter Export/Import Adds More Whitespace to Markdown Cells

Environment data

VS Code version: 1.38.0-insider
Extension version (available under the Extensions sidebar): 2019.8.30787
OS and version: Ubuntu 16.04, accessed via the Remote-SSH extension
Python version (& distribution if applicable, e.g. Anaconda): 3.5.7
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): pipenv
Relevant/affected Python packages and their versions: N/A
Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info #3977): Set to false

Expected behaviour

Export a script to a Jupyter Notebook
Import the Jupyter Notebook back into a separate Python script.
diff script1.py script2.py should show no difference.

Actual behaviour

Export a script to a Jupyter Notebook
Import the Jupyter Notebook back into a separate Python script.
diff script1.py script2.py shows that Markdown cells' whitespace after # have increased.

If you repeat these steps multiple times, the amount of whitespace increases every time.
Steps to reproduce:
Make a Python file and call it script1.py:
# To add a new cell, type '#%%'
# To add a new markdown cell, type '#%% [markdown]'

#%% [markdown]
# # Interactive Map

# This app displays features from a [GPX](https://en.wikipedia.org/wiki/GPS_Exchange_Format) log file.

#%%
#!/usr/bin/env python
# coding: utf-8

#%%
import datetime
import json
import os

Ctrl + Shift + P  --> Python: Export Current Python File as Jupyter Notebook --> call it script1.ipynb
Ctrl + Shift + P  --> Python: Import Jupyter Notebook --> select script1.ipynb
Save it as script2.py
Run diff script1.py script2.py
(my-scripts) vagrant@vagrant:~/my-scripts$ diff script1.py script2.py
4,5c4,5
< # # Interactive Map

< # This app displays features from a [GPX](https://en.wikipedia.org/wiki/GPS_Exchange_Format) log file.
---
> #  # Interactive Map

> #  This app displays features from a [GPX](https://en.wikipedia.org/wiki/GPS_Exchange_Format) log file.
15c15,17
< import os
\ No newline at end of file
---
> import os
>
>
(my-scripts) vagrant@vagrant:~/my-scripts$

Notice that an extra space was added after the #'s. Also it looks like the new script (script2.py) has some extra newlines thrown in as well.

Logs
Python logs:
> ~/.local/share/virtualenvs/my-scripts-mbmSyDy9/bin/python -m jupyter nbconvert ~/my-scripts/interactive_map.ipynb --to python --stdout --template /tmp/tmp-5741Wldk8efc5T5K.tpl

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)
2
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:14: Cached data exists getEnvironmentVariables, <No Resource>
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:14: Activation Commands received source /home/vagrant/.local/share/virtualenvs/my-scripts-mbmSyDy9/bin/activate for shell bash
2
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:14: Cached data exists getEnvironmentVariables, <No Resource>
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:14: Has Custom Env Vars
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:14: Activating Environment to capture Environment variables, . /home/vagrant/.local/share/virtualenvs/my-scripts-mbmSyDy9/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python /home/vagrant/.vscode-server-insiders/extensions/ms-python.python-2019.8.30787/pythonFiles/printEnvVariables.py
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:14: parseEnvironmentOutput, Class name = b, completed in 0ms
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:14: getActivatedEnvironmentVariables, Class name = b, completed in 48ms, Arg 1: undefined, Arg 2: {"architecture":3,"path":"/home/vagrant/.local/share/virtualenvs/my-scripts-mbmSyDy9/bin/python","version":{"raw":"3.5.7-final","major":3,"minor":5,"patch":7,"prerelease":["final"],"build":[],"version":"3.5.7-final"},"sysPrefix":"/home/vagrant/.local/share/virtualenvs/my-scripts-mbmSyDy9","fileHash":"a77442936cf833efcdf5343c1e77e597183d3a233b444b83fb9d5577a938fab7aa8a0ff3db5dfbefae69ed160743c4bb3ab798086b4422e79f0dd55d668f314c","type":"PipEnv","pipEnvWorkspaceFolder":"/home/vagrant/my-scripts","envName":"my-scripts","displayName":"Python 3.5.7 64-bit ('my-scripts': pipenv)"}, Arg 3: true
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:14: > ~/.local/share/virtualenvs/my-scripts-mbmSyDy9/bin/python -m jupyter notebook --version
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:14: Cached data exists ActivatedEnvironmentVariables, <No Resource>
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:14: getActivatedEnvironmentVariables, Class name = b, completed in 2ms, Arg 1: undefined, Arg 2: {"architecture":3,"path":"/home/vagrant/.local/share/virtualenvs/my-scripts-mbmSyDy9/bin/python","version":{"raw":"3.5.7-final","major":3,"minor":5,"patch":7,"prerelease":["final"],"build":[],"version":"3.5.7-final"},"sysPrefix":"/home/vagrant/.local/share/virtualenvs/my-scripts-mbmSyDy9","fileHash":"a77442936cf833efcdf5343c1e77e597183d3a233b444b83fb9d5577a938fab7aa8a0ff3db5dfbefae69ed160743c4bb3ab798086b4422e79f0dd55d668f314c","type":"PipEnv","pipEnvWorkspaceFolder":"/home/vagrant/my-scripts","envName":"my-scripts","displayName":"Python 3.5.7 64-bit ('my-scripts': pipenv)"}, Arg 3: true
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:32: Cached data exists getEnvironmentVariables, <No Resource>
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:32: Cached data exists ActivatedEnvironmentVariables, <No Resource>
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:32: getActivatedEnvironmentVariables, Class name = b, completed in 1ms, Arg 1: undefined, Arg 2: {"architecture":3,"path":"/home/vagrant/.local/share/virtualenvs/my-scripts-mbmSyDy9/bin/python","version":{"raw":"3.5.7-final","major":3,"minor":5,"patch":7,"prerelease":["final"],"build":[],"version":"3.5.7-final"},"sysPrefix":"/home/vagrant/.local/share/virtualenvs/my-scripts-mbmSyDy9","fileHash":"a77442936cf833efcdf5343c1e77e597183d3a233b444b83fb9d5577a938fab7aa8a0ff3db5dfbefae69ed160743c4bb3ab798086b4422e79f0dd55d668f314c","type":"PipEnv","pipEnvWorkspaceFolder":"/home/vagrant/my-scripts","envName":"my-scripts","displayName":"Python 3.5.7 64-bit ('my-scripts': pipenv)"}, Arg 3: true
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:32: > ~/.local/share/virtualenvs/my-scripts-mbmSyDy9/bin/python -m jupyter nbconvert --version
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:33: Cached data exists ActivatedEnvironmentVariables, <No Resource>
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:33: getActivatedEnvironmentVariables, Class name = b, completed in 1ms, Arg 1: undefined, Arg 2: {"architecture":3,"path":"/home/vagrant/.local/share/virtualenvs/my-scripts-mbmSyDy9/bin/python","version":{"raw":"3.5.7-final","major":3,"minor":5,"patch":7,"prerelease":["final"],"build":[],"version":"3.5.7-final"},"sysPrefix":"/home/vagrant/.local/share/virtualenvs/my-scripts-mbmSyDy9","fileHash":"a77442936cf833efcdf5343c1e77e597183d3a233b444b83fb9d5577a938fab7aa8a0ff3db5dfbefae69ed160743c4bb3ab798086b4422e79f0dd55d668f314c","type":"PipEnv","pipEnvWorkspaceFolder":"/home/vagrant/my-scripts","envName":"my-scripts","displayName":"Python 3.5.7 64-bit ('my-scripts': pipenv)"}, Arg 3: true
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:33: > ~/.local/share/virtualenvs/my-scripts-mbmSyDy9/bin/python -m jupyter nbconvert ~/my-scripts/interactive_map.ipynb --to python --stdout --template /tmp/tmp-8654GvzgJVsK4ws3.tpl
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:34: [NbConvertApp] Converting notebook /home/vagrant/my-scripts/interactive_map.ipynb to python
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:34: Cached data exists getEnvironmentVariables, Untitled-1
console.ts:137 [Extension Host] Info Python Extension: 2019-08-28 20:53:52: Cached data exists getEnvironmentVariables, /home/vagrant/my-scripts

My Own Thoughts/Opinions
I feel that the Export/Import Jupyter functionality should be kept fully idempotent. It might make sense to add a test case that checks that running diff on two copies of the same script after Export/Import takes place returns no diffs.
Thanks for the awesome extension!

@MBetters Thanks for filing the issue! We rely on nbconvert when importing notebooks, but we will check to make sure we aren't doing additional processing.

Thanks for the feedback! However, we don't have plans on adding this functionality at this time.


Add an empty rakefile

Since someone at Google hooked up travis improperly, all pull requests
are marked as failing.

I'll make a PR later today adding linting so we actually have something to run for travis :)


Unable to send notifications with DatabaseChannel

Error while sending notifications with Adonis notifications package.
Getting this error:
TypeError: notifiable.routeNotificationFor is not a function
    at DatabaseChannel.send (C:\Users\Ethiel ADIASSA\Desktop\Projets\smartjobs\node_modules\adonis-notifications\src\Channels\DatabaseChannel.js:8:23)

i make sure everything is well set from my side.

have you installed https://github.com/enniel/adonis-lucid-polymorphic
that's the requirements


SentryLaravelEventHandler does not use \Illuminate\Contracts\Events\Dispatcher

I want to work with the sentry/sentry-laravel package and the fntneves/laravel-transactional-events, however the latter provides a different event dispatcher implementation than the default laravel \Illuminate\Events\Dispatcher so the two packages are incompatible.
If the SentryLaravelEventHandler::subscribe and SentryLaravelEventHandler::subscribeAuthEvents methods accepted the \Illuminate\Contracts\Events\Dispatcher contract instead of the \Illuminate\Events\Dispatcher implementation I believe these packages would be compatible.
Is there any chance this could be changed?

Hey @srottem are you already on 0.9.0?
This version included a change to depend on the contract instead of the implementation for this exact reason (see #107).

Doh.  I was on 0.8.0 - apologies.

No problem ofcourse, it was only released a few days ago 😎 glad that fixes it for you 👌


Repository => Gateway

We renamed Repository to Gateway so this PR makes the rename here too

Cool, Thanks!


Expose transformer implementation as a public lib

Polymer would like to create a single transformer which includes both the web_components and reflectable transformers. This is almost possible today except for one small mismatch, which is that reflectable doesn't know how to deal with html entry points. If the transformer implementation was exposed more directly I could create a simple wrapper around it which does know how to deal with them without requiring reflectable itself to take a dependency on package:html in order to support it.

Ideally it would actually expose something which just took a single entry point, a resolver, and some options and returned the new entry point in either Asset or String form.

Actually I am going to close this for now, I can work around this for polymer.


Images and data text files being saved empty.

Hello,
I ran the Examples_Of_Table_To_Image_Conversion.py file and it produced the expected output images. I now am attempting to create my own images, but when I add my own test data, and follow the same steps as Examples_Of_Table_To_Image_Conversion.py, it produces empty images and data text files. The script runs otherwise as expected, and produces all other images and text files correctly with no error. Is there a way I can debug what is going on with the algo?

For those of you in the future reading this, this happens when your total feature # does not match the colsrows number. my feature list was including my index accidentally and thus was rowscols + 1.


feat(facade): add facade for freeze and getCellData

close #xxx

Pull Request Checklist

[ ] Related tickets or issues have been linked in the PR description (or missing issue).
[ ] Naming convention is followed (do please check it especially when you created new plugins, commands and resources).
[ ] Unit tests have been added for the changes (if applicable).
[ ] Breaking changes have been documented (or no breaking changes introduced in this PR).

I'll add
setFrozenColumns(columns) void Freezes the given number of columns.
setFrozenRows(rows) void Freezes the given number of rows.
getFrozenColumns() Integer Returns the number of frozen columns.
getFrozenRows() Integer Returns the number of frozen rows.


fast switch language layouts

At the moment I'm having trouble switching languages. Is there a quick way to switch the language? I have to go to the layout settings to select the desired language.
Can you add a button or double tap on the circle or a hold effect to select preselected language layouts?

Dupe of #308 and related to #437.

That's something which is planned...


Race in PoolArena.allocate

It appears there might be a race in io.netty.buffer.PoolArena.allocate, where allocationsTiny is incremented.  While the value is incremented under a lock, it also appears to be mutated elsewhere in the file without a lock.  A race detector suggested the two places where the class accesses this field in an unsynchronized way:
  Read of size 8 at 0x7f2f7be16108 by thread T53 (mutexes: write M95278):
    #0 io.netty.buffer.PoolArena.allocate(Lio/netty/buffer/PoolThreadCache;Lio/netty/buffer/PooledByteBuf;I)V (PoolArena.java:198)
    #1 io.netty.buffer.PoolArena.reallocate(Lio/netty/buffer/PooledByteBuf;IZ)V (PoolArena.java:359)
    #2 io.netty.buffer.PooledByteBuf.capacity(I)Lio/netty/buffer/ByteBuf; (PooledByteBuf.java:120)
    #3 io.netty.buffer.AbstractByteBuf.ensureWritable0(I)V (AbstractByteBuf.java:269)
    #4 io.netty.buffer.AbstractByteBuf.ensureWritable(I)Lio/netty/buffer/ByteBuf; (AbstractByteBuf.java:250)

and
    #0 io.netty.buffer.PoolArena.allocate(Lio/netty/buffer/PoolThreadCache;Lio/netty/buffer/PooledByteBuf;I)V (PoolArena.java:198)
    #1 io.netty.buffer.PoolArena.allocate(Lio/netty/buffer/PoolThreadCache;II)Lio/netty/buffer/PooledByteBuf; (PoolArena.java:133)
    #2 io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(II)Lio/netty/buffer/ByteBuf; (PooledByteBufAllocator.java:262)
    #3 io.netty.buffer.AbstractByteBufAllocator.directBuffer(II)Lio/netty/buffer/ByteBuf; (AbstractByteBufAllocator.java:157)
    #4 io.netty.buffer.AbstractByteBufAllocator.directBuffer(I)Lio/netty/buffer/ByteBuf; (AbstractByteBufAllocator.java:148)

cc: @nmittler & @normanmaurer

Will check...

yep that's a possible race, but it doesn't really happen in practice and I think if it does nobody will notice or care :).
It may happen if two threads have the same arena's assigned to them, and those two threads need to concurrently allocate from two different tiny size classes and there must not be any free regions available in the thread caches for both of those two tiny size classes (same is true for allocationsSmall).
However, by default Netty settings, each thread has its own arena (cores * 2) and almost all tiny allocations should be served by the cache. If it happens it shouldn't matter, cause allocationsTiny and allocationsSmall are just counter variables for stats, that aren't really in use.
Just saying ... still a bug :smile:

@buchgr I agree... I'm not sure if it really worth to care here and take the performance hit. That said maybe we should and just allow to disable recording metrics. WDYT ?

@normanmaurer up to you :-). I think correctness goes over performance and options for disabling/enabling is just adding unnecessary complexity. That being said, I think if the thread caches work correctly (>99.9% hit rate), there will not be a performance impact.
You can also leave this open for me to have a look at in ~4 weeks. I am planning to add some stat reporting to the allocator, to better understand what's happening in there for grpc.

@buchgr let me maybe just fix it for now and we can revisit later. WDYT ?

@normanmaurer sgtm

@buchgr fwiw the race detector found this when running our unit tests, which I believe are not atypical users of netty.  The race detector is similar to valgrind, which reports suspicious memory accesses.  To this end, I don't believe the race is hypothetical.

@carl-mastrangelo do you know which unit test it was? If my understanding of the code is correct, a necessary condition for the race to happen would be that this test uses more than 2 * ncores many event loops? You are right in saying that it's not hypothetical, it can happen, but if it happens nothing happens so to say :). it's just an unused stat counter. I am all for fixing it.

@carl-mastrangelo that would actually be really helpful. If it happens with lte 2 * ncores threads, then this might point at the logic assigning arenas to threads having a bug, which would be important to fix.

@buchgr I found it in this test: https://github.com/grpc/grpc-java/blob/master/interop-testing/src/test/java/io/grpc/testing/integration/Http2NettyLocalChannelTest.java  but It happens for most of the other tests in the same directory.
If it is infact a harmless race, then its probably okay.  I don't think I can configure my race detector to ignore particular races, so fixing this would help de-noise my test runs.

@carl-mastrangelo thanks! I ll take a look. I assume the race detector is google internal?

Yeah, sadly.  Otherwise I would be more forthcoming with details.

Any chance to opensource it ? hint

Am 04.02.2016 um 18:16 schrieb Carl Mastrangelo<EMAIL_ADDRESS>Yeah, sadly. Otherwise I would be more forthcoming with details.
—
Reply to this email directly or view it on GitHub.

Fixed by https://github.com/netty/netty/pull/4974


Wind and water mills

Wind and water mills would look nice, and are part of any older rural landscape.
However, we probably don't want a production chain. So which purpose could mills have?
Ideas:

Could be reqoured to build farms at all. E.g. max #farms = 5 + 5 * #mills

Implemented as mills limiting total population in #82.


radiotap: Remove the word Sanity.

Remove the oppressive word sanity from the file radiotap.c and use more
inclusive term Validity.

Actually this is a very broadly used term for this type of check, so it's unclear what you mean by oppressive. See: sanity check.


[HOLD for payment 2024-11-28] [$250] Attachment - The position of the attachment "Menu" modal is incorrect

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 9.0.49-0
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail: N/A
Issue reported by: Applause - Internal Team
Action Performed:

Navigate to https://staging.new.expensify.com/
Log in with any account
Navigate to any 1:1 DM
Send any video attachment
Send any image attachment
Click on the video to play it
Hover over the video and click on the "Fullscreen" button
Minimize the video
Hover over the image attachment and click on the "..." button
Hover over the video attachment and click on the "..." button

Expected Result:
"Menu" modal should appear close the attachment, but not overlapping any elements
Actual Result:
The position of the attachment "Menu" modal is incorrect. Sometimes it appears far away from the attachment, sometimes it overlaps the quick menu buttons and sometimes it appears on the LHN
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos

Add any screenshot/video evidence

https://github.com/user-attachments/assets/d831f671-2491-4f6c-8a70-363883be190d
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~021846647666370282624
Upwork Job ID:<PHONE_NUMBER>370282624
Last Price Increase: 2024-10-16
Automatic offers:

rojiphil | Reviewer | 104546765

Issue OwnerCurrent Issue Owner: @RachCHopkins

@RachCHopkins  can i start work on this issue.

Contributor details
Your Expensify account email<EMAIL_ADDRESS>Upwork Profile Link: https://www.upwork.com/freelancers/~01e9eca845d1af9067

/assign

I can't reproduce this - it's exactly where it should be every time for me. I'm on v9.0.49-1 so maybe it's been fixed organically.
Can we get a retest, please?

This has been labelled "Needs Reproduction". Follow the steps here: https://stackoverflowteams.com/c/expensify/questions/16989

@siddhakdak The process is for the issue to be triaged before any assignment. Right now we're still working out if it's a problem. If you can reproduce the issue, then that can help move things along towards the proposal stage. Otherwise I'm asking our testers to check anyhow.

BTW @RachCHopkins I easily reproduced it following the steps, are U interested in fixing it
https://github.com/user-attachments/assets/4ff4d0b7-6cb9-48da-aee2-92be0a7a1dd5

Issue is still reproducible
Build v9.0.49-1
https://github.com/user-attachments/assets/b0428d9c-2022-46c7-a5cb-b87390dee15b

Thanks everyone, I think we can confirm it needs fixing now!

Hey! I'm Eto Olei from Callstack. I will like to take a look at this issue. Please assign it to me. Thanks!

Hi there . I has Help Wanted label and I am working on it already

Oops! @shahinyan11 go for it then.

Proposal
Please re-state the problem that we are trying to solve in this issue.
Attachment - The position of the attachment "Menu" modal is incorrect
What is the root cause of that problem?
Here we have a logic to lock and unlock window dimensions. And we should unlock  dimensions when user exit fullscreen mode as described here. But sometimes after exiting fullscreen mode the unlock logic is not called because didScreenReturnToOriginalSize is false.
The didScreenReturnToOriginalSize sometimes  is false because it is the result of the comparison if the lockedWindowDimensionsRef dimensions are equal to the current window dimensions and we set lockedWindowDimensionsRef after that the isFullScreenRef.current becomes true. But sometimes  the activating fullscreen mode works before the setting of lockedWindowDimensionsRef and sometimes after it. When activation fullscreen works faster then we set the full screen dimensions to lockedWindowDimensionsRef and after exiting fullscreen mode the current window dimensions will not be equal to lockedWindowDimensionsRef
What changes do you think we should make in order to solve the problem?
The existence of  didScreenReturnToOriginalSize in this condition does not ensure that the code will always work after exiting fullscreen mode. So that can remove it from condition
if (lockedWindowDimensionsRef.current && !isFullScreenRef.current ) {

What alternative solutions did you explore? (Optional)
We can call unlockWindowDimensions() in this line to ensure that it will be called on exiting fullscreen mode

@shahinyan11 the Help Wanted label means we are inviting proposals, there is no guarantee your proposal will be accepted. You cannot lay "claim" to an issue like this. Please refamiliarise yourself with the Contributor Guidelines.
@teneeto works for an agency contracted to us, and is welcome to take on the job right up until a proposal is accepted, without first submitting a proposal.

@shahinyan11 the Help Wanted label means we are inviting proposals, there is no guarantee your proposal will be accepted. You cannot lay "claim" to an issue like this. Please refamiliarise yourself with the Contributor Guidelines.
@teneeto works for an agency contracted to us, and is welcome to take on the job right up until a proposal is accepted, without first submitting a proposal.

@RachCHopkins Sorry but I do not agree with you. I ready the  Contributor Guidelines and understand that the "Help Wonted" label gives a guarantee for following -  if for resolving issue will use your solution you should be paid.  If not then the contribution is not guaranteed that his proposal will not be used without compensation. Sorry if I am wrong

Hi @RachCHopkins @rojiphil, I got halted on this topic to investigate another issue, but I am checking this now. I will drop updates as I move forward.

Thanks @teneeto, looking forward to it!

@shahinyan11 I don't think you entirely understand the established process. Please take this to the Slack channel if you want to discuss it, as this issue is not really the place for this conversation! 😄

Hey, I'm able to reproduce this, and it certainly has to do with the window dimensions. I had initially come up with a solution, but then i noticed it might not be the best. I'm trying an alternative and will continue with updates here.

How's the alternative plan going @teneeto ?

Hi @RachCHopkins looking at raising PR today. should be good on that soon.

Hi @RachCHopkins, a PR is ready, let's take a look.

@rojiphil I'm just waiting on your checklist here before I issue payment.

BugZero Checklist:

[x] [Contributor] Classify the bug:

Bug classification

Source of bug:

[x] 1a. Result of the original design (eg. a case wasn't considered)
[ ] 1b. Mistake during implementation
[ ] 1c. Backend bug
[ ] 1z. Other:

Where bug was reported:

[x] 2a. Reported on production
[ ] 2b. Reported on staging (deploy blocker)
[ ] 2c. Reported on a PR
[ ] 2z. Other:

Who reported the bug:

[ ] 3a. Expensify user
[ ] 3b. Expensify employee
[ ] 3c. Contributor
[x] 3d. QA
[ ] 3z. Other:

[x] [Contributor] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake.
Link to comment:

[x] [Contributor] If the regression was CRITICAL (e.g. interrupts a core flow) A discussion in #expensify-open-source has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner.
Link to discussion: Not Required. The existing checklist is good enough to capture such issues.

[x] [Contributor] If it was decided to create a regression test for the bug, please propose the regression test steps using the template below to ensure the same bug will not reach production again.

Regression Test Proposal
Precondition:
Test:

Navigate to any 1:1 DM
Send any video attachment
Send any image attachment
Click on the video to play it
Hover over the video and click on the "Fullscreen" button
Minimize the video
Hover over the image attachment and click on the "..." button
Hover over the video attachment and click on the "..." button
Verify that the popover menu position is correct with respect to the report actions in steps 7 and 8

Do we agree 👍 or 👎

I'm just waiting on your checklist here before I issue payment.

@RachCHopkins Sorry for the delay here. I have completed the BZ checklist here. Thanks.

Payment Summary:

Contributor: @teneeto paid via contract
Contributor+: @rojiphil paid $250 via Upwork

Upwork job here

Contributor has been paid, the contract has been completed, and the Upwork post has been closed.


Use conditional numpy dependencies for Python 2/3

Fixes downgrades under Python 3
poetry uses the information in the pypi json API at https://pypi.org/pypi/nmslib/json to decide that nmslib requires nmslib < 1.17.
requires_dist currently comes out as
    "requires_dist": [
      "pybind11 (>=2.2.3)",
      "psutil",
      "numpy (<1.17,>=1.10.0)"
    ],

which causes poetry to downgrade numpy even for Python 3, which in my case forces a downgrade of another package, which results in dependency resolution failing.
This change should prevent it specifying numpy<1.17 for Python 3.
Will now come out as
    "requires_dist": [
      "pybind11 (>=2.2.3)",
      "psutil",
      "numpy (<1.17,>=1.10.0) ; python_version=='2.7'"
      "numpy (>=1.10.0) ; python_version>='3.5'"
    ],

Hopefully doesn't break anything I don't know about...

Hi @jjjamie sorry for the delay. Could you resubmit your request to develop, I should merge it ASAP.

No problem (imagine you have a lot of other things on) - thanks for looking. Have resubmitted as https://github.com/nmslib/nmslib/pull/446


i am new to chatbot . after executing rasa init command i executed rasa shell where i am getting NLU model is loaded type the message to parse it, but when i type the message i am not getting any replay from bot.

Help me in this..

nlu is just to analyze inputs not reply. try rasa shell
also these kind of questions are better asked on the forum. this is for rasa engine questions, not so much rasa usage.
https://forum.rasa.com/

Thanks for the issue, @JustinaPetr will get back to you about it soon!
You may find help in the docs and the forum, too 🤗


Add DF Bot a common web scraper bot that steals websites content

Add DF Bot a common web scraper bot that steals websites content.

Coverage increased (+0.02%) to 99.97% when pulling 322c85d3fc2722ee95c809bdb74ae8b081916363 on ayumi-cloud:df-bot into 880b9fa797401d14b28956442944c3daa70240ff on WhichBrowser:master.


On tenant restart, the cache entries are not refreshed

try to reproduce
/cc @hannan-azam

@sebastienros i reproduced the error and found two issues with Output Cache. Following are the steps.
Note: OutputCache and ImportExport should be enabled. Set Default Cache Duration to some longer period of time like one hour etc.
I am also attaching two Recipes. Recipe1 Recipe2
Recipe1 imports a test page of type Page and a html widget in Default Layer.
Recipe2 modifies the content of html widget and appends a string "12345" in its content
Issue 1: On Import of recipe UpdateShell restarts the tenant which expires the TagCache but the OutputCache doesn't expire

Evict all cache in Output cache
Import the recipe "Recipe1". A new page will be imported with url /test-page
In anonymous mode request the page /test-page through browser to create a cache entry
Ensure that the widget content is displayed on page and that a cache entry is created in Output Cache.
Import the recipe "Recipe2". It will modify the widget content and append "12345" in its content
In anonymous mode again request the page /test-page It will still show the old widget content where as it should show the fresh content.
Possible Solution: Implement IShellDescriptorManagerEventHandler in Orchard.OutputCache. On Changed event, remove all cache for that specific tenant.

Issue 2: Removing the item doesn't expire OutputCache

Remove the item.
OutputCache entry is still visible in Admin utility
Possible Solution: Implement OnRemoved and OnDestroyed event and remove cache for that specific content.

I think I understand, the default cache provider uses HttpContext.Cache to store the entries, hence they are not released. This could be seen as a feature too, so probably the solution is to store the tags in the same medium.
For the second issue I filed a separate bug.


[nrf fromlist] boot: zephyr: reset SPLIM registers before boot

to avoid causing stack overflow in the image being booted.
PR: https://github.com/JuulLabs-OSS/mcuboot/pull/824
Ref: NCSDK-6530
Signed-off-by: Håkon Øye Amundsen<EMAIL_ADDRESS>

cherry-picked here: https://github.com/nrfconnect/sdk-mcuboot/pull/120

already included by merge of #120


Quantum Adapter Dev docs

Quantum Adapter dev docs

Related PR: prebid/Prebid.js#2051

The adapter has been merged and will be part of 1.5


Update documentation

Documentation updated to refer to the leader key by '<Leader>', instead of its default value ''. Fixes #34.

Thanks Phil.


fatal error: mesh_msgs/TriangleIndices.h: No such file or directory

Hello, I highly appreciate your work. While replicating your work, I encountered the following issues:
Fatal error: mesh_msgs/GetUUID.h: No such file or directory
Fatal error: mesh_msgs/TriangleIndices.h: No such file or directory
Could you provide the corresponding GetUUID.srv and TriangleIndices.msg files?

I will close this issue due to inactivity

Hi USTCerzmj,
thanks for your interest. Could you provide some more details about your setup?

Are you installing from source or do you use the precompiled binaries (apt install ros-*-mesh-tools)?

Which operating system do you use?

Which ROS version do you use?

If you compile from source:

Which branch of mesh_tools do you use?

Furthermore, it would be very helpful if you post all of your error message.
Greetings Alex

same problem
ubuntu20.04    ROS noetic    noetic branch

Hello, I highly appreciate your work. While replicating your work, I encountered the following issues:
Fatal error: mesh_msgs/GetUUID.h: No such file or directory Fatal error: mesh_msgs/TriangleIndices.h: No such file or directory Could you provide the corresponding GetUUID.srv and TriangleIndices.msg files?

兄弟怎么解决的


addon manager - incorrectly says app needs upgrade

I ran
clib -upgradeall
and
clib -upgrade beta1
the siteproperty is 5.00.00006177
clib -version  is 5.00.00006177

clib.exe needs to copy new DLLs to app's bin folder for aspx pattern.
copy by hand until clib.exe is fixed.