None gradient from GRU

Hello everybody,
Introduction:
I am trying to reproduce the work of Szegedy et al: Intriguing properties of neural networks and Dezfooli et al: Universal adversarial perturbations. What they do, briefly, is that they take the gradient g of the network with respect to the input image x, then slightly modify the input image x "against" the obtained gradient, i.e. x-0.001g. The modified images are consequently considered as another network input, hence new gradients g' might be computed and obtained. The result gradients are manually set to the average of those two set obtains gradients, i.e. g/2 + g'/2. I hope I explained that clearly.
I successfully implemented it for images in TF 0.10 (working on 0.12RC as well). Now I want to do something similar with recurrent networks (sentiment analysis task). So I have a input sequence of tokens (numbers), a variable embeddings which is used by tf.nn.embedding_lookup. After the lookup, the sequence of proper embeddings is traversed by GRU (dynamically). It's last state is considered as the representation of the sequence. Then a MLP with a single hidden layer and ReLU activation is applied, resulting in 2 neurons representing the positive and negative logits respectively. The loss is traditional categorical cross entropy.
Bug
Similarly to the papers mentioned above, I obtain gradients of the embedding, and modify the current embeddings slightly "against" the obtained gradient. Than I compute the new loss and new gradients (here is the problem!) and update the network by their mean.
The problem is that the returned new gradients are all Nones. I originally worked with TF 0.10, however, I got an error explaining that second order derivatives are impossible to obtain from scan. I upgraded to recently released TF 0.12RC which doesn't throw this error but returns Nones instead.
Related Issues and SOs
#783 is somewhat similar, but not the same. This question deals with very the same problem  but the solution isn't sufficient as replacing the None value with zero is really worthless in my case since I need to modify the variable against the gradient.
Minimal Not-Working Example
I am sorry the code is so long. It is partially caused by the fact that the problem is non-trivial and requires few lines and partially by the amount of comments I tried to write in order to make the code more readable. All platform/version information is stored at beginning of the file.
The main() function constructs two SimpleSentiments as the models that are trained. Each one is trained for few epochs and batches. First instance is without the adversarial, hence we perform no trick regarding modifying the gradients (for a sanity check). The second uses the complicated if branch in the constructor and obtains Nones as described above.
I believe this is a bug and not my mistake, however, it can't be ruled out as I am not as experienced with TF as I'd like to be.
# #!/usr/bin/env python3

# python 3.5 (anaconda)
# TF 0.12 RC from https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.12.0rc0-cp35-cp35m-linux_x86_64.whl
# $ python -c "import tensorflow; print(tensorflow.__version__)"
#  0.12.0-rc0

# OS: RedHat 7.2
# CPU version only, no GPU, no CUDA, no CUDNN

import numpy as np
import tensorflow as tf
from tensorflow.python.ops.rnn_cell import GRUCell

BATCH = 5  # batch size
MAX_LEN = 10  # max length of the sequence
MLP_HIDDEN_DIM = 128  # number of hidden neurons in the MLP
EMBEDDING_DIM = 300  # embedding dimension
VOCAB_SIZE = 8  # vocabulary size

THREADS = 4  # number of threads to be used
STD=0.001  # standard deviation of ariable initializers

class SimpleSentiment:
    def __init__(self, adversarial=False, device='/cpu:0'):

        self.embeddings = tf.get_variable('word_embeddings',
                                          initializer=tf.random_uniform([VOCAB_SIZE, EMBEDDING_DIM], -1.0, 1.0))

        with tf.variable_scope('sentiment') as scope:
            with tf.device(device):
                # Inputs
                self.text = tf.placeholder(tf.int32, [BATCH, MAX_LEN])
                self.text_len = tf.placeholder(tf.int32, [BATCH])
                self.sentiment = tf.placeholder(tf.int32, [BATCH])

                # Normal loss
                loss_normal = self._loss(self.text, self.text_len, self.sentiment)

                # Define the optimizer
                # Note: I've tried multiple of optimizers and none helped
                optimizer = tf.train.AdamOptimizer(learning_rate=0.0005)

                if adversarial:  # Define adversarial loss
                    # Let's acces already defined variables
                    scope.reuse_variables()

                    # Gradients of all variable (according to normal loss)
                    gradients = optimizer.compute_gradients(loss_normal)
                    print(len(gradients), gradients)

                    # gradients of the embeddings
                    emb_gradient = optimizer.compute_gradients(loss_normal, [self.embeddings])[0][0]

                    # this how much we want to shift the embeddings, i.e. going "against" the gradient
                    delta = 0.001*tf.sign(emb_gradient)

                    # let's compute the loss once again but this time we add the delta to the embeddings
                    loss_adversarial = self._loss(self.text, self.text_len, self.sentiment, delta)

                    # new gradient of the whole computational graph
                    adversarial_gradients = optimizer.compute_gradients(loss_adversarial)
                    print(len(adversarial_gradients), adversarial_gradients)  # everything is None!

                    # Now we compute an average of old and new gradients
                    new_gradients = [((g + ag)/2, vg) for ((g, vg), (ag, avg)) in zip(gradients, adversarial_gradients)]

                    # and apply them
                    self.training = optimizer.apply_gradients(new_gradients)

                    # Btw this doesn't work either
                    # self.training = optimizer.apply_gradients(adversarial_gradients)

                    self.loss_final = (loss_normal + loss_adversarial) / 2

                else:  # Normal loss
                    # simply minimize according to the gradients
                    self.loss_final = loss_normal
                    self.training = optimizer.minimize(loss_normal)

                # Create the session
                self.session = tf.Session(config=tf.ConfigProto(inter_op_parallelism_threads=THREADS,
                                                                intra_op_parallelism_threads=THREADS,
                                                                allow_soft_placement=True))

                # init everything (still deprecated way)
                self.session.run(tf.initialize_all_variables())

    def _loss(self, text, text_len, sentiment, emb_delta=0):
        # use embedding
        # note that emb_delta is zero as long as adversarial=False
        # if adversarial=False then each embedding is shifted by appropriate emb_delta
        text = tf.nn.embedding_lookup(self.embeddings + emb_delta, text)

        # run gru
        gru_cell = GRUCell(MLP_HIDDEN_DIM)
        outputs, state = tf.nn.dynamic_rnn(cell=gru_cell,
                                           inputs=text,
                                           sequence_length=text_len,
                                           dtype=tf.float32)

        # define MLP
        W1 = tf.get_variable(name='MLP_W1',
                             shape=[state.get_shape()[1], MLP_HIDDEN_DIM],
                             initializer=tf.random_normal_initializer(mean=0, stddev=STD))
        W2 = tf.get_variable(name='MLP_W2',
                             shape=[MLP_HIDDEN_DIM, 2],
                             initializer=tf.random_normal_initializer(mean=0, stddev=STD))
        h1 = tf.get_variable(name='MLP_h1',
                             shape=[MLP_HIDDEN_DIM],
                             initializer=tf.random_normal_initializer(mean=0, stddev=STD))
        h2 = tf.get_variable(name='MLP_h2',
                             shape=[2],
                             initializer=tf.random_normal_initializer(mean=0, stddev=STD))

        # apply MLP of the last GRU state
        after_first_layer = tf.nn.relu(tf.matmul(state, W1) + h1)
        logits = tf.matmul(after_first_layer, W2) + h2

        # compute loss via categorial cross entropy
        loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits, sentiment))
        return loss

def main():
    for adversarial in [False, True]:
        print('\n=================================')
        if adversarial:
            print('Using the adversarial loss')
        else:
            print('Using the standard loss')

        net = SimpleSentiment(adversarial=adversarial)

        for epoch in range(5):
            print('Epoch {}'.format(epoch))

            for batch in range(3):
                _, loss_final = net.session.run([net.training, net.loss_final],
                                                {net.text: np.array([[3, 1, 1, 2, 1, 0, 0, 0, 0, 0],
                                                                     [3, 4, 1, 2, 1, 4, 4, 0, 0, 0],
                                                                     [1, 1, 1, 2, 0, 0, 0, 0, 0, 0],
                                                                     [3, 3, 3, 2, 1, 7, 0, 0, 0, 0],
                                                                     [7, 1, 5, 2, 4, 2, 2, 2, 1, 7]], dtype='int32'),
                                                 net.text_len: np.array([5, 7, 4, 6, 10], dtype='int32'),
                                                 net.sentiment: np.array([0, 0, 1, 1, 0], dtype='int32')})

                print('\tBatch {}: {}'.format(batch, loss_final))

if __name__ == '__main__':
    main()

Thanks for help
Petr

@rmlarsen @vrv Could this be related to a known missing 2nd order derivative?

I'm am also getting None gradients when trying to compute second order derivatives of a GRU.

If I revert to tensorflow 0.10 I get the error message:
TypeError: Second-order gradient for while loops not supported.

The error disappears if I switch from using tf.nn.dynamic_rnn to tf.nn.rnn, but this appears to slow things down quite a bit.

@vladfi1 Does it really work for you with tf.nn.rnn? I updated my code so that is uses tf.nn.rnn, i.e. used tf.split on the input tensor in order to support it. However, the gradients are still Nones and I get a really weird error AttributeError: 'NoneType' object has no attribute 'pred'.
I attach the code here in a gist so that it doesn't mess with the original post. I still use TF 0.12. but I suppose it is back-compatible.
Thanks for help.

@petrbel It runs fine for me with tensorflow 0.10.0.

I'm not sure, since this uses dynamic_rnn, I'm assigning to @ebrevdo .

@vladfi1 It is as you said, it runs fine with 0.10. However I still think it is an issue as it doesn't work 0.12.
Petr

tf.while_loop definitely doesn't support second derivatives.  You should have received an error instead of None.  Which seems like a bug. If you only needed to use the first derivative values as values and didn't mean to differentiate them, wrap them in tf.stop_grdients to avoid trying to calculate their derivatives in your second gradients calculation.

@ebrevdo Will this change in the future? If while_loop has a first derivative, then that first derivative itself should probably itself be a while_loop (this is probably where I'm mistaken), in which case there's no reason not to take the derivative again.

That is the general idea, but the implementation is complex enough, for
many reasons, that we do not plan to implement a second derivative.
On Dec 16, 2016 6:50 PM, "Vlad Firoiu"<EMAIL_ADDRESS>wrote:

@ebrevdo https://github.com/ebrevdo Will this change in the future? If
while_loop has a first derivative, then that first derivative itself should
probably itself be a while_loop (this is probably where I'm mistaken), in
which case there's no reason not to take the derivative again.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/5985#issuecomment-267743388,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtim8l9aUZijyU3mq36EhhAGSYNvq3Eks5rI2owgaJpZM4LAZNz
.

@ebrevdo: you mean in its current formulation, right?  That is, if we redesigned how while_loop was implemented, it could be done more easily?

I don't know.  The current implementation requires a lot of special casing
and careful graph introspection to properly identify which tensors should
stay in the stack for gradient purposes, and for e.g. handling of external
and within body stop gradient calls.  There is also a lot of special
handling in the runtime.  Compound that with nested while loop handling and
possibly needing to modify TensorArray to support second+ derivatives...
At the very least, first we should move while loop and gradients to the c++
api, and decide whether we want full Jacobian support, then we can consider
this.
On Dec 17, 2016 7:43 AM, "Vijay Vasudevan"<EMAIL_ADDRESS>wrote:

@ebrevdo https://github.com/ebrevdo: you mean in its current
formulation, right? That is, if we redesigned how while_loop was
implemented, it could be done more easily?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/5985#issuecomment-267776014,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtim7vVjKRbtSsZ7jpoveltRo7CjeOMks5rJB8ngaJpZM4LAZNz
.

Is it possible to compute second order derivatives with tf.nn.rnn if I switch to 0.10, or does it silently suppress any errors? cc @petrbel @vladfi1 @ebrevdo

regarding second or derivatives, i think we fixed bugs from 0.10 to 1.0 and
in the nightlies; so definitely use that instead.
keep in mind the following ops don't have second derivatives:
tf.nn.sparse_softmax_cross_entropy_with_logits
tf.nn.softmax_cross_entropy_with_logits
so if you're using those for your loss, you'll have trouble.  implement
them yourself using tensorflow primitives and you'll be able to use second
derivatives.
On Wed, Apr 5, 2017 at 10:09 AM, Ankesh Anand<EMAIL_ADDRESS>wrote:

Is it possible to compute second order derivatives with tf.nn.rnn if I
switch to 0.10, or does it silently suppress any errors? cc @petrbel
https://github.com/petrbel @vladfi1 https://github.com/vladfi1
@ebrevdo https://github.com/ebrevdo
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/5985#issuecomment-291929879,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtim5Rc1STYBZLGoOOc8fwAz7FY_wy0ks5rs8q6gaJpZM4LAZNz
.

Slightly confused here, you mean I should use 1.0? I am already using that and it doesn't allow me to compute second order derivatives for RNNs

Do you have some example code and the error you get?

The relevant portion is this:
gradients = tf.gradients(discriminator(x_dash, reuse=True)[0], [x_dash])[0]
The discriminator here is a simple LSTM implemented using tf.nn.dynamic_rnn.
The error I get is: TypeError: Second-order gradient for while loops not supported.
I am trying to implement the gradient penalty term from this paper: https://arxiv.org/abs/1704.00028 (Improved training of WassGANs)

Ah.
That's right, tf.while_loop (used by dynamic_rnn) does not support second
derivatives; and there are no plans for support in the near or mid-term
future.  I'm afraid if you want this, you'll have to use a static RNN.
On Wed, Apr 5, 2017 at 5:18 PM, Ankesh Anand<EMAIL_ADDRESS>wrote:

The relevant portion is this:
gradients = tf.gradients(discriminator(x_dash, reuse=True)[0],
[x_dash])[0]
The discriminator here is a simple LSTM implemented using
tf.nn.dynamic_rnn.
The error I get is: TypeError: Second-order gradient for while loops not
supported.
I am trying to implement the gradient penalty term from this paper:
https://arxiv.org/abs/1704.00028 (Improved training of WassGANs)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/5985#issuecomment-292033647,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtimxZMeUPnCyEaRmHnKQAQOh6Q0z5Hks5rtC9SgaJpZM4LAZNz
.

@asimshankar @skye I do know there's been work to move While loops into C API already, but I don't know how gradient support within it works, if at all yet.  Just an FYI of something that would be great to have, though I don't know the scope of the work required.

@ebrevdo static_rnn doesn't work either, gives me nan losses.

@ankeshanand I think it might work in 0.10/0.11, but I don't remember for sure. I just ended up writing my own unroller:
def rnn(cell, inputs, initial_state):
  inputs = tf.unpack(inputs, axis=1)
  outputs = []
  state = initial_state
  for input_ in inputs:
    output, state = cell(input_, state)
    outputs.append(output)
  outputs = tf.pack(outputs, axis=1)
  return outputs, state

This assumes the time axis is 1. In more recent tensorflow versions, [un]pack has been renamed to [un]stack.

@ankeshanand https://github.com/ankeshanand are you passing the
sequence_length= argument to the static rnn call?
On Thu, Apr 6, 2017 at 2:43 AM, Vlad Firoiu<EMAIL_ADDRESS>wrote:

@ankeshanand https://github.com/ankeshanand I think it might work in
0.10/0.11, but I don't remember for sure. I just ended up writing my own
unroller:
def rnn(cell, inputs, initial_state):
inputs = tf.unpack(inputs, axis=1)
outputs = []
state = initial_state
for input_ in inputs:
output, state = cell(input_, state)
outputs.append(output)
outputs = tf.pack(outputs, axis=1)
return outputs, state
This assumes the time axis is 1. In more recent tensorflow versions,
[un]pack has been renamed to [un]stack.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/5985#issuecomment-292122739,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtim0rbxiPCNfdmX3QipRXehOp8B-JSks5rtLO4gaJpZM4LAZNz
.

@ebrevdo I didn't, I realized from the docs that passing sequence_length would again result in dynamic computation.
Even the authors of the paper Improved WassGan faced this issue it seems (https://www.reddit.com/r/MachineLearning/comments/63dfun/r170400028_improved_training_of_wasserstein_gans/dftqi72/)

Any updates on this thread?

Any update? Is there already any way of computing second order gradients for dynamic_rnn modules?

I'm not aware if the second derivative of while_loop has been added yet. However, the static RNN solution isn't convenient for experimentation when there are longer time-steps [I needed about 1k steps and graph creation takes more than 5 minutes!]. So, I wrote code to compute both the derivatives and activations as part of the SAME FORWARD while loop using some Dynamic Programming and Math! However, I believe it would have been much more efficient if this had been native.
Posting the code (adequately commented) for reuse for anyone else who lands on this page with the same problem. Note that the following code is only for computing the (jacobian) equivalent of tf.gradients(hidden_activations,input_vectors). It is easy to derive the end-to-end gradients using the same idea that is implemented.
I have tested this with TF 1.7.0.
def get_outputs_and_gradients(rnn_cell,steps,embedding_size,hidden_size,inputs):
    """
    :param rnn_cell: Assumes GRU_CELL.
    :param steps: Integer
    :param embedding_size: Integer
    :param hidden_size: Integer
    :param inputs: The inputs to the RNN of shape (time_steps,embedding_size). (Can be extended to batch computations, if needed.)
    :return:
    """
    time = tf.constant(0)
    hidden_states = tf.constant(value=np.zeros(shape=(steps+1,hidden_size)),dtype=tf.float64)
    hidden_input_jacob = tf.constant(value=np.zeros(shape=(steps,embedding_size,hidden_size)),dtype=tf.float64)
    input_grad = tf.constant(value=np.zeros(shape=(steps,embedding_size,hidden_size)),dtype=tf.float64)

    def _body(time,hidden_states,hidden_input_jacob,input_grad,ins):
        """
        :param time: Current time step of RNN
        :param hidden_states: The array containing all the hidden state activations.
        :param hidden_input_jacob: The array containing the jacobian between activations of the latest time step and all other time steps
        :param input_grad: The gradients accummulated so far.
        :param ins: The inputs.
        :return:
        """
        inputs = ins[time:time+1] # 1 x embedding_size
        prev_state = hidden_states[time:time+1] # 1 x hidden_size
        state = prev_state

        # forward
        gate_inputs = tf.matmul(
            tf.concat([inputs, state], 1), rnn_cell._gate_kernel)
        gate_inputs = tf.nn.bias_add(gate_inputs, rnn_cell._gate_bias)

        value = tf.sigmoid(gate_inputs)
        r, u = tf.split(value=value, num_or_size_splits=2, axis=1)

        r_state = r * state

        candidate = tf.matmul(
            tf.concat([inputs, r_state], 1), rnn_cell._candidate_kernel)
        candidate = tf.nn.bias_add(candidate, rnn_cell._candidate_bias)

        c = rnn_cell._activation(candidate)
        new_h = u * state + (1 - u) * c

        # Jacobian Computations
        start = tf.eye(hidden_size, dtype=tf.float64)
        du = tf.multiply(start, (state))
        dstate = tf.multiply(start, (u))
        du = du - tf.multiply(start, (c))
        dc = tf.multiply(start, (1 - u))
        dcandidate = (dc * tf.gradients(c, candidate))[0]
        dinputs_rstate = tf.matmul(dcandidate, tf.transpose(rnn_cell._candidate_kernel))
        dinputs = dinputs_rstate[:, :embedding_size]
        drstate = dinputs_rstate[:, embedding_size:]
        dstate = dstate + r * drstate
        dr = state * drstate
        dru = tf.concat([dr, du], axis=1)
        dgateinputs = dru * (value * (1 - value))
        dinputs_state = tf.matmul(dgateinputs, tf.transpose(rnn_cell._gate_kernel))
        dinputs = dinputs + dinputs_state[:, :embedding_size]
        dstate = dstate + dinputs_state[:, embedding_size:]

        mask = tf.equal(tf.range(start=0, limit=steps+1), time+1)
        hidden_states = tf.where(mask, (hidden_states + new_h) , hidden_states)

        # jacobs
        hid_hid_jac = dstate  # hidden_size x hidden_size
        hid_inp_jac = tf.transpose(dinputs) # embedding_size x hidden_size

        mask = tf.equal(tf.range(start=0, limit=steps), time)
        hidden_input_jacob = ndmatmul(hidden_input_jacob,tf.transpose(hid_hid_jac))
        hidden_input_jacob = tf.where(mask, (hidden_input_jacob + hid_inp_jac), hidden_input_jacob)
        input_grad = input_grad + hidden_input_jacob

        return [time+1,hidden_states,hidden_input_jacob,input_grad,ins]

    outs = tf.while_loop(cond=lambda time, *_: time<steps,
                  body=_body,
                  loop_vars=[time, #time. Useless after iteration
                             hidden_states, # time_steps,embedding_size
                             hidden_input_jacob,# time_steps,embedding_size,hidden_size. This is useless after the iteration.
                             input_grad, # time_steps,embedding_size,hidden_size. This contains the jacobians. tf.reduce_sum(outs[3],axis=[2]) == tf.gradients(self.hidden_states,inputs)
                             inputs # time_steps,embedding_size. Unchanged during iteration.
                             ],
                  parallel_iterations=10)

    return outs[1],outs[3]

def ndmatmul(mat3d,mat2d):
    return tf.einsum("abc,cd->abd",mat3d,mat2d)

(I'll defer to @ebrevdo for an update on that, but would expressing the model with eager execution be an option that works for you? That may be more natural and will work with higher order gradients. See the instructional notebooks).

@ebrevdo @asimshankar Has any progress been made on taking 2nd order gradients in dynamic_rnn ?

There is work in progress to implement while loops using different
primitives. These should allow 2nd order gradients.  Unfortunately there's
no estimate when this will be ready.
On Sun, Aug 19, 2018, 5:46 PM pythonanonuser<EMAIL_ADDRESS>wrote:

@ebrevdo https://github.com/ebrevdo @asimshankar
https://github.com/asimshankar Has any progress been made on taking 2nd
order gradients in dynamic_rnn ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/5985#issuecomment-414168300,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtim1D_HUm-Jfr9sZI0qwXt3t7IEzKdks5uSgbbgaJpZM4LAZNz
.

@ebrevdo Is there a branch or PR you can point me to? Perhaps I can help contribute

@skye Thanks have emailed you!


Exposures with libcamera-still are very different from raspistill.

libcamera-still images are much darker in low light than raspistill images.  The images below were taken about 20 minutes apart in late afternoon with some overcast.  The libcamera-still image is from a Java motion detection program and that is why I'm holding the piece of paper in front of the lense.  The second image is using a Java program to capture from raspistill.
RPi3B+, RPi camera version 2.1, up to date bullseye with recently compiled libcamera, libcamera-apps and userland.

Better pictures.  The raspistill image was taken 5-10 minutes later than the libcamera-still image.

This is not entirely unexpected, the libcamera stack uses an entirely different set of algorithms and image quality tuning parameters when compared with the old stack.  The advantage of libcamera is that everything is open source, so you are free to tweak the settings to match your application.  For your case, you might want to have a look at the AGC/AEC section of our camera tuning guide here.
A possibly simpler way would be to use the --ev command line argument to increase the metering brightness.  This controls the Exposure Value (EV) in the AGC algorithm.  A value of +1 gives 2x the relative brightness from EV 0.  Likewise a value of -1 would give 1/2x of the relative brightness from EV 0.

Is there more to discuss on this issue?  If not, I will close this down.

Closing this now.  Please do reopen if there are further discussions needed.

Sorry to come back to this but I think there are two issues here.  The first being the significant light sensitivity difference between libcamera-still and raspistill and an apparent lack of what I can only describe as dynamic range (I'm sure there is another name for it) or exposure adjustment as light levels change.
The other is that if run for a period of time there is inadequate correction or a drift in exposure.  I ran my motion detection program from yesterday afternoon to this morning and the image looked really dark but the light outside was not that low.  I stopped and restarted the program and the image was much brighter.
See this post too:  https://forums.raspberrypi.com/viewtopic.php?t=328083

Hi, did you try those suggestions from a week or so ago? I would recommend adding --framerate 0. If you want it even brighter (and have the latest libcamera-apps code) you could also add --exposure long.
The question about the exposure getting "stuck" is puzzling. When I first read about it I thought it might be because the timelapse interval is so short that it never gets to run any preview frames in between. But when I tried it, it seems it does still adapt (though slowly, as it should). Are you able to bump up the timelapse interval so that we're guaranteed a few preview frames?
If that doesn't explain it we can always turn on the AGC debug, though it sounds like that would get quite long.
As regards that forum post, it looks to be using OpenCV so I guess it must be using the proprietary Broadcom stack, which works totally differently and shares nothing in common. So I'm not really sure if it's at all related. Trying to figure out what that stack is even trying to do is quite difficult...

Sorry to be slow getting back to this.  I did try using --framerate 0 and it did not appear to have any impact.  I did try another test, I took a picture with the same Pi and same camera every three minutes alternating between raspistill and libcamera-still.  Selected images here: http://knutejohnson.com/cameratest1.  These were taken without the --framerate 0 and the camera program was run for each photo.  I will run these again this evening and tomorrow morning with --framerate 0 and --exposure long and see how it goes.
What would be a good timelapse interval to get the preview frames?  Do I need to enable preview?

Thanks for posting those. It would be interesting to see some of the exif data for some of those images, specifically the exposure time and ISO. It's a bit strange how the difference seems more pronounced as they're getting darker. Is it the case that, when those images are rather dark, that stopping and doing a single "standalone" capture would be brighter, do you think? I'll try and do some experiments myself too.
As regards a good timelapse gap - I've actually just pushed a branch to the repo called "timelapse-fix" which enforces half a dozen preview frames between timelapse captures, however small the requested interval. You could give that a go and see if it makes any difference.

Is it the case that, when those images are rather dark, that stopping and doing a single "standalone" capture would be brighter, do you think? I'll try and do some experiments myself too.

  I start the program and capture the first image out of the stream.
I looked at the Exiftool and it doesn't give the exposure or ISO.  How do find that information?

For the last test I used a Java program to capture the image and save it as a file.  Turns out when the image data is converted to a Java image not all of the EXIF data is saved.  Tomorrow I'll try using just the programs to save the files and we'll get the complete EXIF.

Two new sets of images.  http://knutejohnson.com/cameratest2 are images from the stdout converted to Java images.  These have framerate 0 and exposure long.  Not much different from the ones that didn't have those options in cameratest1.
http://knutejohnson.com/cameratest3 are images taken directly with raspistill and libcamera-still and saved to a file.  Both raspistill and libcamera-still were fresh compiles yesterday.  Mouse over the image for a tooltip with the EXIF data for that image.  raspistill required a 2000ms timeout or the image was severely washed out (not sure what's up with that).  Command lines are on the top of the web page.
I've discovered that when you read a JPEG file and create a Java BufferedImage with that data and subsequently save it as a JPEG file again, that the EXIF data in the image is lost.
I hope this is somewhat helpful.
This is the kernel in my Pi3 after an rpi-update: Linux raspberrypi4B 5.10.92-v7l+ #1514 SMP Mon Jan 17 17:38:03 GMT 2022 armvl.  It appears to be the same kernel that is current.  Does this kernel have the fix for the pink images?

Hi, thanks very much for posting those two sets of images. I think they're very helpful, especially that second set where the EXIF pops up when you hover!
Anyway, broadly speaking I thought the exposures were relatively comparable. Looking at the "cameratest3" set first, I thought perhaps the biggest difference might be in images 1820-raspicam.jpg and 1822-libcam.jpg, though it didn't seem anywhere near as huge as you were seeing in those earlier image sets. Looking at the EXIF data:
1820-raspicam.jpg has an exposure of 1/8s and ISO of 320. We believe raspicam counts a gain of 1 as being about ISO 54, so this makes the gain here about 6x.
1822-libcam.jpg has an exposure of 1/8s and an ISO of 600. Looking at imx219.json, I think the exposure for "long" mode tops out at 120ms, so the EXIF seems to be approximating slightly there. In this case the code takes ISO 100 as a gain of 1, so the gain here is 6x too, and the ISO difference is not anything "real".
There is certainly some variation here in these images, I don't think that's too surprising, but overall I don't think I'm seeing too much wrong. If you wanted the libcamera images to be a bit brighter in those dark conditions, you could increase the maximum gain in the imx219.json tuning file. For imx219 you could go up to 10x. If you wanted all the images brighter (specifically the ones that are not being limited by the darkness) then you could dial in some extra EV on the command line (e.g. --ev 0.5). Does that help enough?

I think with the --framerate 0 and the --exposure long that they are close enough.  I'll try playing with the options you suggested to see if I can get a little better result.
I ran another whole set of images this weekend with libcamera running continuously using the STDOUT option.    You can see those at http://knutejohnson.com/cameratest4.  It's sunny today, I'll run another set of those this evening and tomorrow morning.  I was hoping with this latest set of images to capture the fade out that I have seen in the past.    That program uses the same scheme to capture images as the set in cameratest4.
I really appreciate the time you have taken to help me get this running better.  Thanks!

Well, thanks for taking the time to help us figure out most of the problems, it's much appreciated. I guess I'll close this one now, but if you want any more advice on anything, please don't hesitate to post back or create a new issue!


class ArrayImpl has no to_py

In your file car/driving.py, I got an error: ArrayImpl object has no attribute to_py. These errors are in line 396, 408, 520, 521, 558, 559
Idk for sure why the errors happened but I fix it by use np.array instead of to_py.
For example: p.to_py( ) becomes np.array(p).
Hope it could help someone.

Thank you for pointing the fix to this error: I was using an older version of jax at the time (jax 0.2.11 and jaxlib 0.1.63). to_py() is deprecated since jax 0.3.17.
Note that it is recommended to use np.asarray(p) instead of np.array(p) to avoid copying the array in memory, see https://github.com/google/jax/issues/1961#issuecomment-572306453.


Masters - Tornament Leaderboard

Tournament Leaderboard data provided by Master's API

Created the view for desktop, need to adjust based on browser size


Proposal - GitHub Repo Files (Independent Publisher)

When submitting a connector, please make sure that you follow the requirements below, otherwise your PR might be rejected. We want to make you have a well-built connector, a smooth certification experience, and your users are happy :)
If this is your first time submitting to GitHub and you need some help, please sign up for this session.

[X ] I attest that the connector doesn't exist on the Power Platform today. I've verified by checking the pull requests in GitHub and by searching for the connector on the platform or in the documentation.
[ X] I attest that the connector works and I verified by deploying and testing all the operations.
[ X] I attest that I have added detailed descriptions for all operations and parameters in the swagger file.
[ X] I attest that I have added response schemas to my actions, unless the response schema is dynamic.
[ X] I validated the swagger file, apiDefinition.swagger.json, by running paconn validate command.
[X ] If this is a certified connector, I confirm that apiProperties.json has a valid brand color and doesn't use an invalid brand color, #007ee5 or #ffffff. If this is an independent publisher connector, I confirm that I am not submitting a connector icon.

If you are an Independent Publisher, you must also attest to the following to ensure a smooth publishing process:

[X ] I have named this PR after the pattern of "Connector Name (Independent Publisher)" ex: HubSpot Marketing (Independent Publisher)
[ ] Within this PR markdown file, I have pasted screenshots that show: 3 unique operations (actions/triggers) working within a Flow. This can be in one flow or part of multiple flows. For each one of those flows, I have pasted in screenshots of the Flow succeeding.
[ X] Within this PR markdown file, I have pasted in a screenshot from the Test operations section within the Custom Connector UI.

[ N/A] If the connector uses OAuth, I have provided detailed steps on how to create an app in the readme.md.

@microsoft-github-policy-service agree [company="{none}"]
Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: microsoft-github-policy-service[bot] @.>
Sent: Saturday, March 9, 2024 8:53:18 AM
To: microsoft/PowerPlatformConnectors @.>
Cc: Nathalie Leenders @.>; Mention @.>
Subject: Re: [microsoft/PowerPlatformConnectors] Requesting new custom connector - github as a datasource (PR #3290)
@Nathalie-Leendershttps://github.com/Nathalie-Leenders please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]
Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"
Contributor License Agreement
Contribution License Agreement
This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your
contributions to Microsoft open source projects. This Agreement is effective as of the latest signature
date below.

Definitions.
“Code” means the computer software code, whether in human-readable or machine-executable form,
that is delivered by You to Microsoft under this Agreement.
“Project” means any of the projects owned or managed by Microsoft and offered under a license
approved by the Open Source Initiative (www.opensource.orghttp://www.opensource.org).
“Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
Project, including but not limited to communication on electronic mailing lists, source code control
systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
discussing and improving that Project, but excluding communication that is conspicuously marked or
otherwise designated in writing by You as “Not a Submission.”
“Submission” means the Code and any other copyrightable material Submitted by You, including any
associated comments and documentation.
Your Submission. You must agree to the terms of this Agreement before making a Submission to any
Project. This Agreement covers any and all Submissions that You, now or in the future (except as
described in Section 4 below), Submit to any Project.
Originality of Work. You represent that each of Your Submissions is entirely Your original work.
Should You wish to Submit materials that are not Your original work, You may Submit them separately
to the Project if You (a) retain all copyright and license information that was in the materials as You
received them, (b) in the description accompanying Your Submission, include the phrase “Submission
containing materials of a third party:” followed by the names of the third party and any licenses or other
restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
guidelines concerning Submissions.
Your Employer. References to “employer” in this Agreement include Your employer or anyone else
for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
Submission is made in the course of Your work for an employer or Your employer has intellectual
property rights in Your Submission by contract or applicable law, You must secure permission from Your
employer to make the Submission before signing this Agreement. In that case, the term “You” in this
Agreement will refer to You and the employer collectively. If You change employers in the future and
desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
and secure permission from the new employer before Submitting those Submissions.
Licenses.

Copyright License. You grant Microsoft, and those who receive the Submission directly or
indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
parties.
Patent License. You grant Microsoft, and those who receive the Submission directly or
indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
Your patent claims that are necessarily infringed by the Submission or the combination of the
Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
import or otherwise dispose of the Submission alone or with the Project.
Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
granted by implication, exhaustion, estoppel or otherwise.

Representations and Warranties. You represent that You are legally entitled to grant the above
licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
have disclosed under Section 3). You represent that You have secured permission from Your employer to
make the Submission in cases where Your Submission is made in the course of Your work for Your
employer or Your employer has intellectual property rights in Your Submission by contract or applicable
law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
You later become aware that would make Your representations in this Agreement inaccurate in any
respect.
Information about Submissions. You agree that contributions to Projects and information about
contributions may be maintained indefinitely and disclosed publicly, including Your name and other
information that You submit with Your Submission.
Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
defenses of lack of personal jurisdiction and forum non-conveniens.
Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
supersedes any and all prior agreements, understandings or communications, written or oral, between
the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.

—
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/PowerPlatformConnectors/pull/3290#issuecomment-1986915903, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3N5N4PKGMSKADNPH64NNRTYXM475AVCNFSM6AAAAABEOHSGHWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWHEYTKOJQGM.
You are receiving this because you were mentioned.Message ID: @.***>

@microsoft-github-policy-service agree
Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: microsoft-github-policy-service[bot] @.>
Sent: Saturday, March 9, 2024 9:07:50 AM
To: microsoft/PowerPlatformConnectors @.>
Cc: Nathalie Leenders @.>; Mention @.>
Subject: Re: [microsoft/PowerPlatformConnectors] Requesting new custom connector - github as a datasource (PR #3290)
@Nathalie-Leendershttps://github.com/Nathalie-Leenders the command you issued was incorrect. Please try again.
Examples are:
@microsoft-github-policy-service agree
and
@microsoft-github-policy-service agree company="your company"
—
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/PowerPlatformConnectors/pull/3290#issuecomment-1986919239, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3N5N4PWZXKYIBCECSDJRZLYXM6WNAVCNFSM6AAAAABEOHSGHWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWHEYTSMRTHE.
You are receiving this because you were mentioned.Message ID: @.***>

Hi!
I did, what is missing from the request? It's my first time so I probably did something wrong somewhere.
I appreciate any help you can provide.
Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: vmanoharas @.>
Sent: Tuesday, March 12, 2024 7:45:18 AM
To: microsoft/PowerPlatformConnectors @.>
Cc: Nathalie Leenders @.>; Mention @.>
Subject: Re: [microsoft/PowerPlatformConnectors] Requesting new custom connector - github as a datasource (PR #3290)
@vmanoharas commented on this pull request.
Hello @Nathalie-Leendershttps://github.com/Nathalie-Leenders,
Can you please go through this document and submit your Independent Publisher Connector?
https://learn.microsoft.com/en-us/connectors/custom-connectors/certification-submission-ip
—
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/PowerPlatformConnectors/pull/3290#pullrequestreview-1931283671, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3N5N4NYS75S6KO4AW5RWKTYX4IH5AVCNFSM6AAAAABEOHSGHWVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSMZRGI4DGNRXGE.
You are receiving this because you were mentioned.Message ID: @.***>

Hi @Nathalie-Leenders, Greetings from the airport.
Let me try to assist with some suggested changes to resolve issues I see in your submitted PR:

Please change your PR title to Proposal - GitHub Repo Files (Independent Publisher).
When you believe all issues have been resolved, remove Proposal - from the title.
In apiProperties.json, please change:
-- Per the February change for OAuth, please update the redirect Model to GlobalPerConnector. The following line 11 for redirectURL is then unnecessary.
-- Please change the OAuth identity provider to github, as found in the fourth example here
-- Please change stackOwner to GitHub, Inc.
-- Remove the closing curly bracket on the final line and reformat file to correct JSON tab spacing.
In apiDefinition.swagger.json, please change:
--   For all strings, double quotation marks (") are used, not single quotation marks('). Please change all.
-- There is a mix of YAML and JSON in this file. All JSON object key lines do not start with "- ", e.g. line 36: - "name": ... Also, all JSON object lines need to end with a comma. The pac CLI should have caught those errors, but I would suggest VS Code or an online JSON validator to fix issues.
-- Line 19 for consumes should be an array with the single item application/json. The following produces also needs to be an array.
-- The current line 23 has an unused action path - this must be removed.
-- Your action path is set to require both folder name and folder name 2 - If you want both (or neither?), you could create separate actions. Path parameters are always required based on the defined path.
-- Your path parameters need to include both both x-ms-summary and description. Please review this doc.
-- Your response needs to be changed from default to 200. The description can stay the same, and since the response is the markdown file content, you can change the schema to string.
-- At the end of the file, you must also include the connector metadata

After you make those changes and include them in your branch, please let me know and I can review again.
CC: @ShefaaliP

@troystaylor I have made changes to the files, can you please help in checking if they're ok now?

Hello @Nathalie-Leenders,
I am following up on this PR, kindly resolve review comments mentioned by Troy.

@Nathalie-Leenders, I am happy to collaborate on this connector to resolve the comments if you need assistance.

Hi all,
Thank you! Due to life getting crazy busy I had to put this one on the backburner, I will definitely retry in getting my manifest ok and reach out to you Troy if I need any help (which I'm pretty sure I will, but will let you know)
Thanks again for following up
Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Troy Taylor @.>
Sent: Wednesday, June 19, 2024 4:43:37 PM
To: microsoft/PowerPlatformConnectors @.>
Cc: Nathalie Leenders @.>; Mention @.>
Subject: Re: [microsoft/PowerPlatformConnectors] Proposal - GitHub Repo Files (Independent Publisher) (PR #3290)
Hello @Nathalie-Leendershttps://github.com/Nathalie-Leenders,
I am following up on this PR, kindly resolve review comments mentioned by Troy.
@Nathalie-Leendershttps://github.com/Nathalie-Leenders, I am happy to collaborate on this connector to resolve the comments if you need assistance.
—
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/PowerPlatformConnectors/pull/3290#issuecomment-2178886645, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3N5N4PHWXZOPNJ42Q2Y4NLZIGKJTAVCNFSM6AAAAABEOHSGHWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZYHA4DMNRUGU.
You are receiving this because you were mentioned.Message ID: @.***>

Hi @troystaylor and @vmanoharas I think I got it (or at least visual studio code copilot helped) I really hope this is OK, can you please check for me? 🙏 I really appreciate the help, this is really not my forte

Hi @troystaylor and @vmanoharas I think I got it (or at least visual studio code copilot helped) I really hope this is OK, can you please check for me? 🙏 I really appreciate the help, this is really not my forte

Thank you @Nathalie-Leenders, I will wait @troystaylor to respond, if he has any comments before I review your files.

Hi @troystaylor and @vmanoharas I think I got it (or at least visual studio code copilot helped) I really hope this is OK, can you please check for me? 🙏 I really appreciate the help, this is really not my forte

Hi Nathalie,
Happy to provide some assistance here. First thing I did was run the paconn CLI tool (or pac works too) against the apiDefinition.swagger.json file:

You've defined one path parameter, but not the other 3-4. As I mentioned previously, path parameters are required, so with the single action you've defined, users will be required to give a folder and subfolder name, which I wouldn't do. The easiest way would be to give two different actions, with one having the single folder and a second action to put the file in a subfolder. I've also thought about this and you could use custom code if you set the folder name parameters to query parameters (which don't have to be required). You also need to define Reponame and Markdownfile - those are required.
The second major error is the summary length, it cannot be over 80 characters. I would suggest 'Get GitHub file' and 'Get GitHub file in subfolder', fixing the additional error that summaries cannot end in punctuation.
Next, the connector metadata property 'Privacy policy' is missing.
The final error is the missing key for each path parameters - "x-ms-url-encoding": "single"
The CLI tool doesn't catch it, but GitHub should have the correct capitalization for every instance. Another suggestion I would use is to use either camelcase or PascalCase for all your parameter - it makes the code easier to read. I was also thinking about the usefulness of the connector and while I think being able to directly return markdown as a string to Power Automate is useful, I think being able to retrieve any file raw is more useful, so I've changed the produces to application/vnd.github.raw+json.
In looking at your apiProperties file, I don't understand what providing the OAuth setting as part of the URL path is suppose to do. The more I thought about with the host you are using, I realized that you would only be able to retrieve public files, because the OAuth would never be valid against that host - the OAuth settings need to be removed. To make this connector work with OAuth, it needs to properly call the api.github.com host and use the correct endpoint: https://docs.github.com/en/rest/repos/contents?apiVersion=2022-11-28#get-repository-content . I'm going to work on the actions using OAuth and I'll share my suggestions soon.
@vmanoharas the current artifacts are not ready for certification.

Hi Nathalie,
Please review the files I've created that have passed CLI validation: https://github.com/troystaylor/Testing/tree/main/GitHub Data
As I mentioned, this action will work will any public repo. I am going to continue to investigate the endpoints for the Repositories API and will submit them for a separate connector.

I've modified my file, can you please review at your convenience? My past 6 months were a bit crazy but looking forward to getting this published. @vmanoharas


Restrict Protected to some heap types.

This PR restrict Protected types to some heap types.
The goal is to avoid wrongly using Protected over some structure on stack.
I think this should be enough for current usage, going further will probably lead to write https://github.com/iqlusioninc/crates/tree/develop/secrecy .

I had a first implementation with Box but I remember some slight api difference (at least for the sized array case) and did add a new structure for the Box case. Actually looking at 'secrecy' crate from the author of zeroize, that is exactly the three cases that get covered (box, vec and string).
Knowing myself I probably looked at the only usage being Protected<Box> and felt like it was not ok (on a different branch I simply switch Protected to a struct ProtectedString but it was way bigger diff overall).
So at this point, I did drop the box code, I think we are using this 'simple' Protected, Protected is identic code, but Protected<Box> would be adding new code, so if the need arise for it we should ask if we should use Secrecy crate (Protected for the current use looks lighter than a new dependency).
Will be adding some doc.

I think the secrecy crate serves mostly as reminder and grep fodder for whenever you access a secret.  Use it directly maybe?
I'd wait for https://github.com/rust-lang/rfcs/pull/2859 myself before doing anything else.
I think heap vs stack remains unconvincing without further effort, ala mlock, which all sounds like a distraction now.  We've an Arc in the key store which helps avoid keys being cloned much, but obviously a Box does nothing against clone, and Vec even less.

Use it directly maybe?

I consider that, honestly it will not be to much work to replace Protected by Secrecy, but I step back because I was not sure that having to follow a new crate code was worth it. cc @tomusdrw @bkchr what do you think?

I'm fine to use secrecy, the crate is pretty small (you can check the code in like 5 mins). AFAICT the point was that due to rust move semantics, stack-allocated secrets will remain in memory (drop is not being called when you move the value, but the value actually is copied), hence we want to make sure that whatever is Protected is heap-allocated, so that when Protected value is moved around only the memory pointer is being copied. And secrecy doesn't really have anything that supports this pattern, so we would need to wrap it or document correctly  to make sure it's used correctly.

Yes, I suppose more discipline in how substrate handles secrets makes sense.
All rust crypto crates use almost exclusively stack for secrets.  I doubt any would change this until https://github.com/rust-lang/rfcs/pull/2859 and https://github.com/rust-lang/rfcs/pull/2884 works, and maybe someone needs to figure out mlock from rust too, so optimistically a couple years away.  It's rather a different problem in substrate though, so sure..

Last commit simply replace Protected by SecrecyString.

I'd wait for rust-lang/rfcs#2859 myself before doing anything else.

RFC co-author here.
I'm probably not saying anything you don't already know, but I just want to point out that the RFC is dead in the water so far. The lang team hasn't shown any interest yet (that might change with the new MCP process, but I'm skeptical), and until they do there isn't even a guarantee that the RFC will be officialized as something we want in Rust.
So "a couple years away" is indeed optimistic.

@bkchr I did remove the drop on the struct opt parameter, looks better now.

bot merge

bot merge

@cheme will need a master-merge in order to fix cargo-deny error.

bot merge


feature: add weight for node config

Signed-off-by: Starnop<EMAIL_ADDRESS>
Ⅰ. Describe what this PR did
Now we choose the supernodes randomly and fairly. In the actual scenario, the service capabilities of multiple supernodes may be different. And we hope that the machine with good serviceability will be selected more times. So I add a weight configuration for supernode which defined by dfget.
For now, users can specify the supernode with weight like format: host:port/weight. For example:
Using cli flagdfget  --node <IP_ADDRESS>/2 --node <IP_ADDRESS>/1
or using config file
# cat /etc/dragonfly/dfget.yml
nodes:
    - <IP_ADDRESS>/2
    - <IP_ADDRESS>/1

And then the supernode with <IP_ADDRESS> and the supernode with <IP_ADDRESS> will be selected in a 2:1 ratio.
Ⅱ. Does this pull request fix one issue?

NONE.
Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)
The host is necessary,  port and weight is optional
Added
Ⅳ. Describe how to verify it
Take a look at dfclient.log file and search do register to one of
And then you will find something like this
do register to one of [<IP_ADDRESS>:8002 <IP_ADDRESS>:8002 <IP_ADDRESS>:8002 <IP_ADDRESS>:8002 <IP_ADDRESS>:8002 <IP_ADDRESS>:8002]

And the order in which you perform download or from different clients is random each time.
Ⅴ. Special notes for reviews

Codecov Report

Merging #1021 into master will increase coverage by 0.31%.
The diff coverage is 87.23%.

@@            Coverage Diff             @@
##           master    #1021      +/-   ##
==========================================
+ Coverage   46.81%   47.12%   +0.31%
==========================================
  Files         115      115
  Lines        6972     7002      +30
==========================================
+ Hits         3264     3300      +36
+ Misses       3452     3445       -7
- Partials      256      257       +1

Impacted Files
Coverage Δ

dfget/config/config.go
91.86% <100%> (ø)
:arrow_up:

cmd/dfget/app/root.go
75.72% <86.66%> (+8.58%)
:arrow_up:

supernode/daemon/mgr/scheduler/manager.go
22.6% <0%> (+0.68%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6bc6db4...4857671. Read the comment docs.

Nice! For this specific flag type, isn't it better to introduce a new flag.Value for it?

IMOP, it's not user-friendly if there are two flags that can configure nodes. WDYT?

I now feel / might not be a good separator: Imagine that I configure supernode as a backend service for an ingress, it might have an address like https://foo.example.com/supernode.

Yeah, I agree with you. Thanks very much for pointing this out. And how about * as the
separator? In fact, I have no good idea for this at present. 🤕

IMOP, it's not user-friendly if there are two flags that can configure nodes. WDYT?

I didn't mean that. I mean, since this flag is special, I recommend create a specific flag value for it instead of keeping using stringslice. Something like below:
type superNodeVal struct {
    ...
}

// implement flag.Value
func (s *superNodeVal) String() string {
    ...
}

func (s *superNodeVal) Set(string) error {

}

// then use is as:
flag.Var(&superNodeVal{}, "superNodes", ...)

request a review again.
And something that you need to know:
If you use dfget --node <IP_ADDRESS>=1 --node <IP_ADDRESS>=1, and then only the <IP_ADDRESS> will be applied. The last one will override all previous node values. And what's the reason that I don't combine them? In my implementation, I need to calculate the gcd(greatest common divisor) for all nodes. And it's difficult to handle by multi cli params.


No search or browse

No way to get around or find nodes, relatioships, within the network.
There should be a way to search for nodes (with given relations). Or at least nodes and relation names.
Also search within the bodies of annotations.

NodeSearch feature is added and deployed via Heroku.
https://linkyway.herokuapp.com/home


Forecast.io API support

It will be great if someone can add support for The Dark Sky Forecast API. API calls are limited but we can use our own API keys.

If someone is interested in adding this I would definitely accept the pull request.

I'll have a try (:

I want to try too :)

Just wanted to see if anyone has made progress on this before I spec out this job for a contractor.

I've made some progress on my local branch, I'm hoping it have it released this week.

Weather underground support was added in version 6.0, which I believe should cover underlying issue (that OpenWeatherMap data is terrible.) If there is significant enough demand, this issue could be reopened.


Update requirement for .NET Framework

Update requirement for .NET Framework - from 5.4 to 4.5.1

@stasku : Thanks for your contribution! The author, @georgewallace, has been notified to review your proposed change.


Adding Module fallback function

Hi! If you need help, I would like to try implementing a shadow module like this, but rather it senses your errors in the code first, and if there aren't any, the modules get reloaded. The way I would do this, would be to clone the modules, and then delete them, and if it hasn't got any errors, it pastes the modules and tries to see if there is any errors. Hope to get a reply! My email is<EMAIL_ADDRESS>

I don't understand. Is this an issue report for the node-sandboxed-module project, or did you mean to post this somewhere else?

@ev1stensberg I think I see what you have in mind. This module might indeed not be the best place to implement this but you might want to look into Glagol, another sandboxing and live reloading system which has such a feature planned but not yet implemented. Perhaps you could lend a hand?

First: @domenic : It really isn't a issue, rather a suggestion!
@egasimus : Yeah, that would be cool, where do you prefer me to contact you?

And yea, the node-sandboxed-module project was the intended project @domenic

@ev1stensberg drop me a line at<EMAIL_ADDRESS>and let me know if Glagol is making any sense to you :)


Expected code release schedule?

Hi,
Thanks for the wonderful work. Is there a timeline for the expected release date for the code?
Thanks

Sorry for the delay. Now that the paper has been published, I released the code.


Task16

Building a DataTable and printing it in Excel

@gayathrigundaveni
Task -16
Code verified.
Task -17
Code verified.
Task -18
Code verified.


Adds Range and NdRange as supported types in numba_dpex.dpjit.

[X] Have you provided a meaningful PR description?

The numba_dpex.core.kernel_interface.indexers.Range and
NdRange classes can now be used inside a numba_dpex.dpjit
decorated function, as arguments to a dpjit decorated function,
and returned from a dpjit decorated function.
Defines a datamodel, type class, overloads for the constructors,
box, and unbox functions for both classes.

[x] Have you added a test, reproducer or referred to an issue with a reproducer?
[x] Have you tested your changes locally for CPU and GPU devices?
[X] Have you made sure that new changes do not introduce compiler warnings?
[ ] If this PR is a work in progress, are you filing the PR as a draft?
The PR should be merged post 0.21.3 tag

Also NDRange concept still unclear for me.

https://enccs.github.io/sycl-workshop/expressing-parallelism-nd-range/


./euroscope-afv-wine_install.sh: line 188: syntax error near unexpected token `>'

Receiving the above errror when trying to install Euroscope. Wine and required packages installed.
Also raises this:  /euroscope-afv-wine_install.sh: line 151: /usr/local/bin/wine: Bad CPU type in executable
System: macOS Monterey 12.1 (intel machine)
Note: error is most likely user side (on me)

&>> is a syntax introduced in bash 4. I don´t know if this is supported by zsh (default mac shell)
You will need to edit lines 188, 194, 200 in euroscope-afv-wine_install.sh
For example:
"$wineBin" "$@" &>>"$logFilename"
to
"$wineBin" "$@" >> "$logFilename" 2>&1

Sorry, I seem to have missed the earlier notification about this issue.
While I believe that even in a zsh shell that script is run by bash (due to the #!/bin/bash "shebang") if that made problems let's fix it.
Thanks @FValent3 for finding the cause and pointing that out!


Add moko-resources dependencies option also.

Please add moko resources dependencies option also, it's somewhat tricky with version catalogues

I will be happy to see a PR with it. :)
Or you can ask moko authors to add it

I try to help on the wizard... hope the following pr is like you want it :) its a try to add moko mvvm compose https://github.com/terrakok/Compose-Multiplatform-Wizard/pull/33


sort: provide guidance on behavior if data is sorted, or almost-sorted

It's not uncommon to have data that is almost sorted, or some data that is sorted and some data that is not sorted. For example, I was just working with a function that had one unsorted element, either at the beginning or end of an array, that needed to be sorted.
It might be nice to provide some guidance to users about the performance in this case; would Go still issue around n*log(n) calls to sort the data, or would the actual number be closer to n?
If it's not a good fit in the documentation, a blog post explaining the different optimizations in the sort package (if the data is small, we use this algorithm; large, this algorithm, here are the experiments we tried that weren't faster, etc.) might be neat.

I don't think this would be appropriate for the documentation because we can change the algorithm, and in fact we have done so in the past.  A blog post would be fine.

can this be closed?  or is something else needed?

Closed is fine I think


Fixes: #3107

Minor adaptations to the text describing the loop construct, and addition of an example illustrating use of a loop initializer variable in the outer scope after the loop has ended.

Thanks!


BugFix: improve dark mode function

get rid of two useEffect and make it a bit cleaner

Warning: viewport meta tags should not be used in _document.js's . https://nextjs.org/docs/messages/no-document-viewport-meta

Great input. Thank you!

Should be done with #114


Fix grammatical issues in documentation

Changes proposed in this pull request:
Changed "user's voting power" to "users' voting power" (apostrophe position fix for plural possessive)
Changed "users voting power" to "users' voting power" (added missing apostrophe for plural possessive)
Changed "for each voters" to "for each voter" (grammatical correction for singular consistency)
Changed "a IPFS url" to "an IPFS URL" (correct article usage before vowel sound and capitalization)
Changed "a API url" to "an API URL" (correct article usage before vowel sound and capitalization)
Changed "help making" to "help make" (correct infinitive form after 'help')

@thomasscovell


Attach fails in Jest test

Describe the bug
I'm able to attach a file via REPL, but when I put it in a Jest test it fails with the following error:  Invalid element passed as parameter.
Unfortunately, I'm not able to attach the use to or fieldField instead as it doesn't seem to detect it in my app. Here is how the css elements look likes:
To reproduce, try attaching a file via css selector input
`await attach(path.join(__dirname, 'data', 'sample-list.csv'), $('input'));`

Paste any log or error messages here

Expected behavior
Test case should behave the same as in REPL.
Screenshots
REPL command:

Jest test:

Versions:

Taiko: 1.22.10
OS [e.g. 10.15.6 (19G2021) please be specific ] macOs Big Sur 11.4
Node.js [e.g. v.12.18.14] 14.16.1
Jest latest

This usually happens if Taiko does not find a element matched in the selector $('input'). It may have worked in the REPL because the site would have rendered the input element and not rendered it on headless mode.
Can you try using a filefield?

I've tried with fileField, but it doesn't seem to detect it as such. Here is the html for this component:

Can you share the modified code snippet that you are using to attach the file?

The id of the class is generated dynamically, so I can't put the full name. But I've noticed that for click it works even with partial class names.

I was running the test without headless mode.

Please try
await attach(path.join(__dirname, 'data', 'sample-list.csv'), to(fileField(near('Drag and drop....'))));

Please note use the full text. If the text Only .csv file types... is visible try using that.

Still doesn't work, but I was able to make it run using this:
await attach(path.join(__dirname, 'data', 'sample-list.csv'), to(fileField(below("Upload a file you wish to import."))));
Thanks for your help and prompt responses :)


Neus Training failed on Blender Dataset downloaded from the nerfstudio website

Hello, I'm trying to run neus on the blender dataset (ficus and drums more specifically). When running on Ficus, I see relatively reasonable result in the first few evaluation iterations, but later at step 2500, I get this error for the accumulation image. Does this mean the training failed? What could be the reason for it? My training command is ns-train neus --vis tensorboard blender-data --data ../data/blender/ficus -- makes me wonder if sdfstudio support blender dataparser? If not, what's a good way to convert it to a format sdfstudio supports? Thank you in advance!
Traceback (most recent call last):   File "/home/zling/anaconda3/envs/sdfstudio/bin/ns-train", line 8, in <module>     sys.exit(entrypoint())   File "/home/zling/Documents/projects/shellnerf/sdfstudio/scripts/train.py", line 250, in entrypoint     main(   File "/home/zling/Documents/projects/shellnerf/sdfstudio/scripts/train.py", line 236, in main     launch(   File "/home/zling/Documents/projects/shellnerf/sdfstudio/scripts/train.py", line 175, in launch     main_func(local_rank=0, world_size=world_size, config=config)   File "/home/zling/Documents/projects/shellnerf/sdfstudio/scripts/train.py", line 90, in train_loop     trainer.train()   File "/home/zling/Documents/projects/shellnerf/sdfstudio/nerfstudio/engine/trainer.py", line 174, in train     self.eval_iteration(step)   File "/home/zling/Documents/projects/shellnerf/sdfstudio/nerfstudio/utils/decorators.py", line 70, in wrapper     ret = func(self, *args, **kwargs)   File "/home/zling/Documents/projects/shellnerf/sdfstudio/nerfstudio/utils/profiler.py", line 43, in wrapper     ret = func(*args, **kwargs)   File "/home/zling/Documents/projects/shellnerf/sdfstudio/nerfstudio/engine/trainer.py", line 349, in eval_iteration     metrics_dict, images_dict = self.pipeline.get_eval_image_metrics_and_images(step=step)   File "/home/zling/Documents/projects/shellnerf/sdfstudio/nerfstudio/utils/profiler.py", line 43, in wrapper     ret = func(*args, **kwargs)   File "/home/zling/Documents/projects/shellnerf/sdfstudio/nerfstudio/pipelines/base_pipeline.py", line 312, in get_eval_image_metrics_and_images     metrics_dict, images_dict = self.model.get_image_metrics_and_images(outputs, batch)   File "/home/zling/Documents/projects/shellnerf/sdfstudio/nerfstudio/models/base_surface_model.py", line 486, in get_image_metrics_and_images     acc = colormaps.apply_colormap(outputs["accumulation"])   File "/home/zling/Documents/projects/shellnerf/sdfstudio/nerfstudio/utils/colormaps.py", line 42, in apply_colormap     assert image_long_min >= 0, f"the min value is {image_long_min}" AssertionError: the min value is -9223372036854775808

Here are the evaluation images from step 2000.

Adding the flags in this issue's training command helped.


Not technically a bug but driving me crazy.

When interacting with any of Zenith tables if I am holding piece of armor it equips it.

That does sound annoying, but I don't know exactly how to fix that... hmm
I'll look into it, but no promises on finding anything

After further play I can confirm its not just armor but anything that has a use. For example if I have a spectrum knowledge gem in the inventory slot that I am using interacting with zenith tables the interaction is also having you interact with the knowledge gem to causing it to give you the experience in the gem like a continuous right click. I hope that info helps.

That does sound annoying, but I don't know exactly how to fix that... hmm
I'll look into it, but no promises on finding anything

most likely what is happening is you're not cancelling the right click action before opening the screen

To clarify, I was just using the knowledge gem as an example of what it happening being not only an action but a persistent interact of the right click because it is in continuous use. The Spectrum knowledge gem only gives
experience back when actively being right clicked with.

That does sound annoying, but I don't know exactly how to fix that... hmm
I'll look into it, but no promises on finding anything

most likely what is happening is you're not cancelling the right click action before opening the screen

Ah yeah that was it, was PASS instead of CONSUME
I feel really dumb now lol
Fixed in the next version


feat(events): Refactor Functions into Event Handlers

This PR implements:

[x] Remove help command
[x] Create events namespace and move all event-related logic there
[x] Rename "function" to "event handler"
[x] Make handlers no longer required for intialization

Breaking changes

The help command has been removed - it is no longer useful because commands can describe themselves.
Bot functions can no longer receieve a name and a description.
BotFunction is now BotEventHandler and is passed to the initialization method as handlers instead of functions

:tada: This PR is included in version 3.0.0-beta.6 :tada:
The release is available on:

npm package (@beta dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 3.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


feat(Algebra/Group/Units): Add a lemma isUnit_map_iff'

Add a lemma isUnit_map_iff'.

isUnit_map_iff is already declared as
@[simp]
theorem isUnit_map_iff (f : R →+* S) [IsLocalRingHom f] (a) : IsUnit (f a) ↔ IsUnit a

Can you start a zulip thread suggesting that we generalize IsLocalRingHom to IsLocalMonoidHom or something? It should work fine on MonoidHom too...

This PR is obsolete due to #6045 . This theorem is achieved by stating isUnit_map_iff for every IsLocalHom f, and add the instance for any MulEquiv being IsLocalHom.


NoMethodError: undefined method `discard_all_from_jobs' for SolidQueue::FailedExecution:Class

https://app.honeybadger.io/fault/118296/e20dd1c4fb08900f8b53b2d17734bad6

Hey @devaniljr, like with https://github.com/basecamp/mission_control-jobs/issues/52, this should be solved by updating solid_queue.

It worked :)


Enhanced Coresight SSL check

-SSL check is now done on Web Site and Web App level
-Issuer of the SSL certificate is now checked (self-signed vs issued by a CA), although there could be cases where this check gives false positive.
-Severity of this check is based on whether Basic Authentication is enabled or not

Some bugs need to be squashed first :)


No lasers in Space Block R2

I recently started playing this modpack called Space Block R2, which runs Mekanism, MekanismTools and MekanismGenerators all v<IP_ADDRESS>1. In this modpack there aren't any lasers to start a fusion reactor. Is this because of the modpack or is it because there are no lasers in this version?

What did the modpack author(s) say when you first brought this to their attention?

I tried to contact them multiple times, but they only responded just now. They have indeed removed the items, and did not mention this in any banned or removed item lists. The problem has been solved.

I take it there isn't an other way of jumpstarting the reactor? I built it all in survival except for the lasers, and it is quite a hard task to do in a SkyBlock environment, to say the least.

I looked around on the forums of the modpack authors, they are a group called OhGaming, and I only found threads that said they were disabled in other modpacks by the same authors, and none were saying anything about them being disabled on Space block. I have started a new thread, but no response yet.

Did you checked mekanism.cfg in your minecraft instance folder/config for B:LaserEnabled=true


Swagger-UI not display paramters by the correct order as define in controller method

using spring boot + swagger , maven project
springfox artifacts

io.springfox
springfox-swagger2
2.9.2

        <dependency>
            <groupId>io.springfox</groupId>
            <artifactId>springfox-swagger-ui</artifactId>
            <version>2.9.2</version>
        </dependency>

spring boot version - (v2.1.6.RELEASE)
it seems that swagger by default ignore paramters order define on controller method , and by defualt sort paramers order by paramter name (alphabetically)
I think that that the correct approch is first display paramters order as they define on controller method , as fallback to that you can add postion argument to swagger annotation / or enable some configuration to disable this default alphabetically sorting .
method on spring boot controller

result on swagger-ui.html

as you can see swagger ui html not display the paramter order as they define on controller method

https://github.com/springfox/springfox/issues/2418

Duplicate of # 2705


Inexplicable errors with mixture of imports, typedefs, mixins, and callbacks

This test has three files.
test1.dart:
import 'test2.dart';
class ConcreteClass extends Object with MixIn { }
void main() {
  new ConcreteClass().test();
}

test2.dart:
import 'test3.dart';
typedef _unused();
class MixIn {
  var test3 = new Test3(() { });
  void test() {
    test3.test();
  }
}

test3.dart:
class Test3 {
  final fn;
  Test3(this.fn);
  void test() {
    fn();
  }
}

If you run this by just calling `dart test1.dart' with the current stable SDK (1.13), on Linux, you get:
'file:///.../test1.dart': error: line 4 pos 22: unexpected token '.'
  new ConcreteClass().test();

As far as I can tell, the error occurs when the parser is trying to parse the anonymous closure passed as argument to Test3 on this line:
var test3 = new Test3(() { });
For some reason, the parser thinks this closure is in script test1.dart instead of test2.dart, and it obviously finds unexpected tokens at the correct token position, but in the wrong script.
The closure is question has the internal name Object&MixIn_Object&MixIn._
I am reassigning to Matthias.

I have run into this same issue and it appears to only reproduce if you have the mixin defined in a different file.  It works in one file as you can see in dartpad.  If you split it into a separate file, it seems to cause a parsing issue and gives seemingly unrelated error which makes debugging the issue almost impossible.  See mixin_issue.zip - if you remove the import of 'mixme.dart' and put the mixin definition in main.dart, it will run.  Otherwise you get something like this:
$ dart main.dart
'mixin_issue/main.dart': error: line 4 pos 23: unexpected token ';'
	var test = new Test();
                      ^

@crelier Based on the workaround which was identified by @a-siva and your analysis it is pretty clear that the closure is being created with the wrong owner class. I am wondering whether this is a problem only in field initializers or whether this is a more common problem.
@Hixie regarding the pattern of using an empty closure by writing it out everywhere such as "() {}": Not sure whether this is only in the example code provided or generally used, but it would be beneficial to allocate a single closure object and assign it to a static. This way you only define a single function instead of at every source location where you repeat that pattern.

@iposva-google Thanks. In this case it was just for the example (the original code is a sorting function for a HeapPriorityQueue).

https://codereview.chromium.org/1533823004/

Committed patchset #2 (id:20001) manually as
8293e9665b8cf08615098e61236097bfccdedcfd (presubmit successful).


⚠ Bofrost.at has degraded performance

In d2bee32, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1938 ms

Resolved: Bofrost.at performance has improved in 3bcef53 after 6 minutes.


Drag gestures not recognized in web

I have a GestureDetector that detects pointer drags.  It is not detecting drag events now.
$ flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel master, v1.12.16-pre.35, on Mac OS X 10.14.6 18G1012, locale en-US)
[✓] Chrome - develop for the web
[✓] Android Studio
[✓] IntelliJ IDEA Community Edition (version 2019.1.3)
[✓] Connected device (4 available)

Closing as duplicate of https://github.com/flutter/flutter/issues/45510


Add (token) symbol to "lockingPendingRewards"

Please add the (token) symbol to the "lockingPendingRewards" query/output.

closed until we move further with locking


Mobile app to update delivery

I am not sure this is right issue,
We should have mobile app/website module,
which will update delivery status.
please provide links as solution

webapp? build with react,vue,anglar?

Personally, I've had bad experiences trying to make Angular sites that also interact with .NET sites/code/projects and have them be stables, fast, and work properly.  I would recommend either a Xamarin project so the app could work on UWP, Android, and iOS or a Blazor project; this is a website built using just .NET and can be setup to use server-side rendering which makes it more secure and reduces the load on mobile devices.

@dmarciano I had no problem at all, just make sure to use Typescript and some generator for your models and APIs 🙂... SPAs don't care what language you use for backend (C#, golang, rust, python)...
Blazor is still in an early stage so I wouldn't use it before the first stable client-side release. Xamarin is horrible to work with, sorry.
For native mobile app use flutter/react native or something else that's not C# ;) It's a more pleasant experience with a much more active community.

Blazor IMO is stable enough to use.  I've built an entire, professional website with it which was used for contractors to create blueprints with project estimates, invoicing, client/project management, QuickBooks/Stripe integration along with a SQL and MongoDB.  Blazor basically just compiled down to JS and uses SignalR for real-time communication, and can be designed either as a client-server model, REST service, or server-side rendering.


Fixed the issue with the Search example not working

I have updated the Search Autocomplete example which stopped working as the words don't have unique diffIdentifier's. This caused by the check that good introduced to ensure all objects assigned to a section controller are unique. Fixed this issue by removing all the double words.
Checklist

[X] All tests pass. Demo project builds and runs.
[ ] I added tests, an experiment, or detailed why my change isn't tested.
[ ] I added an entry to the CHANGELOG.md for any breaking changes, enhancements, or bug fixes.
[X] I have reviewed the contributing guide

thanks for doing this! i noticed this on Friday and discussed with @rnystrom. we agreed that going with a Set instead of Array for the var words would solve this and ensure no individual words are lost.

Why would that fix the issue? Would that cause different diffIdentifier values or is the check skipped when using a Set? But I am happy to change it!

it would fix the issue because a Set would only have one instance of each word, and since the word itself is used as the diff identifier there would be no duplicate identifiers. :)

Oh right, somehow I thought you wanted to keep the double words 🙈

@amonshiz Fixed it 👍


Avoid old time dep from chrono

By default, chrono includes the feature oldtime which adds an extra dependency on an old version of the time crate. Since this crate only uses chrono::Utc:now, disabling this feature should be safe.

Approving CI, ty

Great stuff, thanks for the contribution! :tada:


Instantiating unboxing stubs target lookup

unboxing->target info to be encoded in the gcinfo portion (flag + reloc)

@fadimounir Is this done?

Yes. Closing the issue.


Set expiration date for service request

Automatically set expiration date to service request based on configuration parameter

[x] Create service request

Config parameter SERVICE_REQUEST_EXPIRATION_DAYS is added to manage expiration period


Typed query params

Now we match query parameters like this
 TypedParams <$ lit "typedParams" <*> paramInt "i" <*> paramBool "is" <*> paramNum "n"`

And create custom typed decoder using paramWith

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

This looks like a pretty good addition to me. Anyone have any comments?
I think it'd perhaps be nicer to simplify paramWith to deal with Either? I would think the kind of parsing done in there typically would be simple enough that the extra structure in V (Free MatchError) isn't required.
paramWith :: forall a. (String -> Either MatchError a) -> String -> Match a

We are getting a bit of a repetition here (look at toNum, toBool, ... and num, int, bool matchers)
Ideally same toNum, toBool, ... functions could be reused in definition of num, int, bool and param*. for example:
lit :: String -> Match Unit
lit input = eitherMatch $ path ExpectedPathPart <#> \i -> if i == input then Right unit else Left $ UnexpectedPath input

str :: Match String
str = path ExpectedString

path :: MatchError -> Match String
path err = Match \route ->
  case route of
    Cons (Path input) rs ->
      pure $ Tuple rs input
    _ ->
      invalid $ free err

num :: Match Number
num = eitherMatch $ path ExpectedNumber <#> asNum >>> note ExpectedNumber
paramNum :: Match Number
paramNum = paramWith $ asNum >>> note ExpectedNumber

bool :: Match Boolean
bool = eitherMatch $ path ExpectedNumber <#> asBool >>> note ExpectedBoolean
paramBool :: Match Boolean
paramBool = paramWith $ asBool >>> note ExpectedBoolean

We can do this in different PR too.


Not working on Windows 10 21H2 build 19044.1682

The package is not working since I updated my windows to this build, I received this message in keypiranha console
22:24:07.127 ** ERROR: Error from WindowsApps.WindowsApps.on_catalog: <class 'OSError'>: [WinError 87] The parameter is incorrect
Traceback (most recent call last):
File "D:\System\Keypirinha\portable\Profile\InstalledPackages\WindowsApps.keypirinha-package\windowsapps.py", line 173, in on_catalog
File "lib\subprocess.py", line 775, in init
File "lib\subprocess.py", line 1178, in _execute_child

I have no idea what's causing this.
According to https://bugs.python.org/issue35678 the current working directory can be too long.
Can you open the Keypirinha Console (shortcut F2) and run the following:
import os; len(os.getcwd())

It should output 20 in your case. (Then length of D:\System\Keypirinha)

According to https://github.com/py2exe/py2exe/issues/95 or https://github.com/MrSharkSpamBot/ShadowSharkReverseShell/issues/1 it seems to be a problem, if the Microsoft Store version of python is installed.

I'm sorry, apparently the error was caused by comodo. I reset the comodo after updating windows which made keypirinha not working properly. Thank you

Thanks for reporting back.


Move ws-python to ws-python-django

Django isn't Python. It makes more sense to have -django appended because it shows that it isn't regular Python.

This is why it shows as python and django in the UI. Changing the image name now would require a lot of work, and only be visible on the backend. I wish we'd chosen the correct names at the start, but that is why we have contribution guidelines now :)

Understood. Do you know when the Django k logo is going to show up on C9.io? It's present in this repo but isn't shown there.

I made a PR for that the other day, it should be out soon; the wrong logo there was a mistake.

Okay.


Roku cast videos not working

Since October 31, I can't stream videos on Roku. I see just a black screen for 0.5 seconds and after that the media player closes

Yes. You are right. I fixed this problem. I create custom channel and it works good.

Yes. You are right. I fixed this problem. I create custom channel and it works good.

Can you share it?

Yes. You are right. I fixed this problem. I create custom channel and it works good.

Can you share it?

Yes. You are right. I fixed this problem. I create custom channel and it works good.

Can you share it?

default player is working now.
image fix:
replace NSString *applicationPath = [NSString stringWithFormat:@"15985?t=p&u=%@&tr=crossfade", to
NSString *applicationPath = [NSString stringWithFormat:@"15985?t=p&u=%@&h=%%20&k=%%20",
video fix:
applicationPath = [NSString stringWithFormat:@"15985?t=v&u=%@&k=(null)&videoName=%@&videoFormat=%@"
to
applicationPath = [NSString stringWithFormat:@"15985?t=v&u=%@&h=%%20&k=%%20"

@dylancom ,
Roku device not connect.
Any solution?


Does not work without OPENAPI_KEY in environment

Installation -
pip3 install dbt-doc-py
DBT environment -
link to repo
Error log-
`dbt-doc-py
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Running DBT docs generate...
DBT docs successfully generated!
You haven't specified an API Key. No worries, this one's on TextQL!
In return, please type your email address. We don't collect any other data, nor sell your email to third parties.
If you're okay with this, press enter. Otherwise, type 'no' and set the OPENAI_API_KEY environment variable.
Email (type no to abort<EMAIL_ADDRESS>[?] Select the models you want to document: (Press Spacebar to select, Enter to confirm) :

[X] model.magicschool.my_third_dbt_model
[ ] model.magicschool.my_second_dbt_model
[ ] seed.magicschool.customers
[ ] model.magicschool.my_first_dbt_model

Generating docs for: my_third_dbt_model
'UserInfo' object has no attribute 'key'
'UserInfo' object has no attribute 'key'
'UserInfo' object has no attribute 'key'
'UserInfo' object has no attribute 'key'
'UserInfo' object has no attribute 'key'
'UserInfo' object has no attribute 'key'
'UserInfo' object has no attribute 'key'
'UserInfo' object has no attribute 'key'
OAI request to my_third_dbt_model failed: 'UserInfo' object has no attribute 'key'
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/bin/dbt-doc-py", line 33, in
sys.exit(load_entry_point('dbt-doc-py==0.1.19', 'console_scripts', 'dbt-doc-py')())
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/dbt_doc_py/dbt_doc_py.py", line 864, in run_async_main
asyncio.run(async_main(sys.argv[1:]))
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/dbt_doc_py/dbt_doc_py.py", line 858, in async_main
await main(argv)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/dbt_doc_py/dbt_doc_py.py", line 841, in main
summarized_nodes = await asyncio.gather(*[open_ai_summarize(env, r_deps, pair[1]) for pair in nodes_to_process])
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/dbt_doc_py/dbt_doc_py.py", line 322, in open_ai_summarize
raise e  # Reraise the exception to trigger a retry
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/dbt_doc_py/dbt_doc_py.py", line 304, in open_ai_summarize
tbl_result, col_result = await asyncio.gather(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/dbt_doc_py/dbt_doc_py.py", line 279, in run_openai_request
raise e
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/dbt_doc_py/dbt_doc_py.py", line 257, in run_openai_request
if env.api_key.key:
AttributeError: 'UserInfo' object has no attribute 'key'`
Troubleshooting -

[x] Runs successfully if you add your own OPENAI_API_KEY in environment.
[x] Tried running with local installation of dbt, rather than docker. Same issue persists.

A new version was released, fixing this error.
https://pypi.org/project/dbt-doc-py/0.1.25/