Error when using GPT2 model.forward method with DeepSpeed inference

Hi,
I am trying to use DeepSpeed with GPT-2/Neo for inference, but encountering an error when using model.forward method directly. It seems that the error occurs if I don't provide an attention_mask.
import deepspeed
from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2-medium")
model = GPT2LMHeadModel.from_pretrained("gpt2-medium")
model = model.cuda()

model = deepspeed.init_inference(model,
                                 mp_size=1,
                                 dtype=torch.float,
                                 replace_method='auto')

ids = tokenizer.encode("A valley full of unicorns was discovered",
                       add_special_tokens=False, return_tensors="pt").cuda()
# attn = torch.ones((1, 8), dtype=torch.long).cuda()
output = model(
    input_ids=ids,
    # attention_mask=attn, # <-- need to provide this
    return_dict=True,
    use_cache=True
)
print(output)

Traceback (most recent call last):
  File "deepspeed_attn.py", line 25, in <module>
    use_cache=True
  File "torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "deepspeed/inference/engine.py", line 222, in forward
    return self.module(*inputs, **kwargs)
  File "torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "transformers/models/gpt2/modeling_gpt2.py", line 954, in forward
    return_dict=return_dict,
  File "torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "transformers/models/gpt2/modeling_gpt2.py", line 797, in forward
    output_attentions=output_attentions,
  File "torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "deepspeed/ops/transformer/inference/transformer_inference.py", line 611, in forward
    self.norm_b)
  File "torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "deepspeed/ops/transformer/inference/transformer_inference.py", line 393, in forward
    self.qkv_merging)
  File "deepspeed/ops/transformer/inference/transformer_inference.py", line 141, in forward
    while len(input_mask.shape) < 4:
AttributeError: 'NoneType' object has no attribute 'shape'

When I use model.generate, as in the example scripts, I am not getting any errors
as generate takes care of creating attention_mask (but in general I think model.forward should work even without providing attention_mask):
https://github.com/huggingface/transformers/blob/1ed2ebf60d87ef12bd063c7c58e484e19189c754/src/transformers/generation_utils.py#L395

Hi @Express50
Yes, you are right. I will remove this dependency.
Thanks,
Reza

Hi @Express50
Please give this PR a try and see if the issue is resolved.
Thanks,
Reza

Thank you for looking into this @RezaYazdaniAminabadi!
I am now getting a different error when I don't pass attention_mask:
Traceback (most recent call last):
  File "deepspeed_attn.py", line 23, in <module>
    use_cache=True
  File "torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "deepspeed/inference/engine.py", line 230, in forward
    return self.module(*inputs, **kwargs)
  File "torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "transformers/models/gpt2/modeling_gpt2.py", line 954, in forward
    return_dict=return_dict,
  File "torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "transformers/models/gpt2/modeling_gpt2.py", line 797, in forward
    output_attentions=output_attentions,
  File "torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "deepspeed/ops/transformer/inference/transformer_inference.py", line 612, in forward
    self.norm_b)
  File "torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "deepspeed/ops/transformer/inference/transformer_inference.py", line 394, in forward
    self.qkv_merging)
  File "deepspeed/ops/transformer/inference/transformer_inference.py", line 309, in forward
    output, key_layer, value_layer, context_layer = selfAttention_fp()
  File "deepspeed/ops/transformer/inference/transformer_inference.py", line 274, in selfAttention_fp
    context_layer, key_layer, value_layer = compute_attention(qkv_out)
  File "deepspeed/ops/transformer/inference/transformer_inference.py", line 226, in compute_attention
    no_masking)
TypeError: softmax_context_fp32(): incompatible function arguments. The following argument types are supported:
    1. (arg0: at::Tensor, arg1: at::Tensor, arg2: at::Tensor, arg3: at::Tensor, arg4: at::Tensor, arg5: at::Tensor, arg6: int, arg7: float, arg8: bool, arg9: bool, arg10: bool, arg11: int, arg12: bool) -> List[at::Tensor]

Invoked with: tensor([[[ ... ]]],
       device='cuda:0'), tensor([[[ ... ]]],
       device='cuda:0'), tensor([0.]), None, tensor([[[ ... ]]],
       device='cuda:0'), tensor([1.7440e+28]), 16, 0.35355339059327373, False, True, False, 1, True

Seems that arg3 is still expecting a tensor but getting None?

Yes, you are right! I just pushed a fix, can you try again?

@RezaYazdaniAminabadi I am now getting:
...
  File "deepspeed/ops/transformer/inference/transformer_inference.py", line 312, in forward
    output, key_layer, value_layer, context_layer = selfAttention_fp()
  File "deepspeed/ops/transformer/inference/transformer_inference.py", line 277, in selfAttention_fp
    context_layer, key_layer, value_layer = compute_attention(qkv_out)
  File "deepspeed/ops/transformer/inference/transformer_inference.py", line 210, in compute_attention
    if input_mask is None:
UnboundLocalError: local variable 'input_mask' referenced before assignment

weird! input_mask is passed as an argument here!
Let me do some more tests. So, to repro what you are seeing, I have to just remove the mask from the inputs in the huggingface example, right?

The latest error actually happens regardless if I provide attention_mask.

I see. I will fix it soon.
Thanks for trying it out 👍

I retested it w/wo mask and it works for me. Can you give it a try?

Thanks! Just tested it on my end as well and it works both with and without attention_mask.


Pressing Open full report causes a crash on Linux

Describe the bug
Pressing open full report causes a crash on Linux.
Solution: Instead of using the command open for linux here you should probably use xdg-open. I solved this error on my machine by doing sudo ln -s /usr/bin/xdg-open /usr/bin/open.
Desktop (please complete the following information):

OS: Ubuntu 20.04

In my Kali distro, also open works fine.
I'll try with xdg-open and if it works as well on Debian/Kali I'll use it in my code.
Thank you for having discovered this bug on Ubuntu!

I verified and xdg-open also works on my distro.
I heard of people having problems with xdg-open too, but in that case everything should be fixed just by running sudo apt-get install xdg-utils --fix-missing


Added documentation and example of using custom view containers into tree-view-sample

I have added to the tree-view-sample a change that the example also uses a custom view container

@lannonbr An example for custom view container will be definitely helpful. I liked your change. Can you please change the id and title to package-explorer and Package Explorer respectively. Thanks.

And done

LGTM.
Thanks


[WIP]*: client TLS

[skip ci]

looks good in general.

I have made the TLS work.
After the following release, I will push forward the TLS feature and complete the rest of the review.

@hongchaodeng let us clean this up now?


Create Mailchimp sign up

email, name, skills, interest

Created a Facebook signup for "Citizen Labs Movement" - Updates/Interest in Engagement

We already had a Mailchimp account. Just needed the form. Sorry for the confusion.

Yeah, somewhere that is secure yet accessible to make movements on tasks easy


Normalize version tuples in availability attributes coming from clang to use "."

Seen as @available attributes being printed with "_" in interface
generation, but fixing it in the importer means they can't leak into
anywhere else.
rdar://problem/30451293

@swift-ci please smoke test


fix: spread fragment to update mycollection artwork

This PR resolves []
Description
Revert https://github.com/artsy/eigen/pull/10567 as there was already a method to update MyCollection Artwork details

PR Checklist

[ ] I have tested my changes on iOS and Android.
[ ] I hid my changes behind a feature flag, or they don't need one.
[ ] I have included screenshots or videos, or I have not changed the UI.
[ ] I have added tests, or my changes don't require any.
[ ] I added an app state migration, or my changes do not require one.
[ ] I have documented any follow-up work that this PR will require, or it does not require any.
[ ] I have added a changelog entry below, or my changes do not require one.

To the reviewers 👀

[ ] I would like at least one of the reviewers to run this PR on the simulator or device.

Changelog updates
Changelog updates

Cross-platform user-facing changes

iOS user-facing changes

Android user-facing changes

Dev changes

Need help with something? Have a look at our docs, or get in touch with us.

Warnings

:warning:

❌ An error occurred while validating your changelog, please make sure you provided a valid changelog.

  Generated by :no_entry_sign: dangerJS against a3ab597b6353bee6212f3083a367e16265288fb2


[FLINK-9711][CLI] Filter only RUNNING jobs when --running option provided in CLI

What is the purpose of the change
In Flink CLI there's a command list with option --running that according to descriptions "Show only running programs and their JobIDs". However, in practice, it also shows jobs that are in the CANCELED and other states, which does not belong to a RUNNING state.
Brief change log

filter only RUNNING jobs in CLI if corresponding option is provided

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? (no)

-1
The PR description is outdated, the CLI only lists jobs in non-terminal states as running, i.e. CANCELED, FAILED and FINISHED jobs are not included. For all intents and purposes this is the correct behavior, as from a user-perspective a job that is currently restarting is effectively still running.
Finally, this change would list SCHEDULED jobs as terminated, which doesn't make any sense.

I'd agree with you that CREATED, RESTARTING and RUNNING is part of the definition of the running job.
However, CLI does include CANCELED jobs as RUNNING one if --running options is provided.
flink list --running -m host:8081
Waiting for response...
------------------ Running/Restarting Jobs -------------------
03.07.2018 12:07:26 : b14b50d6b6160035d7c62f135a05b5ea : Enriched TrackingClick (CANCELED)
03.07.2018 12:13:39 : 83ab149ad528cfd956da7090543cbc72 : Enriched TrackingClick (RUNNING)
--------------------------------------------------------------

I take it that happens in 1.5.0? If so, this was already fixed for master and 1.5.1.

see https://issues.apache.org/jira/browse/FLINK-9398

yep, it's flink 1.5.0. I'm closing PR then.


NBS: Prevent (in-process) concurrent compactions to a given db

Remove the current probabilistic check

I tried to do this, but I can't figure out where to put a lock. There seem to be lots of different NomsBlockStore instances all triggering the same compaction. So putting a lock on NomsBlockStore that protects Compact() isn't sufficient (in fact there is already one there).


Adds TexturedLine and animated EffectLine to sprites

Adds TexturedLine for drawing advanced textured lines, EffectLine for animating textured lines.
Fixes a small issue in ColorSequence.

lgtm


Add "pip uninstall" to libcompleter

PS: Looks like https://travis-ci.org/ywangd/stash/jobs/540022072#L9881 is not right, but I don't think it is my fault. 😂

@yjqiang thanks for the PR.  Seems like i have forgotten to change this when i renamed pip remove to pip uninstall. Thanks :smile:

PS: Looks like https://travis-ci.org/ywangd/stash/jobs/540022072#L9881 is not right, but I don't think it is my fault. 😂

Yeah, tests/system/test_threads.py::ThreadsTests::test_101 fails randomly. This happens often. We should probably mark it as an xfail.
By the way, could you please make future PRs to the dev branch? It does not realy matter, but this way we can check larger changes for bugs before releasing a new version.


Added std dev & corrected -SavePerIteration

Standard deviation values are printed to screen when -perf all command
line is given. They are also written to file if -perfOutput is given
Corrected - SavePerIteration issue (now outputs eval & bind times for
all iterations)

How is this pull request different from the previous pull request : https://github.com/microsoft/Windows-Machine-Learning/pull/248?

Sorry both pull requests are the same. I am deleting this pull request.


Fix hoisting of components without DOM

@localvoid was right on the money. We were missing a check in coerceToVNode which only checked for vnodes with a _dom. But we actually need that same logic for components, too.
Fixes #1323 .
Adds +2 B.

Coverage remained the same at 100.0% when pulling f505fcdbe761e87e5f1fbfe8f4e70e8a668a3225 on fix_component_hoisting into 44b0de5cc736a973b962b85d56be6061dd0b30cc on master.


Loading Custom Meshes

I have been trying to load a custom mesh into the brain render

tutorial_scene = Scene(jupyter=True)
actor = tutorial_scene.add_from_file("/Users/ranjak/Downloads/cereb_p4.vtk")
actor.opacity(alpha = 0.3)
actor.smoothLaplacian(niter=100, relaxfact=.0002, edgeAngle=10, featureAngle=60)
actor.rotateX(90, rad=False)
actor.x(0)
actor.y(0)
actor.z(0)
tutorial_scene.add_sphere_at_point( pos=[0, 0, 0], radius=100, color="black", alpha=1)
tutorial_scene.render()

doing something like this doesnt help as well
root = tutorial_scene.root
actor.rotateX(90, rad=False)
actor.x(0)
actor.y(0)
actor.z(0)
root.x(0)
root.y(0)
root.z(0)

This should add both the sphere and the structure at the same position but i get an output like.
Even when i change the coordinates of the actor using actor.x(newx) etc or rotate it the structures dont overlap or get placed within each other

hey,
Can you try adding this to your code please:

tutorial_scene = Scene(jupyter=True)
actor = tutorial_scene.add_from_file("/Users/ranjak/Downloads/cereb_p4.vtk")
actor.opacity(alpha = 0.3)
actor.smoothLaplacian(niter=100, relaxfact=.0002, edgeAngle=10, featureAngle=60)
actor.rotateX(90, rad=False)
actor.x(0)
actor.y(0)
actor.z(0)

# new code starts
com = actor.centerOfMass()
print(f"Actor's center of mass: {com}")
tutorial_scene.add_sphere_at_point( pos=com], radius=100, color="blue", alpha=1)
# end of new code

tutorial_scene.add_sphere_at_point( pos=[0, 0, 0], radius=100, color="black", alpha=1)
tutorial_scene.render()

I just want to make sure that there isn't something weird with the bounding box of your actor.

new code starts
com = actor.centerOfMass()
root_com = tutorial_scene.root.centerOfMass()
actor = actor.x(root_com[0])
actor = actor.y(root_com[1])
actor = actor.z(root_com[2])

this works for changing the position i think because actor.x(0) actually returns a mesh object so we do an assignment and then back again
i was able to bring them closer how do i fit it inside is there a way to get the coordinates of the cerebellum within the root scene

the output of the code you provided is

ah nice find, good to know that you can now move it around correctly.
You can use scene.get_region_center_of_mass to get the CoM of the cerebellum, and then move your object there: https://github.com/BrancoLab/BrainRender/blob/74fefa9eefd3bdbd7aef377b016e0f631fb184c2/brainrender/scene.py#L261
It looks like you'll have to rotate the mesh around as well, but that should be easy to do once it's in place.

tutorial_scene = Scene(jupyter=True)
actor = tutorial_scene.add_from_file("/Users/ranjak/Downloads/cereb_p4.vtk")
actor.opacity(alpha = 0.3)
actor.smoothLaplacian(niter=100, relaxfact=.0002, edgeAngle=10, featureAngle=60)

com = actor.centerOfMass()
p1, p2, p3 = tutorial_scene.get_region_CenterOfMass("CB", unilateral=False, hemisphere="right")
print(f"Actor's center of mass: {com}")
print (f"Root's center of mass: {p1, p2, p3}")
actor = actor.pos(p1+9000, p2+3000, p2-2000)

p1_n, p2_n, p3_n = actor.centerOfMass()
print (f"New center of mass : {p1_n, p2_n, p3_n}")

tutorial_scene.add_sphere_at_point( pos=[p1_n, p2_n, p3_n], radius=100, color="blue", alpha=1)

tutorial_scene.add_sphere_at_point( pos=[p1, p2, p3], radius=100, color="blue", alpha=1)
tutorial_scene.add_sphere_at_point( pos=[0, 0, 0], radius=100, color="black", alpha=1)
tutorial_scene.render()

it kind of works but its strange that actor.pos(x1, y1, z1 ) doesnt position according to the center of mass but by some other maybe bounding box kind of logic but my mesh fits inside its smaller than the root brain probably have to scale it up etc but thanks if you can suggest any other cosmetic changes to make it look better that would be nice because my mesh looks ugly and the allen brain ones look much nicer, but the tool looks really amazing thanks a lot

yeah there might be some issue with how you created the mesh which causes the CoM not to be where you'd expect.
You can try smoothing the actor + change color and transparency to make it look better. You can also try playing around with the actor's material to change how it reflects light.

thanks

is there a way to define a new region like i want to overwrite the "CB" region in the brain render with my own mesh region so that i get points inside the new mesh and not the old one

I'm afraid not.  All parts of brainrender where you can specify the name of the brain region directly depend on that and can't be easily changed.
But once you have your brain region rendered as an actor you should be able to do all the same as what the other functions are doing.
What is it that you want to do exactly?

i want to add other brain regions which are subregions of the cerebellum and extend the existing functionalities of brain render to those new regions and have an ontology tree etc basically everything that is in the brain render for the adult p56 mice but for different ages

I see, cool!
It would take a bit of work but we can work on it, it'd be nice to have.
I think maybe @adamltyson  you did something like this before?

It was something I was looking into - how we could support arbitrary atlases in brainrender.
However, I didn't get any further than just getting brain regions from an image-based atlas, extracting the surface and displaying in BR.

Okay, thanks!
One way would be for functions like this: https://github.com/BrancoLab/BrainRender/blob/74fefa9eefd3bdbd7aef377b016e0f631fb184c2/brainrender/scene.py#L261
To not just accept regions acronyms from the Allen atlas, but also to work on custom meshes.
This wouldn't fully achieve the goal of supporting a custom atlas, but would extend some of brainrender's functionality to custom meshes.

Move the conversation here: https://github.com/BrancoLab/BrainRender/issues/56


Add title and description of awesome mechanical FEA list

Add title and description of awesome mechanical FEA list.

@allcontributors please add @tkoyama010 for ideas


Update preflight to v1.9.2

https://github.com/redhat-openshift-ecosystem/openshift-preflight/releases/tag/1.9.2

Doesn't seem like this is possible at this point in time.  There is an underlying dependency that isn't allowing the upgrade.


Split query string parameters and route parameters.

The inverse of #74.
Unfortunately, I didn't know about #74 when it was being discussed else I would have mentioned this over there.  The problem with merging query string and route parameters is that query string parameters may include user input.  In particular, my user provides me with a set of key/value pairs that I want to encode in the query string (so they can easily link back to the current state).  By combining the query string parameters with the route parameters, the user may unintentionally collide with my route parameters (which they know nothing about).

Good point. Currently if there is a name collision the value from the route params will be chosen over the value from the query string.
activate() is passed a third parameter that is the navigation instruction, and the original separate objects are available on that. You should be able to use that to read your user input without fear of a collision. I think the merge simplifies the common case, so I'd like to keep it. This should certainly be documented, though.
activate(params, routeConfig, navigationInstruction) {
  // navigationInstruction.params holds the route params
  // navigationInstruction.queryParams holds the query string params
}

I agree, the merge is the common use case, but you can always get the other pieces from the navigation instruction.


Fix authentication tests

All tests that require token authentication are failing and need to be fixed with github secrets.

Fixed.


waterfox data is not portable

why is waterfox data not saved in the portable folder?

Could you create a PR for the same?

yes #7976


module 'websockets' has no attribute 'serve'

I installed the latest version of websockets (4.0.1),  copied the example from here:
http://websockets.readthedocs.io/en/stable/intro.html
and ran it in Python 3.5.2.
I got an error: "module 'websockets' has no attribute 'serve'".

This is correct! I named the example file "websockets.py".

Ha! You can't be the first one. Perhaps I should warn about this common Python pitfall in the docs.

I ran into this trap as well.  A warning in the docs would be great.

Let's add it to the FAQ.

Better late than never!

I still have that issue wiht version 1.9.0. I'm on Windows, in case that matters. I don't have a websocket.py file. Importing from websockets.server or websockets.sync.server works. But not from websockets directly.


Add admin API operations to promisified and non-promisified API

For example: listing topics, ACL operations, consumer group operations, consumer group offset operations.

Added: ListGroups to list consumer groups

Added: DeleteGroups and DescribeGroups
Added: ListTopics

+1 for describeGroup. We use this function to health check the consumer.
Any idea how to do it without this function?

@brunodomenici , the function is already available in the released version of the library which you can download off npm.
Unfortunately I've not added an example yet, but the usage can be seen in the test: https://github.com/confluentinc/confluent-kafka-javascript/blob/v0.1.12-devel/test/promisified/admin/describe_groups.spec.js#L71


Pulumi Databricks Groups can only add one user

What happened?
We create a databricks group "group1".  -> ok
We add a new user to the groupe "group1".  -> ok
We try to add a new user to the same group "group1" - > ok'is this eliminates the existing user and adds the new user.
Steps to reproduce
Follow the described above.
for users use the id of the user.
Expected Behavior
A new user it's added to the group and all other users added before still belong to the group
Actual Behavior
only one user can be added to the group. If a new user it's added to the group that has a user in it the exisitng user will be removed
Output of pulumi about
CLI
Version      3.50.0
Go Version   go1.19.4
Go Compiler  gc
Host
OS       darwin
Version  13.1
Arch     arm64
Additional context
this is the output when try to add a new user to group:
 pulumi:pulumi:Stack dev-streaming-dfs-data-lake-stream-ingestion-eu-dbw-dev_new_folder_group running
    pulumi:providers:databricks databricks_provider_new_folder
 ++ databricks:index:GroupMember dbmembers_new_folder creating replacement (0s) [diff: ~memberId]
@ updating....
 ++ databricks:index:GroupMember dbmembers_new_folder created replacement (1s) [diff: ~memberId]
@ updating....
 +- databricks:index:GroupMember dbmembers_new_folder replacing (0s) [diff: ~memberId]
 +- databricks:index:GroupMember dbmembers_new_folder replaced (0.00s) [diff: ~memberId]
 -- databricks:index:GroupMember dbmembers_new_folder deleting original (0s) [diff: ~memberId]
@ updating....
 -- databricks:index:GroupMember dbmembers_new_folder deleted original (1s) [diff: ~memberId]
@ updating.....
    pulumi:pulumi:Stack dev-streaming-dfs-data-lake-stream-ingestion-eu-dbw-dev_new_folder_group

Outputs:
    group_id   : "410490906214398"
  ~ members_id : "2162137320760042" => "4637042203598367"
    provider_id: "b278f363-8bc3-47f6-9b5a-515aab131bdb"

Resources:
    +-1 replaced
    2 unchanged

Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

That was an error on my side.
The add of users has to be done in a loop inside the program.


Update all conference Information when available

Update the TO-DOs in the replication package with the correct information, as follows:

Include conference details such as the name, page numbers, date, and location in the README.md and CITATION.ccf files.
Ensure that the ZENODO package information is also updated.

Dear @giis-qabot, I am waiting for the proceedings to be published, I'll update it as soon as it would be available


Instructions to host YOURLS on Heroku

I just put YOURLS on Heroku (a cloud PaaS) successfully. Just checking if there is any interest in adding some sample files in the code or including this HowTo in the README or on a Wiki page. Googling around helped me deal with putting php on Heroku, but nothing directly related to YOURLS and its specific gotchas. So I thought writing it out may help those who are interested.
A rough list of steps:

Modify user/config.php to pull credentials and sensitive info from Heroku's config vars. This is a good practice (separate admin user and db credentials from config.php), so maybe we can even add a user/config-heroku-sample.php so people can just rename when setting up
Use one of the MySQL addons on Heroku (code handling the connection is reflected in config-heroku-sample.php)
Add a dummy, empty index.php to the project root folder so Heroku recognizes this as a PHP app
Add a .htaccess file to the project root folder (and push to heroku) because adding .htaccess dynamically from the web UI doesn't work for Heroku

Still no merge, but I also want to add that creating a Procfile with contents web: vendor/bin/heroku-php-apache2 removes deploy warnings about the default web process selection when using the empty Composer configuration.
Also, for me, same thing with ephemeral file system, password not needed to be hashed, instead it looks like:
 // they're located in ENV vars on heroku, no need to hash them.
define( 'YOURLS_NO_HASH_PASSWORD', true );
$yourls_pass = getenv('YOURLS_PASS');
$yourls_site = getenv('YOURLS_SITE');
$yourls_user_passwords = array($yourls_admin => $yourls_pass);

I don't personally know how but it would be cool to have one of those "Deploy to Heroku" buttons on the readme - https://devcenter.heroku.com/articles/heroku-button.

To cover Heroku usage (and other similar services like Zeit Now) Docker images are available for YOURLS. See https://store.docker.com/images/yourls


Typo in "Sporadic traffic" paragraph?

Hi, would you mind checking the following sentence from the "Sporadic traffic" paragraph.  I'm either misunderstanding something or there's a typo:
"These apps include sync engines, backup providers, search indexers, classification engines, data loss prevention tools, and any other which attempts to reason over the entirety of data and change."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9efe14fc-9d5d-a5fc-07ba-e013e89589e2
Version Independent ID: 5455b54e-997d-75e3-018a-0bf3f3b2180b
Content: Avoid getting throttled or blocked in SharePoint Online
Content Source: docs/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online.md
Product: sharepoint
Technology: sharepoint-framework
GitHub Login: @spdevdocs
Microsoft Alias: spdevdocs

While technically correct, it could be wordsmithed a bit. I'll add that to my backlog.
The last part of the sentence is saying essentially restating the rest of the sentence. Basically "any other tool that attempts to reason over lots of data and any changes to that data" resulting in a lot of HTTP calls...


Added SparkSession configs statements if non-default namespace used

Added SparkSession configs statements if non-default namespace used to the demo notebook. Otherwise the SparkSession would not be created with an error referring to the wrong namespace

Thank you for this PR! We'll take a look at this in the next few days.

Hi @morooshka,
thanks for your contribution :)
The PR looks good overall, i will try to test tomorrow.
We would like to keep the output (charts and tables) in the notebook though. This way you can e.g. see what the results look like e.g. via github as well or if anything goes wrong (e.g. like not enough memory etc.) during execution.
Im fine with you doing it, otherwise i can push the executed notebook again including your changes.
Cheers,
Malte

Hi @morooshka, thanks for your contribution :)
The PR looks good overall, i will try to test tomorrow.
We would like to keep the output (charts and tables) in the notebook though. This way you can e.g. see what the results look like e.g. via github as well or if anything goes wrong (e.g. like not enough memory etc.) during execution.
Im fine with you doing it, otherwise i can push the executed notebook again including your changes.
Cheers, Malte

OK, I'll repush it. I thought someone had forgotten to delete the outputs
Thanks for quick responce

Hi @maltesander! I've repushed the notebook with outputs as requested

Hey @morooshka, tested and works. Thank you!
One nitpick, there is one typo deploynig when defining the namespace (i cannot suggest via git due to large diff).
# Modify this please if using non-default product namespace for demo deploynig
NAMESPACE: str = "default"

Otherwise we can merge :)

Hey @morooshka, tested and works. Thank you!
One nitpick, there is one typo deploynig when defining the namespace (i cannot suggest via git due to large diff).

OMG, thank you for noticing it, I'll repush

@maltesander I have repushed, thank you


summary failure

The function mia::summary is not robust to NAs.
This works for example data as expected:
library(mia)
data(esophagus)
se <- esophagus
mia::summary(se)

Then we add some NAs:
assay(se, "counts")[c(1,50,100)] <- NA
mia::summary(se)

This fails with
Error in if (any(colSums2(assay.x) < 1) | any(colSums2(assay.x) < 0)) { :
missing value where TRUE/FALSE needed
It seems that the na.rm argument is not passed to the auxiliary functions in
summaries.R.
I suggest to add this (allow argument passing in the functions, at least for na.rm).
I would suggest to allow NAs by default with na.rm=TRUE and in addition make sure that the summary throws out a warning, and prints the number of NAs as part of the summary. Or alternatively set na.rm=FALSE by default but make sure there is an informative error that tells where the problem is exactly (rowdata, coldata, particular assay..)

If @microsud has a chance to fix that would be great. If not, perhaps @TuomasBorman ?

Hi Leo- I think the assay objects should never have NAs as these are expected to be raw counts. We have an error message for this. There are samples that sum to 1 or less counts , Try to supply raw counts. I can try to add an informative error message for this.
Best wishes,
Sudarshan

That's fine too - - it is theoretically possible that one would try to use these tools on other types of data that may have missing values. Perhaps the error message can mention that this function is limited to sequencing data only, and that is not expected to contain missing values.

yes. Do I also bump the version for this PR?

yes please


Will make navbar remain on top on scrolling

The navbar leaves the view space when I scroll down , I want to fix this , @YasharF could you please assign me this issue

Please assign this to me

you can assign this to me!

can you assign it to me ?

We don't need this at this point


Documentation claims that all uploads are virus scanned?

In production scenarios, an anti-virus/anti-malware scanner API is used on files before making them available for download or for use by other systems

Seems unlikely; is it reallt true? Or perhaps there is a word missing from the docs and it should instead say "ensure an anti-virus..."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5ea474d5-234a-f393-8609-892a2deb5d2e
Version Independent ID: c11d981c-05af-c19d-a333-feedd5978639
Content: ASP.NET Core Blazor file uploads
Content Source: aspnetcore/blazor/file-uploads.md
Product: aspnet-core
Technology: aspnetcore-blazor
GitHub Login: @guardrex
Microsoft Alias: riande

Hello @CaiusJard ... That's fine ... I can reword that.


Integrated terminal in vscode does not highlight search results

VSCode latest
Mac OSX
The VSCode terminal does not highlight text when searching, nor show "x/y results":

I love running tests in the integrated terminal but it's nice to see where i'm jumping around for matches. Here's an example of what iterm2 does:

dupe of https://github.com/Microsoft/vscode/issues/28768


Tables not being displayed after update

Hi, I've just updated from 1.10.0 to 2.5.0 and after adapting all listed breaking changes I've found that tables are completely gone from my ppt. I've tried using code from sample files, but still nothing :(
        let items = data,
            tableBlock = [],
            chunks = '',
            header = items.splice(0,1),
            toTable = [],
            baseTitle = title,
            iterations = 0;

            while(items.length > 0){
                if((iterations == 0) && items.length < 12){
                    title = baseTitle;
                }
                else{
                    title = baseTitle + ' - Part ' + (iterations + 1);
                }
                toTable = items.splice(0,10);
                toTable.unshift(header[0]);
                let slide = new pptx.addNewSlide('COMPANY_PPT'),
                rows = [];

            toTable.forEach(function(item, parentIndex){
                let row = [];
                item.forEach(function(elem, index){
                    let options = {};
                    if(index === 0 && parentIndex != 0){
                        options.color = '4ca3b5';
                        options.bold = true;
                        options.align = 'left';
                    }
                    if(parentIndex === 0){
                        options.bold = true;
                        options.fontSize = 10;
                    }
                    if(index != 0){
                        options.align = 'right';
                    }
                    options.border = [{pt: 0, color: 'ffffff'},{pt: 0, color: 'ffffff'},{pt: 1, color: 'E3E3E1'}, {pt: 0, color: 'ffffff'}];
                    row.push({text: elem, options: options})
                });
                rows.push(row);
            });
            let options = {w: '60%', y: pptData.simpleLayout ? '21%' : '17%', x: pptData.simpleLayout ? '18%' : '35%', lineWeight: 5,  fontSize: 13, rowH: 0.5, valign: 'middle'};
            slide.addTable(rows, options);

The function gets executed - I'm able to pass any other sample code from the docs and it gets displayed: e.g: texts, images... - but anytime I try adding tables (sample or old working code), they won't display...
Is there any breaking change from v1 to v2 I'm missing? Thanks a lot!

Facing similar issue.

@somprabhsharma did you make it work?

Hi @emartz404
I dont know what the data in let items = data looks like, but tables definitely work in the most recent version using the sample code provided...
Online Demo

I've tried with the sample code and happens the same... Anyway, found that if I downgrade to v2.3.0 they work okay again

@gitbrent Found out it's a Keynote issue: if I open the demo file in google drive it's ok:

but in keynote:

@emartz404 Hey, I could make it work. To make it work in keynote too, you need to set the background color to white. It will work.

@somprabhsharma is correct - the only issue with Keynote is the default background.
If you explicitly set one (e.g.: white) it'll appear as it does in PowerPoint.
This defect will be fixed in the upcoming 3.0 release.

Thanks a lot @somprabhsharma @gitbrent !

The issue of missing default [white] background in Keynote has been fixed in the 3.0 branch and will be released with 3.0.
Thanks @emartz404 @somprabhsharma


Updated to use pagespeed v4

Updated minimum node version to 6.
googleapis module version to 33.0.0
Fixed some tests that fails after update to the new pagespeed version. #

Seems that google redirects to a failure page when test are launched in travis. Due that the title doesn't match with the expected in the test so the test fails. Because of that I have changed the page setted in the test and now all test pass again.

Best regards.

PageSpeed Insights API v5 has now been released, I guess we might as well update to that version instead of targeting v4.
https://developers.google.com/speed/docs/insights/release_notes

Thanks @mansn! I think know. I'll try to update to the new version 👍

Hi! I have tried to use the last version of pagespeed (v5) but seems it's not included yet in the googleapis package used in this tool. What do you think @mansn ? Do you know how to use the last version in the way is used in this tool?? Where can I find it?
Thanks in advance.

@JuanMaRuiz Ah, sorry. I totally missed to do my homework and check the dependencies before making previous comment. My bad!

Hey folks. Really sorry for the radio silence here. As we've shipped PSI's V5 API, would anyone like to explore updating the package to use that instead? It'd be a nice release as we now expose Lighthouse results there too.
I'm also more than happy to take on more co-maintainers if folks are actively using this package.

Hi! You have nothing to be sorry for @addyosmani.
According with npm the number of weekly downloads of this package is ~8000 so it seems folks are still using it.
I can try to update to PSI V5.
Best regards.

@JuanMaRuiz any help at all with updating to PSI V5 would be greatly appreciated. Thank you :)

Many thanks @addyosmani for adding me to the repo. I really appreciate it!

Merging. Let's continue the v5 discussion in #97.


Add beMissing generator to replace beNull generator that uses depreca…

beNull uses a missing query, which is deprecated after ES 2.2.0:
https://www.elastic.co/guide/en/elasticsearch/reference/2.3/query-dsl-missing-query.html
This change will add a new generator called beMissing that achieves the same functionality, and the beNull should be removed at a later date.

Coverage decreased (-1.5%) to 91.525% when pulling c811f1001d720ffb0f70ff98a420da0586463552 on feat/empty-check into 659fc3c8cd51fd5c5c5a9f53a75e3d83d0938710 on master.

Coverage increased (+0.2%) to 93.22% when pulling fa1c6c6914793519fd10e4f7bcf4ac8335862187 on feat/empty-check into 659fc3c8cd51fd5c5c5a9f53a75e3d83d0938710 on master.

is the reason for a new method because exists isn't supported by 2.1? I think this looks good, I would add unit tests though!


stop automatic git commits of schema-reference.md

resolves #997

@jrcheli Looks good to me.
I allowed myself to add one commit. This is not strictly related to your change. Let me describe what I have added:

During the schema markdown generation process we start by resolving references "$ref": e.g.

  "properties": {
    "type": {
      "$ref": "definitions/envelope.schema.json#/$defs/event_type"
    },

This is done by json-dereference.
If "$ref" is invalid e.g. points to no existing definition we will retrieve the syntax error like e.g. following:
SyntaxError: Error resolving $ref pointer "/home/node/schemas/definitions/envelope.schema.json#/$defs/event_type3".

But the json-dereference will still return 0 like in the case of success. I have added an additional logic to check if the file with resolved references was generated. If a file with fixed references was not generated, exit 1 will fail the job.
After merging this Pull Request we should check if the new Github Action is working correctly even with a simple change in a single JSON schema file.

@jrcheli Not sure if it is worthy or not - since I am not aware of the exact process of how deploy works.
Should we extend the .gitignore file with the following entry:
/website/src/pages/docs/schema-reference.md

My rationale is to prevent situation including the /website/src/pages/docs/schema-reference.md file in the commit ?


Error

When i run the application it shows this. I noticed there was an issue with this way back. Would like to know if there's a fix. Thank you
discord-image-downloader-go version 1.36.1
reading config from config.ini
2019/08/25 00:04:25 [DG0] wsapi.go:552:onEvent() error unmarshalling READY event, json: cannot unmarshal number into Go struct field ReadState.last_message_id of type string
Client is now connected as Hiari. Press CTRL-C to exit.

I get the same

could you send a screenshot of your config files (just "censor" the login data and others)
@iLvdd @smokeofc


[broker]Optimize topicMaxMessageSize with topic local cache.

Motivation
Currently, , we update the value of Topic#topicMaxMessageSize in an interval of  maxMessageSizeCheckIntervalInSeconds (default as 60s).
So if we update this value, we will have a delay, max to 60s by default, before it take effects.
We can reduce the latency by updating the value with notification.
Further, we can avoid the blank updates in isExceedMaximumMessageSize.
Modifications

Move topicMaxMessageSize to HierarchyTopicPolicies , and update by onUpdate listener, like other topic policies.
Deprecates maxMessageSizeCheckIntervalInSeconds, since it has no use any more.
Remove lastTopicMaxMessageSizeCheckTimeStamp and topicMaxMessageSizeCheckIntervalMs, no use any more.
Move method registerTopicPolicyListener up to AbstractTopic level, since topicMaxMessageSize also effect non-persist topics.

Verifying this change

[x] Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:
(example:)

org.apache.pulsar.broker.admin.TopicPoliciesTest#testTopicMaxMessageSize, and added test case for non-persisitent topics.
Add org.apache.pulsar.broker.admin.TopicPoliciesTest#testTopicPolicyInitValue to cover  the case that topic loading old settings.

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?

[x] doc

Doc updated in comments.
And checked with old configuration doc, no maxMessageSizeCheckIntervalInSeconds found, and no need to add since it is deprecated.

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@codelipenghui @315157973   Please help take a look.

I think you missed the updateTopicPolicyByBrokerConfig to apply the default broker configuration?

Not really. You mean ServiceConfiguration#maxMessageSize, it take effects in the netty RPC layer.
See https://github.com/apache/pulsar/blob/3a337b8952239136cb77302e22021b3d79154f1c/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarChannelInitializer.java#L130

I think you missed the updateTopicPolicyByBrokerConfig to apply the default broker configuration?

Not really. You mean ServiceConfiguration#maxMessageSize, it take effects in the netty RPC layer.
See https://github.com/apache/pulsar/blob/3a337b8952239136cb77302e22021b3d79154f1c/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarChannelInitializer.java#L130

Yes, but I think we should keep consistent in the local topic policy cache? And for the next step, we can apply the PolicyHierarchyValue to get topic policy.

Great point. Added. And it's easier to find out which policy have broker/ns/topic level setting.

/pulsarbot run-failure-checks

Could you please show me where update the PolicyHierarchyValue.brokerValue?

Last line in method updateTopicPolicyByBrokerConfig


Replace URI -> URL

URI is not defined

It's strange that the URI work before. I test URI in node and browser then throw URI is not defined but URL is ok. Maybe some dependent module export the global URI.

Another error occur. I think node-fetch update.
TypeError [ERR_INVALID_ARG_TYPE]: The "url" argument must be of type string. Received an instance of URL
    at validateString (internal/validators.js:121:11)
    at Url.parse (url.js:159:3)
    at urlParse (url.js:154:13)
    at new Request (/Users/arron/project/local-service/viber/node_modules/node-fetch/lib/request.js:27:16)


Update Person.php

A better italianische personification ;)

Would you mind adding a reference for the data source? It makes future updates easier for other maintainers

Well, I've tried tried to guess. But if you need a referenced source, there's some Open Data about it.
http://www.istat.it/en/products/interactive-contents/baby-names
If you feel interesting I could try to merge

I would love to have a list referencing a single source. That would ease maintenance so much.

I've found this link,
http://demo.istat.it/altridati/IscrittiNascita/2013/T2.9.xls
It's a yearly report of 30 most used names for babies born in Italy

We need to reference a web page mentioning the content license, and it should be compatible with MIT.

Open Data licence from Istat website is described at http://www.istat.it/en/about-istat/activities/open-data they chose CC Attribute 3.0 but I can't say if it's compatible with MIT.
Feel free to merge my PR there's no link between my own activity and Istat open-data distributors.

If we can't reference an external source, I'm not sure your PR has a high value. Another contributor could argue that the names you provided aren't better than the ones in place, I have no way to tell which one is better.
I really want to have a reference for e


HUD is jittery.

NomaiVR 0.51.0, game version <IP_ADDRESS>3 on Epic, Vive Pro.
The HUD, tools when they're in the toolbelt, and mask at the end of the loop are all slightly jittery. It's not a tracking issue on my end as far as I can tell, as my hands and my view are all completely steady.
That being said, I wouldn't be surprised if it was something else on my end. Feel free to delete this if that's the case.

It's just the way I make stuff attached to player / camera. I'm very limited in how I can attach things, because breaking the hierarchy of the scene can often completely break the game.
I'll leave this open since it's a real issue, but I won't look into it until after releasing version 1.0.


ETCDSnapshot CRs not cleaned up/not following etcd-snapshot-retention value

Environmental Info:
RKE2 Version:

v1.23.9+rke2r1
Node(s) CPU architecture, OS, and Version:

Linux k8s01-rancher 5.4.0-124-generic rancher/rke2#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration:

3 server nodes, Etcd snapshot enabled.
Describe the bug:

ETCDSnapshot CRs are not cleaned up when there are things like failures.
Steps To Reproduce:

Installed RKE2: curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION=v1.23.9+rke2r1 sh -
Configure RKE2 with ETCD support (In my case using S3) (Make sure that you configure it incorrectly so that the backup fails)

server: https://<snipped>:9345
token: <snipped>
tls-san:
  - <snipped>
  - <snipped>
  - <snipped>
node-name: k8s01-rancher
secrets-encryption: true
disable:
  - rke2-ingress-nginx
etcd-disable-snapshots: false
etcd-snapshot-name: "etcd-snapshot-rancher"
etcd-snapshot-schedule-cron: "0 */1 * * *"
etcd-snapshot-retention: 20
etcd-s3: true
etcd-s3-endpoint: "<s3_compatible_endpoint>"
etcd-s3-access-key: "<snipped>"
etcd-s3-secret-key: "<snipped>"
etcd-s3-bucket: "rke-rancher-backups-<snipped>"
audit-policy-file: "/etc/rancher/rke2/audit-policy.yaml"

Wait for the backup to run (and then fail)
Let it fail for a while
See that there are now >20 fails Snapshot CRs

Fix snapshot configuration
Wait for the backup to run (and then succeed)
See that there are still the failed Snapshot CRs, but also the successful ones

Expected behavior:

I expect that at most there should only ever be etcd-snapshot-retention number of ETCDsnapshot CRs for a given node.
Actual behavior:

I see ETCDsnapshot CRs continue to grow past the etcd-snapshot-retention.
Additional context / logs:

Still relevant (that I'm aware)

Same here. /var getting full and rke2 server becoming unstable with /lib/rancher/rke2/server/db/snapshots having 170+ snapshots despite  "etcd-snapshot-retention": 100 and snapshots being pushed to S3. This one is nasty.

@FredNass that is not the same issue. If you are seeing actual snapshots being retained on the RKE2 node itself in violation of your retention settings, please open an issue in the RKE2 repo. This issue is regarding EtcdSnapshot Custom Resources in the Rancher management cluster.

@brandond You're right, sorry. I will.

Any chance this ticket could be re-opened? It is still an issue with 2.7.4

@BenB196 , I'm taking a look into this issue.
Wile testing I noticed that rancher keeps up to 3 times more ETCDSnapshots  that the expected amount (etcd-snapshot-retention * nodes).
The Failed s3 snapshots, the local snapshots and the s3 snapshots each have their on limit.
Can you confirm to me if the problem that u are having is this one?
If you have jq u can run the following commands to list respectively: number of local snapshots number of s3 snapshots and number of failed s3 snapshots.
kubectl get ETCDSNapshot  -n fleet-default -o json |  jq '.items[].metadata|select(.annotations."etcdsnapshot.rke.io/storage"=="local")|.name' | wc -l;
kubectl get ETCDSNapshot  -n fleet-default -o json |  jq '.items[]|select(.metadata.annotations."etcdsnapshot.rke.io/storage"=="s3" and .snapshotFile.status!="failed")|.metadata.name' | wc -l;
kubectl get ETCDSNapshot  -n fleet-default -o json |  jq '.items[]|select(.metadata.annotations."etcdsnapshot.rke.io/storage"=="s3" and .snapshotFile.status=="failed")|.metadata.name' | wc -l;

Hi @felipe-colussi
Here are the outputs of the commands you requested:
:~$ kubectl get ETCDSNapshot  -n fleet-default -o json |  jq '.items[].metadata|select(.annotations."etcdsnapshot.rke.io/storage"=="local")|.name' | wc -l;
0

:~$ kubectl get ETCDSNapshot  -n fleet-default -o json |  jq '.items[]|select(.metadata.annotations."etcdsnapshot.rke.io/storage"=="s3" and .snapshotFile.status!="failed")|.metadata.name' | wc -l;
45216

:~$ kubectl get ETCDSNapshot  -n fleet-default -o json |  jq '.items[]|select(.metadata.annotations."etcdsnapshot.rke.io/storage"=="s3" and .snapshotFile.status=="failed")|.metadata.name' | wc -l;
0

Note: Based on the above, I don't think the current hypothesis is accurate.
For reference:

This Rancher setup currently manages 9 Kubernetes clusters (including itself)
Of the 9 clusters, 5 are RKE2 clusters (including the Rancher cluster) and 4 are AWS EKS clusters (imported as generic clusters).
Of the 5 RKE2 clusters, 1 is 3 nodes (the Rancher cluster), 3 are 4 nodes, and 1 is 6 nodes
All RKE2 clusters have an S3 backup schedule of etcd-snapshot-schedule-cron: "0 */1 * * *" with etcd-snapshot-retention being set to either 10 or 20 depending on the cluster.

The above configuration is set on all control nodes.
Each cluster has 3 control/etcd/worker nodes and any remaining nodes are purely just worker nodes.

I do believe that I found your problem.
Probably the ETCDSnapshots are being marked to be deleted but rancher is not doing the sync-up to delete them.
Just to double check:

Did you configured the snapshot Backup through RKE2 instead of rancher?

Please run the following commands (fell free to remove/edit the names, if possible let me know what kind of installation/import you used for each one) :

Get number of ETCDSnapshots marked as Missing  per cluster (if there is none or few of them please remove the |select(.status.missing==true)|):

 kubectl get ETCDSnapshots -n fleet-default -o json | jq '.items[]|select(.status.missing==true)|.metadata.labels."rke.cattle.io/cluster-name"' |(tr ' ' '\n' | sort | uniq -c | awk '{print $2"@"$1}')

Check if any cluster have  a machine-plan secret (if you changed names on the previous command  use the same names here).

kubectl get secrets --field-selector type=rke.cattle.io/machine-plan -n fleet-default -o json | jq '.items[].metadata.labels."rke.cattle.io/cluster-name"'

If my hypotheses is right you won't have a machine-plan for the nodes with more ETCDSnapshots that the limit.

Hi @felipe-colussi yes, the backups are configured via RKE2 and not through Rancher.
Running:
 kubectl get ETCDSnapshots -n fleet-default -o json | jq '.items[]|select(.status.missing==true)|.metadata.labels."rke.cattle.io/cluster-name"' |(tr ' ' '\n' | sort | uniq -c | awk '{print $2"@"$1}')

I get:
"k8s-<cluster_1>"@10079
"k8s-<cluster_2>"@12999
"k8s-<cluster_3>"@11754
"k8s-<cluster_4>"@10144

Running
kubectl get secrets --field-selector type=rke.cattle.io/machine-plan -n fleet-default -o json | jq '.items[].metadata.labels."rke.cattle.io/cluster-name"'

I get nothing returned.

@BenB196.
Short explanation of the problem:
When RKE2 creates or deletes an etcd snapshots it populates an configmap called rke2-etcd-snapshots on the downstream cluster.
Rancher monitors that configMap creating ETCDSnapshots on the local cluster and marking them as "missing" once they are not on the configmap anymore.  Those are used to keep references to the restore object and for restoring snapshot through rancher (only for clusters created by rancher).
To avoid some concurrency problems and not lose valid snapshots the deletion of the ETCDSnapshots also depends on some data from rke.cattle.io/machine-plan. As the machine-plan is not created for imported clusters the ETCDSnaphsots aren't being deleted.
We are looking for a good way of fixing it.
Workaround:
All ETCDSnapshots marked as "Missing" can be safety deleted.
I spent some time trying to figure out what would happen if you deleted over 40K objects at once, and unfortunately I wasn't able to find out.
After talking with the team we figured out that would be safer to run a script that delete some of them and waits for some time to avoid hammering down Kubernetes API.
You can run something on those lines:

#/bin/bash
mapfile -t my_array < <( kubectl get ETCDSnapshots -n fleet-default -o json | jq '.items[]|select(.status.missing==true)|.metadata.name' |tr -d '"' | head -n 3000)

deleteafterx=15

deleteargs=0
etcds=""
for name in "${my_array[@]}"; do
  deleteargs=$((deleteargs+1))
  etcds=$etcds"$name "
  if (( $deleteargs > $deleteafterx )); then
    kubectl delete -n fleet-default ETCDSnapshot $etcds --wait=true
    deleteargs=0
    etcds=""
    sleep 5
  fi
done
kubectl delete -n fleet-default ETCDSnapshot $etcds

If you choose to run the script please monitor the output.
The script is also on the safer side, the delete is limited and have a 5s sleep to make sure it will not affect kubernetes on any way.
Fell free to edit it, the values that you can tweak around are:
sleep 5 delay between kubectl delete execution.
deleteafterx the number of names passed on the kubectl delete
head -n 3000 you can even remove it, this will make the script run only in 3000 objects.

Hi @felipe-colussi thanks for the explanation as well as the script! I've run it a few times now and it has been successful in remove a good chunk of the resources already. I'll continue to run the script to clean up the remaining objects.
Do you think it would be a reasonable thing to have this script run nightly to prevent this issue from getting this bad again until a formal fix is implemented in Rancher?

@felipe-colussi one additional update here, I finished running the script, but noticed that there were still ~10k resources remaining. Looking into this a bit more, it seems like the same issue happens with the local snapshots as well.
Running:
:~$ kubectl get ETCDSnapshots -n fleet-local -o json | jq '.items[]|select(.status.missing==true)|.metadata.name' |tr -d '"' | wc -l
10410

(Replacing -n fleet-default with -n fleet-local)
I updated the script to just use --all-namespaces instead to deal with this.

@BenB196.
Sorry I didn't had the snapshots enabled on my local cluster, and I didn't noticed  that they would be stored on a different namespace.
That script won't work just changing by changing -n fleet-local  to --all-namespaces. The  delete ... --all-namespaces won't work.   You probably will need to change the namespace to the one you need or use a variable namespace.
If you didn't use the sleep something on this lines will do:
#/bin/bash
mapfile -t my_array < <( kubectl get ETCDSnapshots --all-namespaces -o json | jq '.items[] | select(.status.missing==true) | .metadata.name + " " + .metadata.namespace'| tr -d '"' | head -n 3000)

for data in "${my_array[@]}"; do
  arr=( $data )
  kubectl delete ETCDSnapshots "${arr[0]}" -n "${arr[1]}"
done

About running it every night: it would be OK, is not the prettiest solutions but it will have the job done.
Today I'll talk with the team, and probably will try to find a better workaround that doesn't include running a bash script every-night.

For internal tracking:
The scope of this issue will be limited to  “Stop storing ETCDSnapshots for imported clusters”

Validation Template
Root Cause
We had no way of setting the trace-level for rancher-agent .
What was fixed, or what change have occurred
Now is possible to to add trace-level using   Agent Environment Vars.
CATTLE_TRACE or RANCHER_TRACE must be  set to true
Areas or cases that should be tested

Using rancher Create a RKE2 cluster  with ETCD snapshots configured, this should work normally.
Create a RKE2 cluster with ETCDSnapshots configured directly on the RKE2. This should not create ETCDSnapshots on rancher.

What areas could experience regressions
None.
Are the repro steps accurate/minimal?
Yes. After checking the scope of the Issue was limited to not storing  ETCDSnapshots for imported RKE2 clusters.

Ticket # - Test Results - Pass ✅
Verified with (HA Helm or Docker) on Rancher v2.8:
Scenario 1 -

Installed RKE2 with etcd support and S3 snapshots
Confirmed that the cluster could be imported with no issues
Confirmed that no snapshots were created on the rancher side.


Add support for Nimiq

Name: Nimiq
Symbol: NIM
Documentation: Protocol Reference
Checklist

[x] Coin configuration

[x] Coin Type / Decimals
[x] Recommended Block Explorer (Nimiq.Watch)

[x] Account / Address

[x] From Private Key
[x] From Public Key
[x] From/To String (User-Friendly Address)

[x] Transaction

[x] Serialization
[x] Signing

[ ] C interfaces

[ ] Validate generated code in Android and iOS projects → in progress

[ ] Tests

[x] Unit Test
[ ] Integration Test

[ ] RPC endpoint

View progress here

Nimiq RPC documentation
Setup

Download the Nimiq source (nimiq-network/core)

Prebuilt binaries: https://nimiq.com/de/#downloads
From source

Clone git@github.com:nimiq-network/core.git
npm install
Binary is under clients/nodejs

I could also throw together a Docker image for you

Write a config file (sample)
Start and sync Nimiq: ./nimiq --config=sample.conf
Implement a client

RPC documentation

Sample config
For a light node listening on wss://nimiq.trustwallet.com:8443 (that serves other peers).
Running a node that doesn't listen is also possible with the dumb mode, which is more resistant against DoS attacks.
Regardless of dumb mode or not, an RPC server is listening on 8648.
{
    host: "nimiq.trustwallet.com",
    port: 8443,
    tls: {
        cert: "./my.domain.cer",
        key: "./my.domain.key"
    },
    protocol: "wss",
    type: "light",
    network: "main",
    statistics: 60, // Print stats every x seconds
    rpcServer: {
        enabled: "yes",
        port: 8648,
        allowip: ["<IP_ADDRESS>/24"],
    },
    log: {
        // Possible values: "trace", "verbose", "debug", "info", "warning", "error", "assert"
        level: "info",
    }
}

Do you guys have public rpc endpoint for testing?

@hewigovens Not that I know. I can start one though.

@hewigovens We now have https://trust-us-01.hexapool.io/
If you have questions during development, I am @terorie on Telegram.

Thanks, I will test it out


Use std::memcpy when targeting little-endian x86_64 with GCC

Noticed pretty big performance boost on Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz, when compiled using GCC and using memcpy for copying associated data/ plain text/ cipher text bytes instead of for loop based construction.
For now on, when it's x86_64, native byte order is little-endian and GCC is in use during compilation --- memcpy is preferred choice.

Also updated benchmark results.


JSON API and test suite for web UI

Introduction
This change set offers a light take on "API" functionality for the web UI.
An initial suite of tests covering all of the endpoints (except reboot/shutdown) has been provided. These are high-level, happy path tests, which provide some basic assurance that the web UI features work as expected in normal circumstances.
Changes
For convenience, a summary of changes:
web.py and common module

Requests sent with the Accept: application/json header get a JSON response
All other requests get the existing render_template() HTML response
Calls to flash() are superseded by a single call to response(message=str|list)
Common data (e.g. used in the base template) are now accessed via the env var in templates
A new /env endpoint exposes the above data for API clients
Data previously passed to render_template() that wasn't JSON serializable has been amended at source
Fixed minor bugs encountered that blocked tests passing

Dependencies and Tooling

Added requirements-dev.txt to separate production and dev dependencies
Added pytest and pytest-httpserver as dev dependencies for running tests
Addedflake8 and black as dev dependencies for code style feedback and reformatting
Updated Dockerfiles to allow testing of more features

Try It Out
Build and launch Docker containers:
git clone https://github.com/nucleogenic/RASCSI.git --branch webui-json-responses
cd RASCSI/docker
cp docker-compose.override.yml.example docker-compose.override.yml
docker compose build --no-cache
docker compose up

Install dependencies:
(new terminal, from 'RASCSI' dir)
cd python/web/
python3 -m venv venv_test
source venv_test/bin/activate
pip install -r requirements.txt
pip install -r requirements-dev.txt

Run tests:
pytest -v

Create image via API:
# Login
curl -v --header "Accept: application/json" \
--cookie-jar rascsi_cookies.txt \
--data "username=pi&password=rascsi" \
 http://localhost:8080/login

# Create image
curl -v --header "Accept: application/json" \
--cookie rascsi_cookies.txt \
--data "file_name=foo&type=hds&size=1" \
http://localhost:8080/files/create

Run tests against a real RaSCSI:
Disclaimer: Make a backup first!
Any attached devices will be automatically detached when the test suite starts.
Each test attempts to clean up the resources it creates.
Firewall note: pytest-httpserver binds to a random port on the computer running the tests, and the RaSCSI instance needs to be able to connect in to this. This is required for the following tests:test_download_url_to_afp_dir, test_download_url_to_images_dir, test_download_url_to_iso and test_extract_file. Please adjust your firewall configuration as necessary.
pytest -v --base_url=http://{RASCSI_IP}/ \
--rascsi_username=pi \
--rascsi_password=raspberry \
--httpserver_host={YOUR_COMPUTER_IP} \
--httpserver_listen_address=<IP_ADDRESS>

Future Direction
Some other ideas, which don't quite fit in this (already large) PR, and in no particular order:

Run Pytest tests in CI
Add stateless authentication
Update development documentation (workflow with Docker, Pytest, etc)
Create API documentation and guidelines for API stability
Move more data from messages property to individual properties in data property
Reduce cases where multiple messages originate from a single action

Credits

Special thanks to @erichelgeson for the ideas and inspiration from the newweb branch

@nucleogenic I haven't used Docker in a long while, but I assume there's an obvious additional dependency here in the Docker executable and libraries themselves? For instance, if I want to run the Docker container on my Windows 10 PC, I basically just download and install a package from their website, right? And then execute the first few bootstrap commands in a Windows shell compatible fashion.

To run the tests from a Linux Docker host:
pytest -v --httpserver_host=<IP_ADDRESS> --httpserver_listen_address=<IP_ADDRESS>

Where <IP_ADDRESS> is the IP address of docker0 (or relevant interface in your setup).


Questions about capturing an image and mass_storage_id

Hello. I am trying to build UNIX program that captures an image from Bebop.
I followed this post (https://github.com/ARDroneSDK3/Samples/issues/14) and found ARCOMMANDS_Generator_GenerateARDrone3MediaRecordPicture.
I compiled and ran the function, but I could not find where it saved an image. I passed mass_sotrage_id as 1. I am not fully sure whether I am using the function correctly.
Could you tell me an instruction of using ARCOMMANDS_Generator_GenerateARDrone3MediaRecordPicture function and how to find a location where it saved an image?
I appreciate your help!
Sincerely,
Dong Ki Kim

Hi,
When you request a picture from the Bebop, the picture is stored in the internal memory. That's the same behavior for video records.
To search and download the media, we provide the libARDataTransfer library (which, basically, is a wrapper around an FTP connection).
You can see a sample for this library here. The sample code is for Objective C, but you can make it work with plain C with a few changes.
Regards,
Nicolas.

I appreciate your comment!
I wanted to clarify. I am trying to make a program that shows real-time images, that are taken from flying Bebop, on a computer (Linux).
For this task, would be using libARDataTransfer library the best way?
Thank you for your time!
Best,
Dong Ki Kim

Hi,
If you want real time, you should go with the streaming option.
We have a basic sample app for unix here, which will display the live streaming through mplayer (if installed), that you can use as a base for your application.
The MediaRecordPicture and MediaRecordVideo functions are not suited for real time display.
Regards,
Nicolas.

Thank you for your quick reply!
Yes, when I first received Bebop, I looked into the UNIX examples (https://github.com/ARDroneSDK3/Samples/tree/master/Unix) and successfully ran them.
However, I faced with two problems:

Somehow streaming quality was not good. I also have ARDrone 2.0. but the streaming quality of Bebop was not better than ARDrone 2.0. I thought this did not make sense because, from my knowledge, Bebop has a better built-in camera than ARDrone 2.0.
I could not find C code for taking snapshot of mplayer in real-time. In order to make my application work, I need a real-time image, not a video, but I could not find a way to do it.

Could you give me your advice? Thank you again!
Regards,
Dong Ki Kim

For my question #2, this might be not the best solution, but I could use 'S' to continually take a snapshot of mplayer.

Would there be any way to receive a real-time image without using mplayer?
From FreeFlight App source codes, I see many functions that are related to taking a picture (ex. ARCOMMANDS_Generator_GenerateARDrone3MediaRecordPicture). Do they all save an image in the internal memory?

Yes, taking a picture saves it on the internal memory.
To get the best streaming quality, be sure to send piloting commands on a regular base (see this issue for more information). You should also use a 5Ghz wifi channel.
Best regards,
Djavan


split-attrs: merge and concatenate

Following on from https://github.com/SciTools/iris/issues/4988
Since (we think) the number-1 use of equalise_attributes is to align cubes for merging, it begs the question whether merge + concatenate may have some cases "broken" by the adoption of split-attributes.

We are mostly concerned about cases where some new behaviour might break existing code.
From offline discussion with @trexfeathers  ...
Looking into the merge + concatenate code, it seems that

merge compares attributes, in the course of doing match checking, but it does not interrogate their contents in any special way (++)
concatenate does not deal with cube attributes specifically, but only does whole metadata comparison

(++)
Except ... merge does make a key-by-key analysis of mismatched cubes to construct an account of "why a merge failed",
This means that, in the new world, we could get a.attributes != b.attributes, but then this message, being based on comparing values per-key, would conclude that there are no keys will different values, leading to a message like
"cube.attributes values differ for keys: ."  (an empty comma-separated list, i.e. invisible!)
So, that bit could be improved -- and actually, we could make use of metadata.attributes.difference instead.


Respect None build action from Web.config

I've created a root project (CoreySmith.Project.Website.XConnect) to deploy xConnect modules to the xConnect web root. At the moment, these projects only contain Model definitions, no code, so I'd rather leave the xConnect Web.config out of the solution entirely. If I set the Build Action of the Web.config in my root project to None, I get the following error on build: No transforms were supplied. Changing the Build Action to ExcludeFromPackageFiles by directly modifying the .csproj gives the same error.
Adding <ExcludeFromPackageFiles Include="Web.config" /> to my PublishSettings.xConnect.targets file indeed stops the Web.config from deploying, but I'd prefer to keep my PublishSettings.*.targets files as lean as possible.
With <ExcludeFromPackageFiles />, if you do a one-click publish in Visual Studio, the Web.config will still get overwritten; so you actually need to add <ExcludeFilesFromDeployment>Web.config</ExcludeFilesFromDeployment> in a <PropertyGroup />  in PublishSettings.*.targets to make sure that it's not deployed under any circumstance. Supporting the None build action would be a lot simpler.

As per our out-of-band discussion, this is actually a quirk of the standatd WPP / build targets and not a HPP-specific thing. Exclusing the Web.config from publish is the only way of prevent it from being included.


Separate building extension and running tests

Following http://www.appveyor.com/docs/appveyor-yml

Thanks!


Bugfix: skip exceptions when looping over backups.

When dirs other than backups exist in the destination,
the call to time.strptime raised an error in delete-old-backups. It is
safe and correct to skip these.

Thanks for the fix!


Add wild card support in search_SAMADhi

Note that SQL wildcards are already supported:

Use % instead of *
Use _ instead of ?

I've opened https://github.com/cp3-llbb/SAMADhi/pull/4 to be able to use bash-like wildcards.


[Rolling] Enable pull request testing for system_tests.

From the discussion in https://github.com/ros2/system_tests/pull/441#issuecomment-657741298 enables pull request testing for the system_tests repository on Rolling.

Should PR jobs also be enabled for other distros?

Should PR jobs also be enabled for other distros?

@jacobperron do you want system_tests PR testing enabled in Foxy? It is the only other distro with a system_tests entry.

do you want system_tests PR testing enabled in Foxy? It is the only other distro with a system_tests entry.

@nuclearsandwich Yes please, if you'd be so kind :)


Implicit ordering of map elements in a scenario

For various reasons, I wrote a filter in python that would read a yaml file and write the same (or modified) yaml file, which would then be used as the configuration input to crank coordinator.
It seems that the order of callouts in the scenario's map is important.
A working scenarios is, for example https://github.com/aspnet/Benchmarks/blob/7bd91967745c974d5fc6ccd1640296d439475793/src/BenchmarksApps/Kestrel/PlatformBenchmarks/platform.benchmarks.yml#L67
scenarios:
single_query:
db: ...
application: ...
load: ...
This ordering works.  The db is spun up first, then the application that will talk with the db (using the connection string, which I had a mistaken concern was  being modified), then the load.
But, of course single_query is a map, so the call-out ordering of keys should not matter.
However, the python filter will print out the key:value pairs in a map in key-alphabetical order.  This results in the enumeration order application: db: load: which for crank's apparent iteration order is the wrong order for the whole stack,  The application starts before the db.
If this were a Makefile, I'd say there was one or more dependencies missing :)

Is there any solution to this on the horizon?

Is there any solution to this on the horizon?

That's interesting, it seems like it works fine in dotnet, as in the order is kept in the dictionary while iterating over it. However I think you are right that the order could be lost in terms of yaml specs, and it should be an array with a name property instead if the order matters. I'd say that optionally we could have an order property on each service such that if you specify it then it would order by it.
Using an array would be a breaking change and all configurations would have to be rewritten. Technically we could detect the format and adapt it, while printing a warning. In the meantime adding an order property would be a solution, and it could be used if any service has this. This would not be a breaking change.


Fix errors in EditCommandTest test cases.

Coverage decreased (-0.4%) to 90.319% when pulling 2c9612296a8f9e5f886d55be0732f70711230526 on Li-Shicheng:master into 17d803ac7e1d23dc949b1c4aa3902cc0ddee2dc2 on CS2103JUN2017-T4:master.


Looking for off-site communities

We're looking for more off-site open source communities to participate in the off-site sample! If you are part of a community that works primarily somewhere other than GitHub (e.g. hosted elsewhere, or via mailing list, etc.), please fill out this form and we'll send you a community-specific link and email template to share.

Are you looking for survey responses specifically from developers of non-GitHub-hosted projects, or from any member of the broader community associated with a project?

Any member! We're interested in hearing from anyone engaged with an open source project, definitely not just developers.
From our design principles:

The universe of this study is anyone who uses or otherwise engages with open source technology and development, whether passively or actively through contributions. This is a broader definition of this community than commonly employed, and we believe one of the major contributions of this study will be a dataset that allows for exploration of open source consumers alongside contributors.

The survey 🚢d! 🎉


[posix] use multicast to simulate radio

This PR changes to use multicast to simulate radio transmission. This
avoids sending a frame multiple times, and also avoid duplicated packets
in captures which is annoying when analyzing traffic.
This implementation uses two separate sockets for tx and rx. This is
because the current implementation of thread-cert requires sniffer
reports the exact sender of each packet. This issue can be addressed by
stateful tracking state of each node, but I don't have the cycles for
now.

Another thought is we may use different RADIO_GROUPs to simulate different channels.


Move form options out of the config and into the controller

Overwriting default form options (max field sizes, for example) is currently in the config. This is clunky and should be moved to the controller, probably in the exports section like the updated CORS functionality.

Created a controller config object that now contains both form options and CORS headers. Flexible enough to be used for potential future controller- and action-level configurations.


Feature request: Add include_init_carry to RNN

RNN returns as the output sequence the sequence of new carries at each timestep, but excludes the initial carry. Inside RNN.__call__, the initial carry gets overwritten by the line carry, outputs = scan_output. Feature request: Add a flag include_initial_carry: bool that includes the initial carry in the output sequence.

Hey @carlosgmartin, RNN.__call__ only returns the last carry to the user. Are you getting the entire history of carry states? If so this is a bug. Can you share the code you are using?

@cgarciae Thanks for getting back to me. Here's an example of what I mean:
import jax
import flax

module = flax.linen.RNN(flax.linen.GRUCell(), 4)
key = jax.random.PRNGKey(0)
x = jax.random.uniform(key, (32, 20, 3))
w = module.init(key, x)
y = module.apply(w, x)
print(y.shape)  # (32, 20, 4)

A recurrent cell like GRU maps an input and state to a new state. The output of the RNN is the sequence of new states (the new state being considered the output of the recurrent cell at each timestep). I was wondering if it would be possible to include the initial state in this sequence. The result would be a sequence of the same length as the input, plus 1. This corresponds to the RNN making a prediction at position zero, before it has seen any of the input sequence.

You can get the initial carry by using the GRUCell.initialize_carry static method. Not sure we want to add this to the RNN API.

@cgarciae Like this? Would you suggest any changes vis-a-vis conforming to the flax API/style?
import jax
from jax import random, numpy as jnp
from flax import linen as nn

def prepend(item, seq, axis):
    return jax.tree_map(
        lambda item, seq: jnp.concatenate([jnp.expand_dims(item, axis), seq], axis),
        item,
        seq,
    )

class FullRNN(nn.RNN):
    def __call__(self, *args, **kwargs):
        y = super().__call__(*args, **kwargs)
        init_key = kwargs.get("init_key", None)
        if init_key is None:
            init_key = random.PRNGKey(0)
        y0 = self.cell.initialize_carry(init_key, y.shape[:-2], y.shape[-1])
        return prepend(y0, y, -2)

def main():
    module = FullRNN(nn.GRUCell(), 4)

    key = random.PRNGKey(0)

    key, subkey = random.split(key)
    x = random.uniform(subkey, (32, 20, 3))

    key, subkey = random.split(key)
    w = module.init(subkey, x)

    y = module.apply(w, x)
    print(y.shape)  # (32, 21, 4)

if __name__ == "__main__":
    main()

I'm using this in an autoregressive recurrent generative model where

token N is stochastically generated from hidden state N
hidden state N+1 is generated from hidden state N and token N

The initial (i.e., zeroth) token to be generated has no preceding tokens as input.

Not sure about using tree_map just to call jnp.concatenate, isn't y0 just an array?
Other than that it looks good.

Thanks for the feedback. I was considering cases like LSTMCell or ConvLSTMCell where it's not just an array.

Converting this to a discussion since it seems to have been a misunderstanding.


MAINTAINERS: Add ngphibang as collaborator in Drivers: Video

Adding Phi Bang Nguyen (ngphibang) as collaborator in Drivers: Video.

+1


Fastrtpsgen 1.5 produces trailing include guard comment with inconsistent naming

For file XXX.idl, the generated file XXX.h defines the include guard "XXX_H", which is fine.
However, the corresponding generated XXXPubSubTypes.h defines an opening guard of "XXX_PUBSUBTYPES_H", and a closing guard comment of " _Xxx_PUBSUBTYPE_H". The inconsistency here is:

The use of all caps for the opening and leading caps only for the closing.
The use of plural form for the opeing and singular for the closing.

This was fixed on commit 4d5f197385b716fe6249003f141174b95c01e7ed


Fiat Linea

Hi, my car is fiat linea, I'm using Arduino nano towards connections. but unfortunately, the connection failed.

did anyone resolved this? even I am getting same error in extended mod, but in standard mode I am getting the data.

Have you solved it now? I have the same problem.

did anyone resolved this? even I am getting same error in extended mod, but in standard mode I am getting the data.

How to select a mode? Isn't this automatically detected in the OBD2 library


[TF] Change APIs to use Int instead of Int32.

Friend PR: https://github.com/apple/swift/pull/24012

This looks really great Dan!

Ready for review.

Locally verified that all tests pass. Merging now, then testing/merging https://github.com/apple/swift/pull/24012.
Note: I built a toolchain and swift build almost works:
$ swift build
Compile Swift Module 'DeepLearning' (8 sources)
/Users/danielzheng/swift-dev/tensorflow-swift-apis/Sources/DeepLearning/Layer.swift:49:35: error: ambiguous use of 'inference'
        return withLearningPhase(.inference) {
                                  ^
/Users/danielzheng/swift-dev/tensorflow-swift-apis/Sources/DeepLearning/Context.swift:28:10: note: found this candidate
    case inference
         ^
TensorFlow.LearningPhase:3:10: note: found this candidate
    case inference
         ^
/Users/danielzheng/swift-dev/tensorflow-swift-apis/Sources/DeepLearning/Layer.swift:60:35: error: ambiguous use of 'inference'
        return withLearningPhase(.inference) {
                                  ^
/Users/danielzheng/swift-dev/tensorflow-swift-apis/Sources/DeepLearning/Context.swift:28:10: note: found this candidate
    case inference
         ^
TensorFlow.LearningPhase:3:10: note: found this candidate
    case inference
         ^

If LearningPhase.inference is explicitly qualified, swift build succeeds.


mat-select model don't update after removing options programmatically

Bug, feature request, or proposal:
Bug
What is the expected behavior?
I expect to get the model updated and the change event called after remove selected options of the options list.
What is the current behavior?
The model don't update and the change event is not called, but visually is updated.
What are the steps to reproduce?
1 - http://plnkr.co/edit/RNAeZbaNrtMQRrjC7IDc?p=preview ;
2 - Click on "Remove options";
3 - mat-select visually update;
4 - model dont update and change event is not called;
What is the use-case or motivation for changing an existing behavior?
If I change the options for any reason, I expect to get the change events and the model to be updated in order to keep data consistent;
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
master
Is there anything else we should know?
No

We will need to determine if changes should propagate as you described.
Personally, I think that if we do propagate some of these changes up, we need to have a different emitter for non selected value affecting.

@josephperrott event if be decided that changes should not propagate, it's necessary to have a way to force this changes thru the component api in order to update the model accordingly with the new set of options.
What do you think?

Any updates here? there's a pull request but no updates ))


Research on security

Major security concerns for online compilers/interpreters are malicious codes submitted from users. In the current system, the code is evaluated on the server-side. From the server perspective, the server may happen to run the potentially malicious code. Some potential problems may arise, at least but not limited to:

DDoS attack to exhaust the server's resources.
Manipulate file structures using os module

There are two main ways of addressing this security concern: running code on client-side and on server-side.

Client-side
In short, we can execute everything client-side using JavaScript, and possibly use tools like Brython that essentially converts Python script to JavaScript and in order for it to be executed on the client-side.

Keypoints

Not work with some of the libraries.
Depend on specific versions of library.
Tend to be 3 to 5 times slower.

Server-side
2.1 Use Sandbox
General ideas are to use the concept of sandboxing that lets the potentially malicious code in that sandbox. See more discussion here. In Python, we have some discussions here.
2.2 Use RestrictedPython
RestrictedPython 4.0b2 is a defined subset of the Python language which allows to provide a program input into a trusted environment.

Keypoints

Faster
Not depend on specific versions of library.

Thoughts on this issue are very appreciated.

http://doc.pypy.org/en/latest/sandbox.html

Running untrusted python code
Safely executing arbitrary code
Best practices for execution of untrusted code

https://docs.docker.com/engine/security/security/

We are prioritizing performance, so closing this for now.


iPad touch support

with this changes the contextmenu works perfect for iPad.
See my issue #1662 for more information

Thank you for the contribution, but unfortunately I can not merge this.
The dist files are generated by a build process, you need to modify the src files as stated in the contributing section of this repo. Please modify the actual jstree.contextmenu.js file and open a new pull request so that I can review.
Best regards,
Ivan


Is the RefundableCrowdsale constructor incomplete?

https://github.com/OpenZeppelin/zeppelin-solidity/blob/84bffb854a1bdb4735aabd457e97eecc8f3f2d02/contracts/crowdsale/RefundableCrowdsale.sol#L26
I could not find any other references in the contract to the "wallet" parameter?

It is defined in Crowdsale.sol. Since RefundableCrowdsale inherits from FinalizableCrowdsale which inherits from Crowdsale, it has access to the wallet variable there.

Aye aye captain:ok_hand:

Thanks for answering this @nedodn!


More notifications setup examples

The bot would recognize "set up a notification" but not "setup a notification" haha

Looks good!