Optimize exception cleanup

Hey everyone, we found that compiling an exception (everything before Notification#deliver_exception_payload) takes around ~340ms on your typical Rails exception. If you just test something like
begin
  raise ArgumentError, "Test"
rescue => ex
  Bugsnag.notify(ex)
end

It will be fast because the stack trace is tiny. In your Rails app, it's more common to see 40-60 lines in a stack trace. In your worst case such as IO errors, they can get 3 layers deep of cause so you're really cleaning up 160 lines of exceptions which includes an additional ~4 arrays to clean.
It turns out that duping a Set is a major time sink. Removing the dups gets it down to 340ms to 40ms. We also call freeze on the strings which causes Rails to do some magic and treat them as constants. As well as moving the encoding options to a constant to reduce unnecessary garbage.
https://github.com/zanker/bugsnag_benchmark has an example of the performance of the old code vs the new. I also included a micro optimization one which further improves performance.
This code alone is a ~50% improvement in the benchmark, and in our real world test it's about 90%, as the 340ms -> 40ms change includes the entirety of Bugsnag.notify.

Thanks for this. Neat trick with memoizing the clean object in the recursion avoider :).
There's probably plenty of low-hanging performance fruit if you're feeling keen ;)

(released in v2.8.5)


Show question elapsed answering time

Show the elapsed time since a question started being answered to course staff and the person asking the question.
Any alternative suggestions for UI are welcome!
Student view:

Staff view:

Fixes #10

Yep! A lot of these issues are blocked by a sensible way to do migrations to add such course/queue flags, so I'll revisit this once that lands.


[Urgent] Llama3 NOT Working in PPO Trainer

Hi, I'm using fine tuned Llama3 model trained using unsloth. I noticed the model needs for_inference() to make sure there is no error for model.generate().  However, in PPO trainer, model is passed as AutoModelForCausalLMWithValueHead.from_pretrained(unsloth_model) and PPO trainer call model.generate() directly. There is the error again, therefore. Is there any way to get rid of the issue or avoid using for_inference()? Much appreciated.
model = AutoModelForCausalLMWithValueHead.from_pretrained(unsloth_model)
response_tensors = ppo_trainer.generate(query_tensors, **generation_kwargs)

Traceback (most recent call last):
File "/home/jovyan/work/ppo_llama3.py", line 156, in
response_tensors = ppo_trainer.generate(query_tensors, **generation_kwargs)
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/trl/trainer/ppo_trainer.py", line 469, in generate
response = self._generate_batched(
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/trl/trainer/ppo_trainer.py", line 556, in _generate_batched
generations = unwrapped_model.generate(**padded_inputs, **generation_kwargs)
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/trl/models/modeling_value_head.py", line 204, in generate
return self.pretrained_model.generate(*args, **kwargs)
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/peft/peft_model.py", line 1638, in generate
outputs = self.base_model.generate(*args, **kwargs)
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/transformers/generation/utils.py", line 1989, in generate
result = self._sample(
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/transformers/generation/utils.py", line 2932, in _sample
outputs = self(**model_inputs, return_dict=True)
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/accelerate/hooks.py", line 169, in new_forward
output = module._old_forward(*args, **kwargs)
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/unsloth/models/llama.py", line 857, in _CausalLM_fast_forward
outputs = fast_forward_inference(
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/unsloth/models/llama.py", line 794, in LlamaModel_fast_forward_inference
seq_len = past_key_values[0][0].shape[-2]
File "/opt/conda/envs/unsloth_env/lib/python3.10/site-packages/transformers/cache_utils.py", line 314, in getitem
raise KeyError(f"Cache only has {len(self)} layers, attempted to access layer with index {layer_idx}")
KeyError: 'Cache only has 0 layers, attempted to access layer with index 0'

Hmm I think someone did mention PPO does not work - I think the generation step isn't working properly

Hey, Yes, the generation is stuck forever if I just load the model directly using the official's code. What I did is to load the merged GGUF fine tuned Llama3 8b bnb 4bit model into the AutoModelForCausalLMWithValueHead by using load_in_4bit = True, which can make the generation proceed. However, the generation output loses precision largely. Do you have any insight why the loading does not work still?
FYI, I the merged model does not lose precision if I load into AutoModelForCausalLM.

`
ppo_config = {"mini_batch_size": 1, "batch_size": 1}
config = PPOConfig(**ppo_config)
ppo_trainer = PPOTrainer(config, model, ref_model, tokenizer)
3. encode a query
query_txt = "This morning I went to the "
query_tensor = tokenizer.encode(query_txt, return_tensors="pt").to(model.pretrained_model.device)
query_tensor
generation_kwargs = {
"min_length": -1,
"top_k": 0.0,
"top_p": 1.0,
"do_sample": True,
"pad_token_id": tokenizer.eos_token_id,
"max_new_tokens": 20,
}
device_type = 'cuda'
dtype = model.config.torch_dtype
if type(dtype) is str:
if   dtype ==  "float16": dtype = torch.float16
elif dtype == "bfloat16": dtype = torch.bfloat16
pass
if model.generate.name != "_fast_generate":
model._unwrapped_old_generate = model.generate
model.generate = unsloth.models.llama._wrap_fast_inference(model.generate, device_type, dtype, model)
pass
response_tensor = ppo_trainer.generate(list(query_tensor), return_prompt=False, **generation_kwargs)
response_txt = tokenizer.decode(response_tensor[0])
response_txt
`
I met the same problem. I replicated the example from trl and I solve it by extracting part of the code from FastLanguageModel.for_inference(model). The reason I did not explicitly call FastLanguageModel.for_inference is that it will fail after calling
model = AutoModelForCausalLMWithValueHead.from_pretrained(model)
and this is the error. it fails at calling internal_model.model
`
AttributeError                            Traceback (most recent call last)
Cell In[20], line 23
20     model.generate = unsloth.models.llama._wrap_fast_inference(model.generate, device_type, dtype, model)
21 pass
---> 23 FastLanguageModel.for_inference(model)
25 response_tensor = ppo_trainer.generate(list(query_tensor), return_prompt=False, **generation_kwargs)
26 response_txt = tokenizer.decode(response_tensor[0])
File ~/miniconda3/lib/python3.10/site-packages/unsloth/models/llama.py:2214, in FastLlamaModel.for_inference(model)
2212 internal_model = model
2213 while not hasattr(internal_model, "lm_head"):
-> 2214     internal_model = internal_model.model
2215 pass
2216 lm_head = internal_model.lm_head.weight
File ~/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1709, in Module.getattr(self, name)
1707     if name in modules:
1708         return modules[name]
-> 1709 raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'AutoModelForCausalLMWithValueHead' object has no attribute 'model'
`
New problem happens when calling ppo_trainer.step. I will get this error exactly as the problem I indicated in the another issue:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 28, 4096]], which is output 0 of MulBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
and I currently did not find way to solve this in ppo_trainer.

ppo_config = {"mini_batch_size": 1, "batch_size": 1}
config = PPOConfig(**ppo_config)
ppo_trainer = PPOTrainer(config, model, ref_model, tokenizer)

query_txt = "This morning I went to the "
query_tensor = tokenizer.encode(query_txt, return_tensors="pt").to(model.pretrained_model.device)
query_tensor

generation_kwargs = {
    "min_length": -1,
    "top_k": 0.0,
    "top_p": 1.0,
    "do_sample": True,
    "pad_token_id": tokenizer.eos_token_id,
    "max_new_tokens": 20,
}

device_type = 'cuda'
dtype = model.config.torch_dtype

if type(dtype) is str:
    if   dtype ==  "float16": dtype = torch.float16
    elif dtype == "bfloat16": dtype = torch.bfloat16
pass

if model.generate.__name__ != "_fast_generate":
    model._unwrapped_old_generate = model.generate
    model.generate = unsloth.models.llama._wrap_fast_inference(model.generate, device_type, dtype, model)
pass

response_tensor = ppo_trainer.generate(list(query_tensor), return_prompt=False, **generation_kwargs)
response_txt = tokenizer.decode(response_tensor[0])
response_txt

I met the same problem. I replicated the example from trl and I solve it by extracting part of the code from FastLanguageModel.for_inference(model). The reason I did not explicitly call FastLanguageModel.for_inference is that it will fail after calling
model = AutoModelForCausalLMWithValueHead.from_pretrained(model)
and this is the error. it fails at calling internal_model.model
AttributeError                            Traceback (most recent call last)
Cell In[20], line 23
     20     model.generate = unsloth.models.llama._wrap_fast_inference(model.generate, device_type, dtype, model)
     21 pass
---> 23 FastLanguageModel.for_inference(model)
     25 response_tensor = ppo_trainer.generate(list(query_tensor), return_prompt=False, **generation_kwargs)
     26 response_txt = tokenizer.decode(response_tensor[0])

File [~/miniconda3/lib/python3.10/site-packages/unsloth/models/llama.py:2214](http://region-9.autodl.pro:50571/jupyter/lab/tree/autodl-tmp/project/~/miniconda3/lib/python3.10/site-packages/unsloth/models/llama.py#line=2213), in FastLlamaModel.for_inference(model)
   2212 internal_model = model
   2213 while not hasattr(internal_model, "lm_head"):
-> 2214     internal_model = internal_model.model
   2215 pass
   2216 lm_head = internal_model.lm_head.weight

File [~/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1709](http://region-9.autodl.pro:50571/jupyter/lab/tree/autodl-tmp/project/~/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1708), in Module.__getattr__(self, name)
   1707     if name in modules:
   1708         return modules[name]
-> 1709 raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")

AttributeError: 'AutoModelForCausalLMWithValueHead' object has no attribute 'model'

New problem happens when calling ppo_trainer.step. I will get this error exactly as the problem I indicated in the another issue:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 28, 4096]], which is output 0 of MulBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
and I currently did not find way to solve this in ppo_trainer.

Hi, have you solved this issue now? I think you can use the merged model to load in valuehead to avoid any loading error, but it's still not training well on side. It's really annoying for researchers who wants to use RLHF... Hope it can be fixed soon.

No sorry sadly I did not have time to look at PPO / RLOO or ones with generation sorry :( I'll try if I have time, but sadly this'll have to wait sorry

ppo_config = {"mini_batch_size": 1, "batch_size": 1}
config = PPOConfig(**ppo_config)
ppo_trainer = PPOTrainer(config, model, ref_model, tokenizer)

query_txt = "This morning I went to the "
query_tensor = tokenizer.encode(query_txt, return_tensors="pt").to(model.pretrained_model.device)
query_tensor

generation_kwargs = {
    "min_length": -1,
    "top_k": 0.0,
    "top_p": 1.0,
    "do_sample": True,
    "pad_token_id": tokenizer.eos_token_id,
    "max_new_tokens": 20,
}

device_type = 'cuda'
dtype = model.config.torch_dtype

if type(dtype) is str:
    if   dtype ==  "float16": dtype = torch.float16
    elif dtype == "bfloat16": dtype = torch.bfloat16
pass

if model.generate.__name__ != "_fast_generate":
    model._unwrapped_old_generate = model.generate
    model.generate = unsloth.models.llama._wrap_fast_inference(model.generate, device_type, dtype, model)
pass

response_tensor = ppo_trainer.generate(list(query_tensor), return_prompt=False, **generation_kwargs)
response_txt = tokenizer.decode(response_tensor[0])
response_txt

I met the same problem. I replicated the example from trl and I solve it by extracting part of the code from FastLanguageModel.for_inference(model). The reason I did not explicitly call FastLanguageModel.for_inference is that it will fail after calling
model = AutoModelForCausalLMWithValueHead.from_pretrained(model)
and this is the error. it fails at calling internal_model.model
AttributeError                            Traceback (most recent call last)
Cell In[20], line 23
     20     model.generate = unsloth.models.llama._wrap_fast_inference(model.generate, device_type, dtype, model)
     21 pass
---> 23 FastLanguageModel.for_inference(model)
     25 response_tensor = ppo_trainer.generate(list(query_tensor), return_prompt=False, **generation_kwargs)
     26 response_txt = tokenizer.decode(response_tensor[0])

File [~/miniconda3/lib/python3.10/site-packages/unsloth/models/llama.py:2214](http://region-9.autodl.pro:50571/jupyter/lab/tree/autodl-tmp/project/~/miniconda3/lib/python3.10/site-packages/unsloth/models/llama.py#line=2213), in FastLlamaModel.for_inference(model)
   2212 internal_model = model
   2213 while not hasattr(internal_model, "lm_head"):
-> 2214     internal_model = internal_model.model
   2215 pass
   2216 lm_head = internal_model.lm_head.weight

File [~/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py:1709](http://region-9.autodl.pro:50571/jupyter/lab/tree/autodl-tmp/project/~/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1708), in Module.__getattr__(self, name)
   1707     if name in modules:
   1708         return modules[name]
-> 1709 raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")

AttributeError: 'AutoModelForCausalLMWithValueHead' object has no attribute 'model'

New problem happens when calling ppo_trainer.step. I will get this error exactly as the problem I indicated in the another issue:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 28, 4096]], which is output 0 of MulBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
and I currently did not find way to solve this in ppo_trainer.

Hey @danielhanchen. Any chances this error was fixed? I am encountering the same error.

hey, any chances it is fixed?
@danielhanchen

Much apologies on PPO - I'll try running PPO manually and seeing what's the issue

Hey @danielhanchen. Thanks for the reply, hope it would be fixed soon. If you can check RLOOTrainer as well that might be very helpful.


We shouldn't allow the user to click on the training button when training is already underway.

Also, we used to have a training animation @broham89

The block is ready. The animation is still missing.

I'll try to see if I can drop the animation to you today @dcalvom

training-animation.svg.zip
@dcalvom  make sure the svg is 24px high to be proportionate with other icons

@dcalvom is there a way to make this block global? So that if any agent is training than no training is allowed?
Also, is your block only on the UI side? it would be nice if the block was also on the API side in case someone tries to start training that way.

The training block in the API is already set!

Closed by https://github.com/samtecspg/articulate/commit/4b97971ad79a297bc10e259020524e834a6f82a5


API to fetch alerts by detector

Signed-off-by: Surya Sashank Nistala<EMAIL_ADDRESS>Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Let's add description for the PR, thanks!


Plugin not working (Sketch 51.3 & MacOS Sierra)

Hey, my plugin doesn't work. I'm using the old version 51.3, my licence expired just today and I can't update the whole Sketch. The plugin I'm using is version 2.7.7. I followed the known instructions (uninstall, quit Sketch, empty the plugin cache, reinstall) but it's not working either. The progress bar freezes right at the start, nothing happens. I tried the newest fork version, but its not working tho.
What can I do? I have to export my design files for the programmer, but in this case I can't do it without listing all the design specs manually. Thats pretty annoying.
Thanks in advance.

@utesahara1893  The plugin is working on Sketch version 56.3 with plugin version 2.8.0, uninstall it, and install it again here: https://github.com/ellunium/sketch-measure-latest

Hey. No, its not working. I followed your instructions, uninstalled the plugin and installed version 2.8.0, now the plugin doesn't hangs on the start of the spec export, it shows a error message when starting the export.

It says:
Error: layer.sharedStyleID is not a function. (In 'layer.sharedStyleID()', 'layer.sharedStyleID' is undefined)


Automated cherry pick of #8836: fix: support import rules for secgroups

Cherry pick of #8836 on release/3.5.
#8836: fix: support import rules for secgroups

Codecov Report

Merging #8837 (30fc56e) into release/3.5 (e3fe6c4) will increase coverage by 1.81%.
The diff coverage is 0.25%.

@@              Coverage Diff               @@
##           release/3.5   #8837      +/-   ##
==============================================
+ Coverage         6.03%   7.84%   +1.81%
==============================================
  Files              795     246     -549
  Lines           124282   33640   -90642
==============================================
- Hits              7503    2640    -4863
+ Misses          116114   30739   -85375
+ Partials           665     261     -404

Flag
Coverage Δ

aFlag
7.84% <0.25%> (+1.81%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/apigateway/handler/auth.go
0.00% <0.00%> (ø)

pkg/apigateway/handler/misc.go
0.00% <0.00%> (ø)

pkg/cloudcommon/db/domainresource.go
0.00% <0.00%> (ø)

pkg/cloudcommon/db/modelbase.go
4.51% <0.00%> (-0.08%)
:arrow_down:

pkg/cloudcommon/db/namevalidator.go
0.00% <0.00%> (ø)

pkg/cloudcommon/db/orderby.go
0.00% <0.00%> (ø)

pkg/cloudcommon/db/virtualresource.go
0.00% <0.00%> (ø)

pkg/cloudprovider/objectstore.go
2.57% <ø> (ø)

pkg/cloudprovider/securitygroup.go
0.00% <ø> (ø)

pkg/compute/regiondrivers/aliyun.go
0.34% <0.00%> (+<0.01%)
:arrow_up:

... and 495 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8738446...30fc56e. Read the comment docs.

/lgtm
/approve


feat: initial ci

Adds build scripts
Adds link check

Link check works. Permission added in settings requiring build and check to pass to be able to merge.


Implements ipnfo tool is_v4in6

yes. net library doesn't have a function implementing this. It is defined in netIP library ( which we are using as inspiration ) but its not imported

Ah, I intended to use it in practice, not just for inspiration @Harisabdullah ! Same is for all the other new tools.


Autoprefixer Question

Do you guys use autoprefixer in background; or this has to be seprate implementation?

We use a custom implementation aiming to be small and performant as its built for runtime prefixing rather than build time (which Autoprefixer is built for).
Yet, we implement the same data source and support a similar set of prefixes.


Playlist order desync

I was playing with friends and made a playlist on Youtube that we all could add songs to. But after we uploaded the playlist to Lethal Company, everyone's track order was different. While someone heard song Number 1, another person heard song Number 5 and another heard song Number 7.
I don't know if this is a problem with the game or this mod. Any idea what the issue could be?

I was playing with friends and made a playlist on Youtube that we all could add songs to. But after we uploaded the playlist to Lethal Company, everyone's track order was different. While someone heard song Number 1, another person heard song Number 5 and another heard song Number 7.
I don't know if this is a problem with the game or this mod. Any idea what the issue could be?

Привет! Насчет того что плейлист запускает не по порядку песни, это так и должно быть. А вот насчет что каждый слышит разные песни, так не должно быть. Скинь мне логи BepInEx. И скинь плейлист который запускаешь.

Sorry for the late reply, here is the log file from a testing session me and a friend just did, the issue was still present.
LogOutput - Kopie.log
And this is the Playlist we used.
https://www.youtube.com/playlist?list=PLvawL6zMrbXllpd01NTzDPlV-3Rfs0al1

Sorry for the late reply, here is the log file from a testing session me and a friend just did, the issue was still present. LogOutput - Kopie.log
And this is the Playlist we used. https://www.youtube.com/playlist?list=PLvawL6zMrbXllpd01NTzDPlV-3Rfs0al1

Привет! Не заметил ошибок в логах. Плейлист загружается. Я пока закрою тему. Если будет какие то проблемы, создайте новую тему либо откройте эту.


Clarify the scope of supported cases in todo spec in e2e demo

Below two cases are not supported yet in some emitters in @markcowl's todo spec pr. I'd like to clarify if both of them are in Jan e2e demo's scope.

Unsupported 1: Model is shared among mfd and non-mfd operations

TodoAttachment will be shared in a json operation createJson(link) and a mfd operation createForm(link). Since currently emitters would  treat mfd model differently so sharing here would be an issue for Java and JavaScript and I didn't check other language yet.
I am also clarifying if there is any limitation for these cases: https://github.com/microsoft/typespec/issues/5376.

Unsupported 2: Union cases with bytes type

I know other lanagues would only generate protocol methods for this. This seems to be a JS issue only, currently we can't handle the deserialization/serialization for union model TodoAttachment without a discriminator property.
Full support of union deserialization/serialization are problematic and a big issue in JS. Not sure we could scope to discriminated union for now?
union TodoAttachment {
  file: TodoFileAttachment,
  url: TodoUrlAttachment,
}

model TodoFileAttachment is File;

model File {
  contentType?: string;
  filename?: string;
  contents: bytes;
}

model TodoUrlAttachment {
  /** A description of the URL */
  description: string;

  /** The url */
  url: url;
}

cc @lmazuel   @joheredi  for awareness.

As we discussed, let's make the TypeSpec what we support, and keep this discussion for Jan/Feb 2025


New name?

As discussed in the Circle review, this app could benefit from a having a more unique name. Some ideas to kick things off:

Tender
Stacks
Ebenezer
Scrooge
Dough
Bucks
Balance
Overdraft
Coins

Maybe other language names? I'm italian so:

Portafoglio = Wallet
Soldi = Money
Denaro = Money
Contanti = Cash

Coin Purse
Porte Monnaie
Cash Box

Non-english words can work, but only in rare cases where the term is somewhat international. At the very least it needs to be easy to pronounce, so definitely not portafoglio :D

My suggestion: "FinoMan" from "Finance Manager"

@bertob I like Denaro
Comment from @fsobolev:

"Denaro" is easy to pronounce and I guess it refers to "dinar" which is a common term in some countries, so if this will work I agree that it's great

Denaro could work, yeah.

Goodbye Money, Hello Denaro!

https://github.com/nlogozzo/NickvisionMoney/releases/tag/2023.1.0-rc1


Plugins Feature Discussion Thread

Initially, we had discussed creating a plugin system with an Agentfile that would be placed at the root folder of the agent. However, with the addition of the info endpoint and specifically the idea of config_options within that route, what if plugins were displayed within that endpoint instead of in a separate file?
The idea here is to essentially define an external resource that we could pull that would define whatever extensions to the protocol exist, which would include detailed specs and any other relevant info.
This might require writing another spec for agent-protocol-plugin so I'm curious what the general opinion is.

I like this idea. It may be good to hear your proposed schema in the info endpoint for the plugin system.

I can see two categories of extensions:
Easy: This involves adding new endpoints under a fresh OpenAPI specification file. Users can implement any number of these, and then specify which ones they've adopted in an 'api/extensions' endpoint or something similar.
Hard: This pertains to introducing new parameters to existing endpoints, like 'user_id'. I'm uncertain about a clean method for this.
Much like the OpenAPI specification, developing the SDK is straightforward for the first type but challenging for the second. The easy extension type can be executed by setting up a FastAPI router for each extension. Subsequently, you can effortlessly integrate as many extension routers as needed into your implementation.
I'm still figuring out a solution for the second type.

Swifty via Discord

Well there's an even harder one, what if you wanted to implement a whole different type of server? A real-time websocket or graphql server or a stream of some kind or any number of crazy configurations an agent might implement. A plugin system is good for that, but the issue is how it would work and how it could do that while also minimizing development time. Somebody mentioned before that they wanted websockets and that's just one person, what if I wanted my agent to live stream video/audio/data to the client? A REST API isn't going to be able to cover that, so then what would a plugin look like?

Me via Discord

A simple extension to the REST API standard is relatively easy to implement, but if the plugin needs to be more than just an extension to the open api spec then it becomes significantly more complex.
How could we go about creating a plugin system that would allow agents to implement complex architectures as well as allow clients to easily implement those architectures without an excessive amount of code?

What are example plugin scenarios? I heard "auth" but that probably needs to be baked into the core protocol... Other concrete examples???
And how would a caller discover which plugins the Agent implements?

Caller can discover with the "info" endpoint. https://github.com/AI-Engineer-Foundation/agent-protocol/issues/39
With the protocol, we are trying to provide a minimum set of functionalities, but also ensuring collaboration from third parties to contribute to protocol extensions such as auth. I can imagine auth be implemented/needed differently by different companies, which is why we think it is good to have it be extension rather than baked into the core protocol.


multisig ui shows transfers even when not direct transfer

actions show native transfers even when account is not involved

Cool thanks


Component View does not properly sync with UI

Version and Platform (required):

Binary Ninja Version: 3.4.4189-dev
OS: Windows 10 Enterprise
OS Version: 1607
CPU Architecture: x64

Bug Description:
if you rename a function in the UI the component view doesn't update to reflect the change
Steps To Reproduce:
Please provide all steps required to reproduce the behavior:

Settings ->  UI -> Components-> "Enable New Symbols UI".
Select a function in the UI
Rename function in the UI

Expected Behavior:
The component view should sync with the UI so that any changes in the UI are reflected back in the component view
Screenshots:
If applicable, please add screenshots here to help explain your problem.
Additional Information:
Please add any other context about the problem here.

Duplicate of #3962


Feature/food endpoint

Describe your changes
Create new endpoint to generate nutritional food groups
Link to issue this resolves
Closes #310
Screenshot of changes

@ageddesi  has confirmed that codacy issues can be ignored.   I'll update this branch with the latest from dev, as it currently shows as needing to be updated.

Just taking a look at the files in a codespace - assuming all good I intend to approve this

Can we add a comment to each of the source files please to state where the data came from, the date it was obtained and a url directing to it if has one.

Hey @ageddesi! Just added that information in the comments.

Hi @GauravRanganath
Thank you for the contribution :)


s3 gateway API docs updates

A few changes to the s3 gateway API docs:

Rewrote docs to align back with the AWS docs, which underwent some recent changes.
Added a missing endpoint (GetBucketVersioning.)
Added a note about getting an object version.

Closes #4413

@svekars not sure if this is on your radar, but the docs netlify site appears to be rendering the wrong version of the docs: https://deploy-preview-4431--pachyderm-docs.netlify.com/latest/reference/s3gateway_api/

@ysimonson it is rendering the right version, just need to switch to master in the dropdown.

@svekars ready for another pass


Firefox: Loading extension ignored

Steps to reproduce
Tell us about your environment:

Puppeteer version: 1.11.0
Platform / OS version: MacOS
Node.js version:v9.5.0

What steps will reproduce the problem?
Using example from docs and any extension from store, just downloading and unpack it for FF and Chrome.
What is the expected result?
If put breakpoint on await browser.close(); and look at Chrome browser, installation process can be seen.
What happens instead?
In FF case, just empty page no matter how long to wait.
I suppose FF has different mechanism for addons and installing them. Probably it's just docs issue and missed some flag or something bigger and related to security stuff...

Using example from docs and any extension from store, just downloading and unpack it for FF and Chrome.

@denar90 The documentation at //docs/api.md covers Chromium Puppeteer. we'll come up with a similar write-up for Puppeteer-Firefox later on as the project matures.

Thanks :)


Merge prql-tool

@snth has created a CLI tool that utilizes PRQL queries to read data from various sources and write them as CSV or parquet files.
Compared to prql-compiler CLI, it is actually useful, which is why I think it should be the binary we are promoting to users to try out.
But I would suggest a name change, because PRQL is the language and it may be confusing to use prql my_query.prql. I suggest the name pipe.
TODO:

[ ] merge snth's fork (or split it into prql/pipe repo to avoid long compilation times)
[ ] setup CI/CD
[ ] change homebrew recipe to install pipe instead of prql-compiler
[ ] change website to promote pipe

Thanks @aljazerzen . I was thinking whether you might want to split it into a separate repo, also to avoid mixing of issues relating to the tool with issues relating to the language.
Regarding the name of the binary, I know it was a bit cheeky to go for prql but I thought it would be good marketing for PRQL the language. I think pipe is too generic and would prefer to keep prql in the name somehow, like prqls, prqlt or prqly (the last one just because it sounds like "prickly" (we'd have backport a meaning for it)). My first choice would be to just stick with prql though.

I definitely think it's worth getting prql-tool out there — it is already very cool!
I also agree that the current prql binary could be named prqlc — maybe @aljazerzen had mentioned it in conversation since I can't find an issue. I can open one.
I think we can see how prql-tool matures, and if it matures successfully then we could anoint it as prql. I am definitely fine having it in the repo now, but up to you @snth.
Even if someone only cared about the language being adopted by other tools, having something like prql-tool lets people see what's possible on their own data immediately (my perspective has probably matured on this over the past six months...)

I will submit a PR as soon I've completed the current batch of work which is to unify the output formats for the different backends. These used to be backend specific so the tables would look different on stdout and the parquet files would be named differently or not available on some backends. I'm just prototyping on the DataFusion backend for now but hopefully I can port that code to the other backends as well. Unfortunately that's where the arrow version incompatibilities come in. I'm hoping I can port the source code even if the materialised binary code will be backend arrow version specific.
At that point I think it will be worth getting into the hands of you folks for some testing as I realise that at the moment I have no CI integration or testing set up. I'll continue working on adding the functionality I still have planned in parallel to that. I just don't want to release it just yet as it currently no longer does what it says in the README while I ripped out the output formatting code for unification.
Regarding the name, I think prql-tool is too long for cli binary. How about prqlt analogous to prqlc? You can also "verb" that like "I just prqlt some of our PROD data to a CSV file for you to do some testing on." 😉

Hi, I've been thinking about the name more this weekend and I want to rename this to prql-query and the binary to pq. I didn't see any obvious clashes for another pq tool.

Great!
Re pq — any concerns about a parquet implication?

Nah, pq is great! Concise and similar to the well known jq.

Cool. The mention of jq in that recent HN columnq-cli thread is what might have actually given me the idea.

@max-sixty Do you have a particular concern around parquet? I wasn't aware of anything but after some more googling I did find:
import pyarrow.parquet as pq

as a fairly common idiom but that is in Python so it's not a direct clash. I see parquet as a major use case for this so some allusions to that I don't see as a bad thing as long as there's no direct clash.

In Ubuntu there's already a package called pq but that seems to be a game called progress-quest so presumably that's not hugely popular and a major clash at the shell for our target market.

The only other usage of PQ I'm aware of is for PowerQuery in the Excel and Power BI world. I think that's also distinct enough though that I think we should be ok.

Great — it reminded me of parquet, but not in some "don't call it py" obvious way — I would go ahead!

Do you want this to be in a prql-query subdirectory / workspace or should we maybe put this in a separate repo under the prql organisation, e.g. github.com/prql/pq?
I started looking at github actions for building release artifacts and it just seems to me that we would probably want a different release schedule for pq to prql. prql-compiler is only used in one place in pq and could easily import the prql crate. While we would want to bump the pq version for each prql version bump, I doubt that we would want to do the converse.

Do you want this to be in a prql-query subdirectory / workspace or should we maybe put this in a separate repo under the prql organisation, e.g. github.com/prql/pq?

That's totally fine — whatever you wish! I think fair point on the release cadence.
I think the only thing I would vote for is that we label it as something like "an experimental way to use PRQL to query data", and make it clear in the Readme that it's just one tool, and that PRQL is fundamentally a tool-agnostic language. I've been to org pages before and been confused on whether something is a tool / language / app / SaaS startup / way of life. (Similar to your comments on Discord today!). Hope that's reasonable.

I think the only thing I would vote for is that we label it as something like "an experimental way to use PRQL to query data", and make it clear in the Readme that it's just one tool, and that PRQL is fundamentally a tool-agnostic language. I've been to org pages before and been confused on whether something is a tool / language / app / SaaS startup / way of life. (Similar to your comments on Discord today!). Hope that's reasonable.

Yes, exactly. Having it in a separate repo would also make that clearer.

Apart from the release cadence, another argument for separating the repos is to not have the pq issues clog the PRQL issue backlog. I anticipate that most the pq issues would be around problems with the integrations with the backends like DuckDB, DataFusion, ... . Where an issue is about PRQL language features, they can be redirected to the PRQL repo but you don't want to have your guys time taken up with integration issues.

Super @snth , very much agree

@max-sixty Are you happy for me to transfer ownership of the pq repo over to the prql org now and carry on development there?

For sure!

Any concerns about the licensing before I transfer ownership?
Currently I've got Apache, MIT and UNLICENSE.
Not sure if there are any incompatibilities. I saw a lot of people dual-licensing with Apache and MIT and then I saw that ripgrep dual-licensed under MIT and UNLICENSE so I added that as well. 🤷♂

I would suggest to keep the amount of licensing to the minimum: a single permissive license, just as the main prql repo.
As far as I understand, having multiple licenses is the same as having any of them, which is the same as having only the most permissive one.

Ok, I think I'm happy that the 0.0.5 version now has enough capabilities to make it worth publishing.
Support this far is for:

CSV
Parquet
JSON
DuckDB
PostgreSQL
(Sqlite) - it should work in principle but I find it doesn't work for anything returning numeric fields.

I'll transfer ownership tomorrow evening and bump version to 0.1.0 probably.

I just tried to do the transfer but I got the following error:
You don’t have the permission to create public repositories on prql

@max-sixty I thought it would send you a request to approve but from my side it appears like it's done nothing. Not sure what the next step is here.
Btw, I also started looking into the crates.io publishing process and it turns out there is already a pq crate there pq - jq for protobuf. I've therefore renamed the repo to prql-query and I'll use that name for the crate. I've kept pq for the binary for now but we could rename that to prqlq (or prql) if need be.

@max-sixty I thought it would send you a request to approve but from my side it appears like it's done nothing. Not sure what the next step is here.

I think you should now have permissions. Thanks for asking — as ever this is an oversight not an intentional restriction!

Thank you that worked. prql-query is now owned by prql org! 🎉
Unfortunately now I'm no longer an admin on that repo so can't set up the branch protection and Discord webhook. @max-sixty can you please make me an admin on the prql-query repo? 🙈

Unfortunately now I'm no longer an admin on that repo so can't set up the branch protection and Discord webhook. @max-sixty can you please make me an admin on the prql-query repo? 🙈

Great!! And done!


Using a tunnel right away?

So I have the code below.
SomeComponentThatRendersToTunnel will try to put something inside footer-tunnel but it wont show on first render. My guess is that it's because it's below what will try to put something inside it. If I move the tunnel above the component it does render.
Is this a bug or expected? Is there a way to get around it?
<TunnelProvider>
                <>
                        <SomeComponentThatRendersToTunnel />
                        <div className={styles.flexFooter}>
                            <TunnelPlaceholder id="footer-tunnel" />
                        </div>
                </>
</TunnelProvider>

https://codesandbox.io/s/keen-bhaskara-vs4dl


Autocompletion popup gets stuck

Description
Autocompletion pop window does not go away and the suggestions are inconsistent. I attach the screen recording.
DBeaver Version
Version <IP_ADDRESS>309250303
Operating System
MacOS Ventura 13.5.2
Database and driver
Database name and version: Snowflake 7.33.1
Driver name and version: Snowflake 3.13.30
Steps to reproduce
Try to insert table names and you'll see that the pop up window gets stuck and does not go away, even if you switch to other programs like Chrome.
https://github.com/dbeaver/dbeaver/assets/114302087/b70b6d38-86b5-4ec2-8085-4c4b69165733
https://github.com/dbeaver/dbeaver/assets/114302087/4915429e-572f-490f-a011-ffc2e89305da
Additional context
No response

Thank you for bug report and detailed videos with explanation.
We already have an issue about it, so I'm closing this one as a duplicate of #19977


Travis builds failing on OSX

Hi,
Recently Travis CI updated the default OSX image. Xcode 7.3 is now the default.
https://blog.travis-ci.com/2016-10-04-osx-73-default-image-live/
This causes some of the picocoin builds to fail on OSX.
As a temporary workaround the old OSX image can still be used by adding the following to .travis.yml:
osx_image: beta-xcode6.1

However, this older image will only be maintained up until Monday, October 31st 2016.
For some reason the aes-util test fails on the newer OSX images:, example here:
https://travis-ci.org/aido/picocoin/jobs/166044768
Valgrind builds on OSX are also failing due to some memory issue with clist and coredefs tests, example here:
https://travis-ci.org/aido/picocoin/jobs/166044770
This issue may be reproduced by restarting any Travis OSX builds that have previously completed successfully.

I have noticed that some of the builds that fail on the latest OSX image do not fail when compiler optimization is disabled, For example the only difference between this build:
https://travis-ci.org/aido/picocoin/builds/173767999
and this build:
https://travis-ci.org/aido/picocoin/builds/173742334
is that optimization is disabled in the first.


chore: upgrade wireshark support from 4.0 to 4.4

It looks like the developers of Wireshark decided to deprecate the tvb_get_guintX and tvb_get_gintX-methods (and more) in Wireshark 4.4, which is what I run on my setup. This aims to make the crate compatible with Wireshark 4.4, though I'm uncertain how compatible it is with older Wireshark versions.
I have tested so that it works on Wireshark 4.4 but I have not tested on earlier versions. I also did a very simple change and didn't regenerate the bindings, just some sed-magic to change the functions. There may be other functions deprecated that will be unsupported.
The reason is simply that when I ran a compiled dissector that used u32s, e.g. as a len_field for a Vec<u8>, it couldn't find said function due to linking to another version.
The functions are still visible in the Wireshark source, but I imagine the inlining makes them disappear when it's compiled so they allow the old interface until they can remove it in a future version.
I would assume the changes to the *.stderr-files are due to me running Rust nightly or being generated with at least a somewhat newer version than when they were created.

[x] tvb_get_guintX-support
[ ] tvb_get_gintX-support

@seffradev Hi, are you still working on this? I'm looking into upgrading to 4.4.1 myself

Hi @amitrahman1026, no I'm not really looking into it anymore due to realizing that wsdf wasn't sufficient for my needs and I didn't have time to specify exactly what I needed due to it being just an experiment for a work thing.
Another thing I realized was that technically we shouldn't have to manually update to Wireshark 4.4 and instead wsdf aims to (but I couldn't get it to work) generate the FFI files required by Rust to compile wsdf based on the Wireshark development files you have installed locally.
So to get 4.4 support, I'd recommend looking into possibly getting the wsdf crate to generate the files from your install, the thing I failed at, and then it either works automatically when you write your wsdf-based code or only minor tweaks and not a large-esque PR like this is necessary.

Thanks for the context. I've pulled some of your changes into my branch as well as a base. I'll see if I can make a difference :)

Good luck!
On October 24, 2024, Espyo @.***> wrote:

Thanks for the context. I've pulled some of your changes into my
branch as well as a base to work off. I'll see if I can make a
difference :)

—
Reply to this email directly, view it on GitHub
https://github.com/ghpr-asia/wsdf/pull/19#issuecomment-2435956542, or
unsubscribe <https://github.com/notifications/unsubscribe-
auth/AHLXZJLLHRNTKAZ7IDOVJL3Z5EWSZAVCNFSM6AAAAABN5DGEUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZVHE2TMNJUGI>.
You are receiving this because you were mentioned.Message ID: <ghpr-
@.***>


Do not deploy the registry-configuration-cleaner on Shoot deletion

How to categorize this PR?

/kind enhancement
What this PR does / why we need it:
If the Shoot is in deletion, then there is no need to clean up the registry configuration from Nodes. The Shoot deletion flows ensures that the Worker is deleted before the Extension deletion.
https://github.com/gardener/gardener/blob/6f8ce7692f793e583734f533cfdee7b9d93fb815/pkg/gardenlet/controller/shoot/shoot/reconciler_delete.go#L436-L440
The deleteExtensionResourcesBeforeKubeAPIServer step depends on waitUntilManagedResourcesDeleted which depends on deleteManagedResources which depends on waitUntilWorkerDeleted.
Hence, there are no Nodes, no need to clean up registry configuration.
Which issue(s) this PR fixes:
N/A
Special notes for your reviewer:
N/A
Release note:

The `registry-configuration-cleaner` is no longer deployed on Shoot deletion with registry-cache extension enabled. The Extension deletion occurs after the Worker deletion. There are no Nodes, hence there is no need to clean up registry configuration.

/test pull-gardener-extension-registry-cache-e2e-kind

Locally, the e2e tests succeed.

/test pull-gardener-extension-registry-cache-e2e-kind

/approve


[REUPLOAD] Privileged Helper Tool doesn't installs.

🪲 Description
What's happening is that when I try to download ISOs of old macOS versions (exemple: Mountain Lion) I get the trigger message of Privileged Helper Tool isn't install. I try to install it, input my password and all, the trigger disappears and it doesn't gets installed. It does the same with every installers, even the most recent one.
📋 Steps to Reproduce
Open Mist.app
Click on any installers
Try to install Privileged Helper Tool
✅ Expected Behaviour
It should install correctly but doesn't
💻 Environment
Mist version (App menu ⇨ About Mist): 0.20
macOS Version (sw_vers): 24A5289h
Hardware (system_profiler SPHardwareDataType):
Model Identifier: MacBookPro15,1
Chip : 6-Core Intel Core i7
ℹ Additional context
Is this also a bug in mist-cli? I don't know how to use the cli version 😭
Provide links to GitHub Issues: https://github.com/ninxsoft/Mist/issues/69
Links to 3rd-party tools / references / documentation: I didn't use any to be honest
I don't have any error messages, it just does then nothing, doesn't works.

Re-works now I only had to reinstall it


Missing titles and styles

Hi,
Thanks for this package - makes it quite easy to get started :)
Got a few problems though.

Fresh laravel installation.
Followed the install guide step-by-step: https://www.proxeuse.com/en/2020/use-laravel-jetstream-with-bootstrap/

I'm not able to see the "change password" titles and the 2FA button is also looking weird.
Rest seems fine.
What can be the reason for this?

Hi @kgp43
Thanks for pointing out those mistakes!
I've released a new version (v1.1.1) which fixes those issues. You should be able to install those updates by running composer update and php artisan fortify-ui:tabler-update. Please note that running the second command will override certain files previously installed by the installer.
Please let me know if you have any further questions!

Hi Roel,
That was fast and worked perfectly!
Thanks a lot for the detailed update instructions.
Love this package :)

Hi @RoelReijn
I found one more minor bug (very small).
It seems there is missing a closing div on this part within the home view:

                         Hi there! Thank you for using the ....

This:
 does not have a closing div.
If I add a closing div, then the footer is moved down a bit.
It works fine as it is, but my editor (VS Code) tells me the page is missing a closing div (home.blade.php)

Hi @kgp43

Hi Roel,
That was fast and worked perfectly!
Thanks a lot for the detailed update instructions.
Love this package :)

Thanks a lot!

Hi RoelReijn
I found one more minor bug (very small).
It seems there is missing a closing div on this part within the home view:

 Hi there! Thank you for using the ....

This:
 does not have a closing div.
If I add a closing div, then the footer is moved down a bit.
It works fine as it is, but my editor (VS Code) tells me the page is missing a closing div (home.blade.php)

Thanks for noticing. I'm about to release a new version which includes a bug fix. Please note that the update command has changed to php artisan tabler:update --type=full as you will be able to see in the README.md file.
I will close this issue now but please open a new issue if you find other bugs or if you have ideas for a new feature.


fix(type): z.objectUtil is not exist

Summary
closes https://github.com/web-infra-dev/rspack/issues/8277

Checklist

[ ] Tests updated (or not required).
[ ] Documentation updated (or not required).

😄 zod increses 3000 lines of core.api.md...

Alternate solution:

#8306


Rename Field's 'name' prop

When using some controls with redux-form, ie. DateTimePicker. I have noticed that any control using "name" as a prop interferes with the Field element wrapper so Redux-form never finds the control and always gives an error, "Encountered a form control node without a name attribute."
Is possible to rename this required name attribute to be more unique( reduxformName, etc) for the Field element? That way controls with an existing "name" prop would not interfere with Redux-form.

Create a wrapper around your component, and rename the prop internally
Use the documented props prop

I am learning redux-forms and tried every variation using wrappers. Can you give me an example of using DateTimePicker with the Field wrapper. I cannot get this to work in redux-forms.

export const RenderCalanderFormControl = (props) => {
return (
    <div>
        <DateTimePicker  className={props.className} time={false} placeholder={props.placeholder}
                                 defaultValue={props.defaultValue} format={"MMM dd, yyyy"}
                                 />
    </div>    );

};

Got it figured out. Thanks for you help


The connection between envoys is slow.

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
<EMAIL_ADDRESS>where the issue will be triaged appropriately.
Title:  The connection between envoys is slow
Description:
The connection time between envoys is unstable. In most time, it use 1,2 ms.But somtime it will rise to hundreds ms.
It only happen in both pod injected envoy sidecar.How can I check where is real question on?
full.log

This is a very broad topic. There could be many reasons. Probably you could start with collecting more data on e.g. what process makes syscalls that take too long (perf trace --duration 100).

thx lot!!After eliminating a large number of possible causes，we found is physic network interface trouble.Now envoy is fast again


Add more Column comparison functions

Currently the many of the existing Column handy functions/operators are missing in the Kotlin API, this issue is to fulfill the gap. Here is the list of functions which can be added to the API
Infix textual functions:

infix fun Column.neq(c: Column): Column
infix fun Column.geq(c: Column): Column
infix fun Column.leq(c: Column): Column
infix fun Column.le(c: Column): Column
infix fun Column.ge(c: Column): Column
Also need to add infix keyword to the existing fun Column.eq(c: Column): Column function

Infix char-based functions:

infix fun Column.`>=`(c: Column): Column
infix fun Column.`<=`(c: Column): Column
infix fun Column.`>`(c: Column): Column
infix fun Column.`<`(c: Column): Column
infix fun Column.`!=`(c: Column): Column
infix fun Column.`||`(c: Column): Column

Also some of the functions can be implemented via Kotlin operator functions such as compareTo/multiply/divide/plus/minus, but this needs to be double-checked

Can I get this one?


Link to source doesn't open browser

On Android 5.1.1 link to source doesn't open a browser. it does on your website.

I guess the error is related to the app compilation of Cordova. I'll take a look into it.
Thank you for the hint! :-)

Should work now. Tested on Android emulator.


add inspect trait for asset roles

It adds a new fungibles trait to inspect roles for assetIds. In particular it adds getters for owner, issuer, freezer, admin

bot merge

bot merge


Exit 1 if conclusion is failure and -fail-on-error for github-check reporter

For #446 and #403

Hi @haya14busa,
I'm currently trying to set up a git hook with -fail-on-error flag for my local environment.
For platforms like Github it's working fine by having this command:
reviewdog -reporter=github-check -fail-on-error -filter-mode=nofilter
However, there seems no option to execute the same on fail-on-error for the local reporter.
Can this be added to reviewdog? Or is there some other way to reach the same result.
I already tried to add like how it's done on this git-hook but I cannot get this working

It should work for local reported as well.

I realized after reading the date of this pull request that my version was not up to date.
Sorry for the inconvenience.


Feature request: support passing zero-copy cuda buffers to IREE

Request description
See https://discord.com/channels/689900678990135345/804403964463546378/1053046022626091130 for background
We'd like to be able to initialize a tensor on a cuda device and pass it to IREE, rather than having to initialize on a host device and force IREE to do a copy.  Ideally this would provide a Python API too.
What component(s) does this issue relate to?
Runtime
Additional context
No response

These two methods need to be implemented by someone: https://github.com/google/iree/blob/main/runtime/src/iree/hal/drivers/cuda/cuda_allocator.c#L300-L318
cuImportExternalMemory can be used for some of the types, and then we can have cuMemPoolImportPointer / cuMemPoolExportPointer for exchanging memory with other processes, etc.

These two methods need to be implemented by someone: https://github.com/google/iree/blob/main/runtime/src/iree/hal/drivers/cuda/cuda_allocator.c#L300-L318 cuImportExternalMemory can be used for some of the types, and then we can have cuMemPoolImportPointer / cuMemPoolExportPointer for exchanging memory with other processes, etc.

Is there an analogous existing flow for any other backend that's also plumbed through to Python? I wouldn't mind tinkering with it to see if I can hack something together.

There are HAL methods for iree_hal_allocator_import_buffer and iree_hal_allocator_export_buffer so each driver can implement their own import/export semantics. They all use the iree_hal_external_buffer_t type which matches what Vulkan/D3D12/CUDA all support for external handles but it could also use some more enums - like IREE_HAL_EXTERNAL_BUFFER_TYPE_DEVICE_ALLOCATION that takes a void* device ptr (or VkBuffer etc) to go alongside the IREE_HAL_EXTERNAL_BUFFER_TYPE_HOST_ALLOCATION - this would let you pass a CUDA device pointer right in/get one out. We'll also be able to extend it over time for things like AHardwareBuffer on Android or MTLBuffer on iOS, etc.
Import is relatively straightforward but export will require a bit more goo as it has implications on the original allocation - to export a buffer it needs to be a dedicated allocation (instead of a suballocation) and unaliased and we may need some additional allocation flags to support that. If all you need is import to start that makes things easier, especially if ownership is retained with the caller (just letting IREE access the buffer but not trying to make IREE manage it).
I'm not sure if they're exposed to python but it should be pretty straightforward. The import call does take a callback that would need to be passed when ownership is transferred but that's just some pybind goo and we may use for other things already that shows how to deal with lifetime management.

I'm interested in extending the python API in this way but it is friendly frighteningly easy to get wrong and my rule for that is to wait until there is demand...

I think we have reached a point in the TorchDynamo integration that having this be zero-copy is necessary for IREE to go beyond "toy" status. So I would consider there to be "demand" here, unless we want to deprioritize IREE running under TorchDynamo for the time being.

Just so that I understand what is being asked: you have device memory initialized and you need to create a DeviceArray from that (via import)? Exposing the API is relatively easy, but it would be really good to have a reduced test case and/or sample demonstrating these more advanced things without requiring a specific use (like torch).
Also just checking: you don't need to do any layout transformation?

Any ideas on the test case are welcome... I'm happy to expose test only APIs for allocating raw device memory to do this if needed.

Basically we have torch.Tensor's that have pointers to CUDA memory and need to compute on them with IREE and then get results back as CUDA buffers. On the input side it looks like:
>>> import torch
>>> t = torch.tensor([1,2,3]).cuda()
>>> t.data_ptr()
140402397020672

(probably doing this with CPU first would be an adequate scale model)
(if you need a quick colab to test this I found the instructions here for GPU colab useful: https://www.geeksforgeeks.org/how-to-run-cuda-c-c-on-jupyter-notebook-in-google-colaboratory/)
I'm not sure the output story, but it seems less pressing. Googling around, some prior art in the ecosystem works exclusively with out params, which might be enough for us (TorchDynamo can always generate statically shaped code). I still need to look more into the buffer donation problem and see how we want to handle this, or whether truly escaping dynamic-sized cuda allocataions are necessary for IREE.
Especially for training loops we aren't going to want the truly escaping allocations anyway, so ignoring that case for a first step is probably fine.

Thanks. Beyond that simple example, what are the chances that the real case has arbitrary strided layouts?

Ok, and doing this for real for output buffers will require that we finally move past the basic calling convention where all outputs are returned imperatively. We have a different compilation mode with full async and signatures needed for these more advanced cases, but exercising it fell victim to holiday vacations. We need that working for pjrt integration anyway so need to get on it. I assume the needs are similar modulo python API exposure so fine pushing on this more -- it'd be good to have multiple users to get the apis and features right.
This will require some design work and that won't start until Monday, earliest. Then we'll have a better view of what we're looking at.

Thanks. Beyond that simple example, what are the chances that the real case has arbitrary strided layouts?

I haven't actually tested this, but I believe the chances of arbitrary strided layouts are low. Or even if there are, it will be for inputs (not huge weights [1]) and the captured graph will be large enough to amortize making them contiguous.
[1] In stock PyTorch the weights are the raw inputs to cudnn operators, so there is already a force keeping them away from being in random strided layouts.

@silvasean @stellaraccident Curious if this is still active? Sounds like a P2 if it is.


chart.exportImg() is throwing blurry image

i am trying to export the full org chart in expanded mode,
i have tried with 50 records its working fine, now as things are getting complexed and with 250 records its printing blurry images
is there any way to achieve the clear pdf for the records

is am using the pdf option sent by dufault just need the highest image quality
can you guide me

Adding scale:5 in place of green line should do the trick

Hi, i have the same problem with my export. I have changed the scale to 6 but I still have blur and a big file (150MB)!
Do you have any tip how can I export smaller file?

@grzegorzCieslik95  scale controls how bit the file is. scale:1 exports exactly the same size as on the screen, scale 6 - makes it 36x bigger image.
Can you post a 3x scale photo of the exported org chart?

Sure, i post scale 3 and 6

I can see the quality difference.
The issue with blurriness is that you are trying to export a horizontally spread chart on a vertical screen :/
Not sure how to approach this - I think I have to apply some kind of cropping to make it look better. I will keep this issue open, not sure when it will be fixed

Hi, I have sth like this https://stackblitz.com/edit/js-7qzbix?file=index.html .
I'm still having problem with rendering images but it works very well in big organisation.

Hello, i paste here my code for export pdf without blur for ~300 nodes
      const { svg } = chart.current.getChartState();       const graphNode = svg.node();       if (!graphNode) {         return null;       }       const nodeWithInlineStyles = computeToInlineStyles(graphNode); // replace style to inline style       await transformImagesToBase64(nodeWithInlineStyles); // replace image URL to base64       const { y: oY, x: oX } = graphNode.getBoundingClientRect();       const { height, width, x, y } =         graphNode.children[0].getBoundingClientRect();       const opt = {         margin: [4, 8],         filename: fileName,         image: { type: 'jpeg', quality: 0.98 },         html2canvas: {           useCORS: true,           allowTaint: true,           letterRendering: true,           scale: getCorrectScale(             graphNode.getElementsByClassName(nodeClassName).length, // get correct scale for nodes count           ),            height,           width,           x: x - oX,           y: y - oY,         },         jsPDF: {           unit: 'px',           format: [height + 12, width + 16],           orientation: height > width ? 'p' : 'l',         },       };       return html2pdf()         .set(opt)         .from(nodeWithInlineStyles)         .save()

Hi,
I tried several variations, for A0 format the scale factor 4.6 is the highest I can choose (or nothing actually happens) now the file is quit large (130MB) but still blurry...Any idea why? Thank you!
  function downloadPdf(chart) {
    chart.exportImg({
      save: false,
      full: true,
      scale: 4.6,
      onLoad: (base64) => {
        var pdf = new jspdf.jsPDF('landscape','mm','a0');
        var img = new Image();
        img.src = base64;
        img.onload = function () {
          var widthInInches = img.width;
          var heightInInches = img.height;

          var aspectRatio = img.height / img.width;

          var pdfWidth = '1189';
          var pdfHeight = pdfWidth * aspectRatio;

          pdf.addImage(
            img,
            'JPEG',
            0,
            0,
            pdfWidth,
            pdfHeight
          );
          pdf.save('chart.pdf');
        };
      },
    });
  }
</script>

dbdaniellozynski Did you tried my method to generate pdf? I dont use native export from library.

@grzegorzCieslik95 Hi, I am not really a JS guy, I simply don't unterstand what to do with your code :-(

Oki, so i dont know how to help you with this. I had the same problem with this library native export

Ok, I tried to understand your code and came up with this (absolutely no idea if that should work that way, but at least it creates an very blurry PDF ;-)
Is that sort of the way you intended to use the code? Do you know why it's blurry?
`

imho you need to add more scale -> I use values between 5 and 50 in my case

hi, @bumbeishvili
is there any update on this thread,
i have a tree with height 7; total of ~400 nodes
the output for my pdf is same as grzegorzCieslik95 posted in this thread [images and text are blurry ]
i tried generating an svg and using that svg inside the pdf also played with svg2pdf lib  -> that didnot worked
is there any idea  to tackle this issue

@vatraiadarsh  no update on my end


Swith to compatible xrd client for old dcache version at FNAL

How urgent is this?

there will be an update in all the FNAL workernodes, we need this patch to be in place for new workflows before that happens. Maybe Alan need to fix the workflow sandboxes for some workflows already created in the wmagents too.

Fixing sandboxes sounds like a crazy idea to me. I'm pretty sure @amaltaro  has better things to do with his time.
If FNAL can't figure out a way around this, then we can drain the site and start over.

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/152/ for details

No pylint warnings for pull request 6085:

Unit test changes for pull request 6085:

WMCore_t.WebTools_t.REST_t.RESTTest:testSanitiseAssertFail changed from error to success
WMCore_t.WebTools_t.NestedModel_t.NestedModelTest:testOuterFooError changed from error to success
WMCore_t.WebTools_t.REST_t.RESTTest:testSanitiseFailHTTP changed from error to success
WMCore_t.WebTools_t.REST_t.RESTTest:testSanitisePass changed from error to success
WMCore_t.WebTools_t.REST_t.RESTTest:testBadPing changed from error to success
WMCore_t.WebTools_t.REST_t.RESTTest:testPing changed from error to success

I guess that's the best option for us. We've updated the sandboxes in the past (very dangerous but possible :))


Rows Reordering Scenario has an issue

Hi
It appears your demo is working but actually there are JS issues, I found this when it went to resave my reordered data back to my database, the actual data hadn't been reordered using the indexes..
Uncaught TypeError: Cannot read property '1' of null
Thanks

Hi,
Thank you for the report. Could you provide more detail on how to reproduce the issue?

Hi,
Simply drag to reorder the list and it throws the error.
Thanks
On 24 August 2016 at 00:20, Artem Tabalin<EMAIL_ADDRESS>wrote:

Hi,
Thank you for the report. Could you provide more detail on how to
reproduce the issue?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tabalinas/jsgrid/issues/407#issuecomment-241910829,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJmXnj48Jne2TRGwUFyDKEF8LH9XyeXQks5qi4BJgaJpZM4Jq0n0
.


[c-lightnig] Implement basic backup mechanism

Seems like a good/clean process here: https://github.com/dani-garcia/vaultwarden/wiki/Backing-up-your-vault#sqlite-database-files
The idea would be to mount a sidecar container to c-lighting and run something like:
sqlite3 /root/.lightning/bitcoin/lightning.sqlite3 ".backup '/path/to/backups/db-$(date '+%Y%m%d-%H%M').sqlite3'"
Using a cron job, maybe once per hour and do some kind of rotation to keep the past 2 or 3 days.

Maybe add a timeout like
sqlite3 /root/.lightning/bitcoin/lightning.sqlite3 ".timeout 1000" ".backup '/path/to/backups/db-$(date '+%Y%m%d-%H%M').sqlite3'"


Mobile Activity Page - Risk factors need more padding to align all text

You are in the 50+ age bracket
You have underlying chronic medical conditions
Your family member feels sick

are not aligned with the other factors. Add padding to fix.
┆Issue is synchronized with this Trello card by Unito

@alexastef @botanical I would like to work on this!

@alexastef @botanical I would like to work on this!

Awesome, go for it 😄 🚀

@botanical Looks like this issue is up for grabs again? Can you assign this to me?

@vividh assigned! Apologies for the delay


Fix game embeds when spelltable has error

Description
Fixes an issue where the bot would create a wrong looking embed when SpellTable has a backend error.
Before

After

Checklist

[x] Add tests for these changes
[ ] Test coverage is not decreased
[ ] Entire test suite passes

Codecov Report

Merging #273 (7d55bf9) into master (ffa625f) will decrease coverage by 0.32%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #273      +/-   ##
==========================================
- Coverage   94.09%   93.77%   -0.33%
==========================================
  Files          43       43
  Lines        2372     2362      -10
  Branches      324      325       +1
==========================================
- Hits         2232     2215      -17
- Misses        108      113       +5
- Partials       32       34       +2

Impacted Files
Coverage Δ

src/spellbot/data.py
97.31% <100.00%> (-0.74%)
:arrow_down:

src/spellbot/__init__.py
90.93% <0.00%> (-0.39%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ffa625f...7d55bf9. Read the comment docs.


fix: Adjust collector profile verification icon alignment

The type of this PR is: Bugfix

This PR resolves CX-2379
Description
Adjust collector profile verification icon alignment.
PR Checklist (tick all before merging)

[ ] I have included screenshots or videos to illustrate my changes, or I have not changed anything that impacts the UI.
[ ] I have tested my changes on iOS and Android.
[ ] I have added tests/stories for my changes, or my changes don't require testing/stories, or I have included a link to a separate Jira ticket covering the tests.
[ ] I have added a feature flag, or my changes don't require a feature flag. (How do I add one?)
[ ] I have documented any follow-up work that this PR will require, or it does not require any.
[ ] I have added an app state migration, or my changes do not require one. (What are migrations?)
[ ] I have added a changelog entry below or my changes do not require one.

To the reviewers 👀

[ ] I would like at least one of the reviewers to run this PR on the simulator or device.

Changelog updates
Changelog updates

Cross-platform user-facing changes

iOS user-facing changes

Android user-facing changes

Dev changes

Warnings

:warning:

❌ An error occurred while validating your changelog, please make sure you provided a valid changelog.

  Generated by :no_entry_sign: dangerJS against 1f8ae5e3d2301e9ade0cae2ddf40c8877f75b38d


WOFF font files are not served with a correct MIME type (application/font-woff)

As WOFF fonts are (slightly) more widely supported by browsers then TTF fonts (see http://caniuse.com/#feat=woff versus http://caniuse.com/#feat=ttf) I wanted to use them in my app, but it turns out, elm-reactor won't serve them with the correct MIME type.
According to https://www.w3.org/TR/WOFF/#appendix-b the MIME type should be application/font-woff

@martincerny does this fail for WOFF2 as well?

I have not tested, but very likely not - the list of MIME types in https://github.com/elm-lang/elm-reactor/blob/master/src/backend/Main.hs (search for "mimeTypeDict") does not contain woff2.

I have a similar issue, not sure if I should have better opened a separate one.
It's also a problem with the mime type.
I'm working on a parser for wavefront .obj files. (here)
Elm reactor serves my .obj files as text/html instead of text/plain.
It also adds some additional unwanted syntax highlighting code to it: This is the output from elm reactor:
<!DOCTYPE HTML>
<meta charset="UTF-8"><title>~/suzanne.obj</title><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.3.0/styles/default.min.css">
# Blender3D v249 OBJ File: suzanne.blend
# www.blender3d.org
v 0.437500 0.164063 0.765625
v -0.437500 0.164063 0.765625
v 0.500000 0.093750 0.687500
...

file tells me it's text/plain:
↪ file --mime-type suzanne.obj
suzanne.obj: text/plain

As a workaround I'm currently using python -m http.server.
But I miss elm reactor :cry:


Remove deprecated ResultTree API

cc @kortschak and #208

All tests pass, FWIW - not sure if there's anything else involved in removing this.

Damn Travis. Tested locally here too. Motion granted. LGTM.


Update identity for 1.0.2

Shouldn't the README be updated too ?

Yes let me update it manually. Removing --sr while running the script changed readmes everywhere


test-branch/execute-job/fork-branch/2a6104a7-5552-4242-ba13-a3d661773993

PR for testing ba0cf4d92f917ac4ce2bba87f0809048f8d0844c

padding /canonical/self-hosted-runners/run-workflows ba0cf4d92f917ac4ce2bba87f0809048f8d0844c padding


Stop ShapeCast3D drawing when debug collisions off

ShapeCast3D is debug drawing when it shouldn't be (when Visible Collision Shapes/is_debugging_collisions_hint() is off).
It turns itself on when the Shape or Target Position is set, this PR fixes that.

Mesh LOD generation only kicks in on imported 3D scenes, not imported 3D meshes. If you change the OBJ's import type to OBJ As Scene, LOD generation should work.

Thanks!


An ArgumentError occurred in media#show

Can reproduce this as a non-Admin user only. These resource requests are generated by the Library page.
@YvonneP noticed this.
Production version v0.18.0-0.
An ArgumentError occurred in media#show:

  No projects provided.
  lib/modules/access/core.rb:147:in `validate_projects'

-------------------------------
Request:
-------------------------------

  * URL        : https://www.ecosounds.org/audio_recordings/213461/media.json?audioEventId=13380&endOffset=655.724&startOffset=650
  * HTTP Method: GET
  * IP address : [CENSORED]
  * Parameters : {"audioEventId"=>"13380", "endOffset"=>"655.724", "startOffset"=>"650", "format"=>"json", "controller"=>"media", "action"=>"show", "audio_recording_id"=>"213461"}
  * Timestamp  : 2016-02-17 15:02:39 +1000
  * Server : web-01
  * Rails root : [CENSORED]/production/releases/20160216074841Z

-------------------------------
Backtrace:
-------------------------------

  lib/modules/access/core.rb:147:in `validate_projects'
  lib/modules/access/level.rb:13:in `projects'
  lib/modules/access/check.rb:69:in `can_any?'
  app/models/ability.rb:167:in `block in to_audio_recording'
  app/controllers/application_controller.rb:122:in `auth_custom_audio_recording'
  app/controllers/media_controller.rb:67:in `authorise_custom'
  app/controllers/media_controller.rb:15:in `show'
  app/controllers/application_controller.rb:487:in `set_then_reset_user_stamper'

Problem is obvious - this site is not in a project. It is an orphan. Just put it into a project using https://www.ecosounds.org/projects/<project_id>/edit_sites and all should be fine.

Fair point will do. I can't work out what this site is, so instead I'm going to un Is_reference the 98 audio_events in that site.
Should we code for this?
OrphanedSiteException maybe?
However, this is an error with the database that only occurred because of data import.

Adding a special Error for this is fine. Even though site should not be orphaned, it is quite possible.
Error will be added in custom_errors.rb, probably as a subclass of StandardError.

:+1:


Solution Checker role?

Shouldn't this page also explain Solution Checker, Knowledge Manager, etc?  (in other words I see more roles in my selection box than are explained in this article).

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3b4508e9-6043-3bad-080a-956f5febb92b
Version Independent ID: 726691e0-81c1-f3d3-d8f2-6eddecb21753
Content: Configure environment security - Power Platform Admin center
Content Source: power-platform/admin/database-security.md
Service: power-platform
GitHub Login: @manasmams
Microsoft Alias: manasma

@lkuhn65
Thanks for the suggestion. We'll consider for future documentation updates.


Go's syscall package FreeBSD is missing S_IRWXG and S_IRWXO

This patch adds a _freebsd.go file that builds instead of the regular
one only on Freebsd (to the best of my cross-platform Go abilities).
It imports the C constants S_IRWXG and S_IRWXO and uses them instead
of syscall.S_IRWXG and syscall.S_IRWXO.
Closes #2078
The relevant diff is:
--- pkg/secrets/check_right.go	2018-07-29 16:56:36.000000000 +0200
+++ pkg/secrets/check_right_freebsd.go	2018-07-29 16:56:36.000000000 +0200
@@ -3,10 +3,17 @@
 // This product includes software developed at Datadog (https://www.datadoghq.com/).
 // Copyright 2018 Datadog, Inc.

-// +build !windows,!freebsd
+// +build freebsd

 package secrets

+// temporary fix until go gets S_IRWXG and S_IRWXO
+
+/*
+#include <sys/stat.h>
+*/
+import "C"
+
 import (
 	"fmt"
 	"os/user"
@@ -20,7 +27,7 @@
 	}

 	// checking that group and others don't have any rights
-	if stat.Mode&(syscall.S_IRWXG|syscall.S_IRWXO) != 0 {
+	if stat.Mode&(C.S_IRWXG|C.S_IRWXO) != 0 {
 		return fmt.Errorf("invalid executable '%s', 'groups' or 'others' have rights on it", path)
 	}

ping cc @masci <3

@olivielpeau thanks for the detailed and thoughtful reply. I didn’t realise secrets was something non-essential to be enabled at build time. That’s definitely good enough for me for now.
That said, an alternative patch could be a bit more “risky”. These constants are ancient Unix 25+year old constants and likely to have the same bitmask values everywhere. Given that this patch introduces a FreeBSD-only compilation path, you could be open to accept using the raw bitmask values in the two places needed. The FreeBSD-only complication path could be removed once you require a go version that this is fixed in.
If you’re interested, I can amend the patch.

@janl Using the raw bitmask values sounds good to me, if you could amend your patch this way that'd be great, thanks 👍. Could you also leave a FIXME comment as a reminder to update it once your patch on golang is shipped in go stable?

@janl  Thanks again for your contributions here. Were you still interested in working through the changes olivielpeau suggested?

@janl Thanks again for your contributions here. It sounds like you might have gotten busy with other projects. Should we close this PR and revisit at a future date?

Thanks again @janl for your contributions here. Any chance you will be able to work on the changes suggested by @olivielpeau or shall we close for now and revisit later?


C# 6: Use exception filters

PR Summary
RCS1236
PR Context

PR Checklist

[ ] PR has a meaningful title

Use the present tense and imperative mood when describing your changes

[ ] Summarized changes
[ ] Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
[ ] This PR is ready to merge and is not Work in Progress.

If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.

Breaking changes

[ ] None
OR
[ ] Experimental feature(s) needed

[ ] Experimental feature name(s):

User-facing changes

[ ] Not Applicable
OR
[ ] Documentation needed

[ ] Issue filed:

Testing - New and feature

[ ] N/A or can only be tested interactively
OR
[ ] Make sure you've added a new test if existing tests do not effectively test the code changed

Tooling

[ ] I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.
OR
[ ] I have considered the user experience from a tooling perspective and enumerated concerns in the summary. This may include:

Impact on PowerShell Editor Services which is used in the PowerShell extension for VSCode (which runs in a different PS Host).
Impact on Completions (both in the console and in editors) - one of PowerShell's most powerful features.
Impact on PSScriptAnalyzer (which provides linting & formatting in the editor extensions).
Impact on EditorSyntax (which provides syntax highlighting with in VSCode, GitHub, and many other editors).

I don't see a value from the PR.
We can get more benefits from Roslyn code analyzers.

I don't see a value from the PR.
We can get more benefits from Roslyn code analyzers.

I presume your objection is this PR would be too much effort to review, not that there is no value in these type of changes?

I do not like such a code. It looks too hard. This is my personal perception.
Since most likely no one will review this code except me, I suggest skipping this PR further with other improvements.


Remove projectId from queries for embeddings and zero-shot recommendations on for downloaded models

Currently the requests for the embeddings and zero shot recommendation require the projectId on the downloaded models page. This can be solved by setting the projectId as optional parameter and in this case all models should be returned.

removing the project id from the query would remove functionality. Currently, we filter for e.g. the zero-shot recommendations depending on the project selection.
That said an optional parameter might still be useful since the model download page might not need the filter


Rails 4.1.4 "ArgumentError: Unknown key: :order"

ArgumentError: Unknown key: :order is happening on Comment.all and the commentable model's .comment_threads and .root_comments methods in Rails 4.1.4.
[1] pry(main)> Comment.all
ArgumentError: Unknown key: :order. Valid keys are: :class_name, :class, :foreign_key, :validate, :autosave, :table_name, :before_add, :after_add, :before_remove, :after_remove, :extend, :primary_key, :dependent, :as, :through, :source, :source_type, :inverse_of, :counter_cache, :join_table
from /Users/erik/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/activesupport-4.1.4/lib/active_support/core_ext/hash/keys.rb:71:in `block in assert_valid_keys'
[2] pry(main)> Post.last.comment_threads
  Post Load (0.1ms)  SELECT  "posts".* FROM "posts"   ORDER BY "posts"."id" DESC LIMIT 1
ArgumentError: Unknown key: :order. Valid keys are: :class_name, :class, :foreign_key, :validate, :autosave, :table_name, :before_add, :after_add, :before_remove, :after_remove, :extend, :primary_key, :dependent, :as, :through, :source, :source_type, :inverse_of, :counter_cache, :join_table
from /Users/erik/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/activesupport-4.1.4/lib/active_support/core_ext/hash/keys.rb:71:in `block in assert_valid_keys'

Edit: I'd really like to add this into Prelang as soon as possible.

gems/ruby-2.2.0/gems/activesupport-4.2.0/lib/active_support/core_ext/hash/keys.rb:75:in `block in assert_valid_keys': Unknown key: :order. Valid keys are: :class_name, :class, :foreign_key, :validate, :autosave, :table_name, :before_add, :after_add, :before_remove, :after_remove, :extend, :primary_key, :dependent, :as, :through, :source, :source_type, :inverse_of, :counter_cache, :join_table, :foreign_type (ArgumentError)

gem 'awesome_nested_set', '2.0.0' -  I used same way. Still  facing same issue.


Delete duplicate 'with'

Delete duplicate 'with'

/assign @shashidharatd

/ok-to-test

/assign @quinton-hoole

/assign @irfanurrehman

Rather than send a lot of small PR's like this making one-word grammar changes to comments, please run a grammar checker over the entire repo and send a single consolidated PR of grammar changes.  Also, this repo is deprecated, and no longer under development.

/close


Low ROC AUC Score, High Validation Accuracy for Binary Classification

I compute an AUC ROC score every epoch as follows:
class Histories(keras.callbacks.Callback):

def __init__(self, validation_generator = None):
		super(Histories, self).__init__()
		self.validation_generator = validation_generator

	def on_train_begin(self, logs={}):
		self.aucs = []
		self.losses = []

	def on_epoch_end(self, epoch, logs={}):
		self.losses.append(logs.get('loss'))
		valid_steps = np.ceil(self.validation_generator.samples/self.validation_generator.batch_size)
		true_classes = self.validation_generator.classes
		predictions = self.model.predict_generator(generator = self.validation_generator, steps = valid_steps)

		self.aucs.append(roc_auc_score(y_true = true_classes, y_score = np.ravel(predictions)))
		print(self.aucs)
		return

The ROC AUC score's end up fluctuating around 0.5, for example:
[0.50508310249307475, 0.4822398100514444, 0.49138405223585269, 0.49789176889592401, 0.5143925603482391, 0.49754946576968734, 0.50938959240205783, 0.50715967550455088]
For every model and data set combination I have tried running, including those which achieve high validation accuracy and low validation loss, and with balanced classes, I get these poor ROC AUC results. Without looking at more of my code, is there anything wrong with calculating an ROC AUC score as I have above? Training basically follows this tutorial: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html
Note: roc_auc_score function is imported from sklearn

AFAIK this is not related with Keras.

I have now tested passing in test data stored as a numpy array called test_data. Using the following, I get an ROC AUC score of 0.85:
predictions = model.predict(validation_data, verbose =1)
roc_auc_score = roc_auc_score(validation_labels, predictions)
print('roc_auc_score from numpy array', roc_auc_score)

With the same model, same data, but locating the data in a directory, and passing this path into flow_from_directory, I get an ROC AUC score of 0.54:
test_datagen = ImageDataGenerator()

test_generator = test_datagen.flow_from_directory(
	        path_validation,
	        target_size=(img_width, img_height),
	        batch_size=batch_size,
	        class_mode='binary')

valid_steps = np.ceil(test_generator.samples/test_generator.batch_size)
true_classes = validation_generator.classes
predictions = model.predict_generator(generator = test_generator, steps = valid_steps)
print('auc score from generator', auc)

Any idea where this discrepancy comes from?

@JJN123 I am basically seeing the same behavior. I calculate all training, validation, and test AUC after very epoch (using ImageDataGenerator) and it basically oscillates between 0.5 while the training accuracy increases. Have you figured out what is going on?

@haeseonii Yes I have figured out the issue. The default behavior of flow_from_directory is to shuffle the data; this means that the predictions from predict generator will not correspond to the ground truth labels which you get from test_generator.classes. So set shuffle = False.


util: extends uv_getrusage by majflt on NT and other NT counters

From the commit message

This extension for the win part of uv_getrusage() adds the
win equivalent of hard page faults to the available properties.
Since there are also similar counters in GetPerformanceInfo() that
are not as easily accessible in posix sytems, it is supposedly
beneficial to add them to  the overall API, while overcoming unices
shortcomings. Note: getrusage() implementation is very different
on unices also.
In consequence uv_rusage_t increaes in size.

Motivation
Pending PRs in node core will likely expose getrusage() to js. While doing research I was seeing that NT implementation lags behind values. Efforts to harmonize this API through the OSes will likely fail, since even the unices differ to much. In consequence I was plainly adding available properties to the API.
My global goal though is to increase performance observability in node in the io and in general in the C/C++ parts.
A Word of Warning
Since this is my first serious C contribution (C is hobby for me up until now), please take this with care. If this is nonsense in the idea as well as the implementation (especially casting shenanigans), I am happy to close this quick.
Also test were flaky in unrelated parts of the library. This also increased my insecurity.
Happy if this moves forward though :smile:

@saghul any thoughts on this. It would break ABI and introduce inconsistencies unless the Unix side were updated as well.

@cjihrig agreed. Since we named it after the Unix side, I'd rather fill in what Unix and Windows have in common than add a bunch of fields which only apply to a single platforms. And there is the ABI break.

OK. I'm going to go ahead and close this. Thanks for the PR though!

Okay. Then I'd extract the page fault param, since this would fill up windows with by this one parameter it shares with unix. If that is okay...
By adding fields for the single platform you mean having platform specific functions and structs?

Changes that add common fields are more likely to get in. If you break the ABI, it will have to wait until 2.0 though.

So additions to the struct are basically already breaking the ABI in those terms? Didn't know, that this is problematic. Then I'll go ahead with filling up. It's small but maybe worth it.
Just out of interest: adding functions to the API would also be problematic then? Sorry, again for being new to grown-up-C. Would have had some suggestions around having a normalized perf_counter API via perf / dtrace (in c) / win PDH

Adding functions is ok. (you could see some xxx2 functions)
Modify structs, function arguments (type or count). break ABI
I'm nobody but don't think about: uv_rusage2_t & uv_getrusage2, it's messy
I suggest to target 2.0

The general rule of thumb is that we only add stuff which is cross platform. This means that while adding fields to the struct is ok (it would break the ABI, so it would be part of the next major release) we should only do that if there is more than one platform which benefits from this.
Someone did mention that some of the currently zeroed fields on Windows could potentially be filled with perf counter information, that would be a welcome addition :-)

Okay. Thanks for sharing the info. I will send in a new PR for the existing metric(s) on windows and think about the rest in the context of versioning. Appreciate any bit of mentorship concerning C work in order to make meaningful contributions here. Cheers.

Thanks Robert! Feel free to send a PR and we'll take it from there.
On Apr 2, 2016 17:04, "Robert Jefe Lindstädt"<EMAIL_ADDRESS>wrote:

Okay. Thanks for sharing the info. I will send in a new PR for the
existing metric(s) on windows and think about the rest in the context of
versioning. Appreciate any bit of mentorship concerning C work in order to
make meaningful contributions here. Cheers.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/libuv/libuv/pull/787#issuecomment-204792765


Gold-mine Submission

As Andrea noted in the email there was no need to spend more than a couple of hours I have created basic path finding only looking a couple of spaces ahead.
Hoping to see my name on the leader board :)

Thanks @DraconianLore !


[ZT] Add note about file path in application check

This note tells users the file path for a device posture application
check MUST point to an actual executable. This comes from multiple
customers attempting to checking if /Application/AppName.app is running
on a machine, when that is just the folder name for that application.

Good callout! Added to #5567


Fix possible missing slashes

This PR makes sure that a trailing slash / always exists at the end of _session.Settings.BaseUri.AbsoluteUri when using absolute reference URIs to keep a valid URL.
Thanks @lordmampf :)

Thanks!


Change type metadata accessors to support incomplete metadata

This includes global generic and non-generic global access functions, protocol associated type access functions, swift_getGenericMetadata, and generic type completion functions.
The main part of this change is that the functions now need to take a MetadataRequest and return a MetadataResponse, which is capable of expressing that the request can fail.  The state of the returned
metadata is reported as an second, independent return value; this allows the caller to easily check the possibility of failure without having to mask it out from the returned metadata pointer, as well as allowing it to be easily ignored.
Also, change metadata access functions to use swiftcc to ensure that this return value is indeed returned in two separate registers.
Also, change protocol associated conformance access functions to use swiftcc.  This isn't really related, but for some reason it snuck in.  Since it's clearly the right thing to do, and since I really didn't want to retroactively tease that back out from all the rest of the test changes, I've left it in.
Also, change generic metadata access functions to either pass all the generic arguments directly or pass them all indirectly.  I don't know how we ended up with the hybrid approach.  I needed to change all the code-generation and calls here anyway in order to pass the request parameter, and I figured I might as well change the ABI to something sensible.

@swift-ci Please test and merge.

The hybrid approach you describe allows us to dynamically construct type metadata.

@slavapestov I think I wasn't very clear about what I've done there.
It used to be that all the metadata access functions took all their arguments directly, like foo(a,b,c) or foo(a,b,c,d,e,f,g,h).  This doesn't support calls from runtime code that's agnostic about argument count, at least not without major complexity.
The hybrid approach is that we're capping the number of direct arguments and passing the rest indirectly, so that we do foo(a,b,c) but foo(a,b,c,&{d,e,f,g,h}).  Except that, to make getting a complete buffer more efficient, it's actually foo(a,b,c,&{null,null,null,d,e,f,g,h}).
My change is that, when we cross the cap, we just pass everything indirectly.  So it's either foo(a,b,c) or foo(&{a,b,c,d,e,f,g,h}).

So it's still a hybrid approach in many senses, and it still supports reflective calls from the runtime.  It's just that, in any particular case, the data is not passed in a hybrid way: it's either all direct (if it fits under the cap) or all indirect.

@rjmccall That makes sense, thanks.

@swift-ci Please smoke test and merge.

@swift-ci Please test and merge.

@swift-ci Please test.

Build failed
Swift Test OS X Platform
Git Sha - 373cbc710d02f9c4bd09658cf9b08a9b3822c9e4

Build failed
Swift Test Linux Platform
Git Sha - 373cbc710d02f9c4bd09658cf9b08a9b3822c9e4

@swift-ci Please test.

@swift-ci Please test.


[Android] Fix error of missing class in the app package

Don't throw an exception when the object of new added class is created
in the library package but the app package lacks corresponding class
because of old version.
BUG=XWALK-5796
(cherry picked from commit 1a90ce31e27823bfba542204b3835b5653e5d28f)

@rakuco Please help to merge


There is a delay when pinging between containers from different hosts on initial ping.

In the first attempts to ping containers on different hosts, there is a delay before the containers are linked together. Typically losing the first 4 packets before the packets are sent back.
Any attempts to ping after the initial set up has no packet loss.
No real issue, but I thought we could investigate to see if the delay could be improved.

I've seen this too and it does seem excessively long.  It might be long enough for an application's connection to timeout and therefore fail to start.

Even with the new DNS changes, this delay still exists as of v0.23.0-rc1.

Yeah, the delay isn't DNS, it's somewhere in setting up the IPSec tunnel between the two points.  We will have to see if it can be improved, or preemptively exchange keys to setup the connection ahead of time.

Verified with v0.56.0 that this is eliminated with the new ipsec networking.


Merge execCmd and execFlags options: provide a common template

We need to provide a common template for execCmd:
java -jar ktlint -R diktat -F %%Test%% --debug

SO we won't need execFlags anymore.

Need to add implementation of com.saveourtool.save.core.utils.CmdExecutorBase for com.saveourtool.save.plugins.fix.FixPlugin
Originally posted by @nulls in https://github.com/saveourtool/save-cli/issues/435#issuecomment-1222064618


Add GPU execution support?

Would it be possible to add GPU execution support?
GPU's are well known for parallellism & working with the cuDF / RAPIDS ecosystem, is it possible to add GPU parallel processing capability in Rayon?

GPU parallelism is powerful, but not a good match for rayon's work-stealing model. I think you'd need something designed for homogenous workloads, and it probably needs compiler support to mix different target machines, if you want the host CPU to transparently "call" functions on the GPU.

Ok ... do you know of any crates currently working on GPU parallellism, or core-agnostic scheduling?
Also, does the Rust compiler support execution on GPUs?

do you know of any crates currently working on GPU parallellism,

There are a few crates that bind OpenCL or CUDA, but I haven't used them.

or core-agnostic scheduling?

Not that I know of.

Also, does the Rust compiler support execution on GPUs?

There's an unstable target for nvptx64-nvidia-cuda, but it doesn't seem like anyone is working on it.
Here's the tracking bug: https://github.com/rust-lang/rust/issues/38789

Related and probably of interest: https://github.com/EmbarkStudios/rust-gpu


How can I avoid Could not find node with given id (-32000)?

Despite the element has assured its existence by WaitReady, clicking sometimes results Could not find node with given id (-32000). Is there any code lacking?
chromedp.Run(ctx,
	chromedp.WaitReady(`a[href = '#foobar']`),
	chromedp.Click(`a[href = '#foobar']`),
)

I guess that's because the elements on the page change after chromedp.WaitReady. If that's the case, there is not way to avoid it.