Update Post “dva-c01-studying-kms”

Automatically generated by Netlify CMS

👷 Deploy Preview for dazzling-dieffenbachia-448a02 processing.

Name
Link

🔨 Latest commit
c8bcaf2aff2f0fd31f70a7556ecd286990beb619

🔍 Latest deploy log
https://app.netlify.com/sites/dazzling-dieffenbachia-448a02/deploys/62e181aa0b994c00080cde5b


947 URL.canParse is not available < chrome 120

https://github.com/breadboard-ai/breadboard/issues/947

Tested in Brave v1.60.118 (chromium 119) and looks good, thanks @Mearman

way ahead of you @dglazkov I've been looking at how to best achieve it with the current setup.


Reply to archived Technet articles should be possible

Is your feature request related to a problem? Please describe.
Often times, when I have a specific problem I'll find articles from other people struggling with the same thing on Technet. More often than not, those are archived, and there's no way to add the solution I eventually find to a place that already comes up to others looking for information on such problems with search engines.
Describe the solution you'd like
Some way to "reply" to archived articles / threads. I understand they aren't supposed to be "active", maybe something like:
Every archived thread can have ONE reference to an ACTIVE thread where its discussion is continued. If such an active thread already exists, answers can be added there. If not, it should be possible to attach a new thread to archived articles as a continuation.
Describe alternatives you've considered
Simply "not archiving" content certainly also would solve this particular issue. To a limited degree, a function to reach the original authors would also help (them), but not leave the solution for search engines to find.

@jmbreuer Technet has closed at https://docs.microsoft.com/en-us/teamblog/msdn-technet-english-forums-closure


Make SinglePoint::result async

Right now, SinglePoint::result is a sync call, which means it waits on the cluster to complete the request. Ordinarily this is fine but if a graph build is required this could block a request-processing thread for quite some time.
Unfortunately Spark does not offer an easy way to do async calls.

Well, this is less of an issue now that #122 is fixed, as we never wait for a graph to build.


Add custom ops for compatibility with PT Compile

This PR adds basic support for torch.compile() for the non-varlen variants of Flash Attention.
This essentially allows for models that use the flash_attn_qkvpacked_func, flash_attn_kvpacked_func, and flash_attn_func to compile without graph breaks.
I can add unit tests if it makes sense. I'll test it with our own training pipeline for performance measurements and I'll post them later.
This uses the new custom operators API in Pytorch 2.4. I can move to the older APIs if needed, or I can look up how to make both coexist

Added varlen functions too

Thank you! If this requires torch 2.4, is there some way to make it optional for older pytorch version? e.g. define torch.library.custom_op to be a no-op? Idk what people usually do.

There's a different API that was used in previous pytorch versions. It's more cumbersome to use, but let me think about how to use both APIs at the same time with a version selection. I can also do a noop wrapper based on version

Ok, I've updated the code to only wrap things if Pytorch is 2.4 or higher. Even though we're comparing strings, torch.__version__ implements semantic comparisons, so it should always work correctly. As soon as I have the training performance comparison I'll update the PR

aah, yikes
@ani300 I had started working on the same thing https://github.com/Dao-AILab/flash-attention/pull/1145 😓
Ill let you handle this 😃

@tridao I've updated the code after testing it with SFTTrainer to make sure everything was getting called correctly and compiling without graph breaks. I've also added a check for the pytorch version so this only activates if the APIs are available.
One question I have is if it's ok to change the C++ pytorch interface so the out/out_padded padding/unpadding can be done outside of the custom op. Right now, Pytorch custom ops don't allow outputs to be aliases of other outputs, which this pair is when head_dim % 8 == 0. This means I have to clone one of the outputs to unalias them, which is a waste of memory bandwidth.

Yes let's change the C++ code to not pad the output, and have the Python part handle that.

@GLivshits I dont think it can be handled in older versions of torch

@tridao @ani300 is there any progress/updates on this?
Its a pretty neat feature to have Flash Attention fully end-to-end traceable natively.

Hey, I'm almost done with the C++ portion, hopefully by Monday I can update the PR and rerun all the CI to make sure it's still good with all the changes. I'll also try the suggestions from @GLivshits to Mae it work with previous versions

@GLivshits the way the code is currently written, the backward functions need to have mutate_args, or we need to do an extra memory copy on GPU, which might significantly impact performance

@tridao this is ready for final review and/or merging. I tested with the whole unit test suite for both pytorch 2.3 and pytorch 2.4 on an A100

@tridao Any update on this?

Thanks @ani300 for taking this on. I started working on this here  (https://github.com/Dao-AILab/flash-attention/pull/1209). But this is way further along. So, I closed mine.
cc @zou3519 for custom ops API usage review @Chillee

Thanks @ani300
Custom ops changes look good to me. Hope @tridao  can take a look at the other parts.

Tests all ran successfully again

Awesome, thanks!

I just built the flash-attention main branch from source and sadly I'm getting the same error message during training with the Hugging Face Trainer with torch_compile=True:
.venv/lib/python3.12/site-packages/torch/_dynamo/variables/functions.py:663: UserWarning: Graph break due to unsupported builtin flash_attn_2_cuda.PyCapsule.fwd. This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind). If it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround. If it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use torch.compiler.allow_in_graph.

It looks like I somehow ran python setup.py bdist_wheel and forgot I was creating a wheel (which I also ended up mistakenly deleting moments ago) 😅 hopefully second time's the charm

I was worried for a second, as I just used it with the Huggingface Trainer yesterday successfully 😥

@ani300 I can confirm that it works! And much faster than PyTorch SDPA, shaved off 50 training hours (400h -> 350h) 😆


Replace current IoC code by Google Guice

Now that we have plugin support, Guice looks more interesting to plugin developers than our current crippled IoC container.

We're using Mycila too, which brings in support for JSR250 stuff (lifecycle management, etc).


Fix slow convergence behaviour in Rational#limitTo.

Fixes #585
This fixes a class of bugs in limitTo which lead to horrible performance in Rational#limitTo. Before, Rational(1).limitToInt would spin for a loooooong time before finally returning. Now it completes almost instantly.

Current coverage is 48.21% (diff: 91.66%)

Merging #586 into master will decrease coverage by 0.19%

@@             master       #586   diff @@
==========================================
  Files           181        181
  Lines         11466      11472     +6
  Methods        9791       9813    +22
  Messages          0          0
  Branches       1631       1615    -16
==========================================
- Hits           5550       5531    -19
- Misses         5916       5941    +25
  Partials          0          0

Powered by Codecov. Last update 7a8a16a...30f5b13

Wouldn't we prefer to use Spire.util.Opt in internal loops/methods, as they avoid allocating memory for Opt[_ <: NotNull]?
Apart from that, LGTM.

Ah, good point. I'll fix!

Feel free to merge afterwards!


[FEATURE] Display deployed version

Description
It would be useful to know what version is deployed, similar to our other apps
Screenshots
No response
Additional information
No response

@eddiejaoude where would you like version to be displayed? On map?

Some where on the top or bottom, similar to our LinkFree project https://linkfree.eddiehub.io

like this or i have to put in the header/footer?

That looks good 👍, we can always move it around later on if needed

uh, I have different prettier settings 😢 . What should I do?

we probably need to fix it so the project has settings, but for now, just change the minimum amount of lines


Different results in AFLGo and AFLFast papers

Recently I read your AFLFast and AFLGo papers.
I found some experiment results on binutils are different in these two papers.
These are the result clips from the papers.

For CVE-2016-4487, AFL found the bug in 2.63h, and AFLFast found it in 0.46h; however in AFLGo paper, AFL found the bug in 4m and AFLGo found in 2m.
(CVE-2016-4488, CVE-2016-4489, CVE-2016-4490, CVE-2016-4492 are similar)
If this is because that the initial seeds in AFLGo are closer to the bugs, then why in AFLFast can find the bugs faster than AFLGo in CVE-2016-4491 and CVE-2016-6131?
If the in each of the paper AFL and the extension tools are provided with a same set of initial seeds, then why the "factor" of AFLFast looks better than the "factor" of AFLGo (second last column)?
I wonder why the directed fuzzer find the bugs slower than the general purpose fuzzer?
Can you open source your initial seeds for binutils?

@mboehme I just noticed that in some other issues, some people are confused with the distance calculation on binutils.

Thanks for your interest in our greybox fuzzing research! Correct, there seems to be a disagreement between the results for AFL in the AFLFast and AFLGo papers. However, the sections on experimental setup in both papers should well explain this difference.
First, we are using two different versions of AFL (the most recent in each case). The version of AFL used in the AFLGo paper (FidgetyAFL) already incorporates the explore-schedule which makes recent versions of AFL substantially faster than earlier versions (i.e., before AFLFast).
Second, in the AFLFast paper, we executed both AFL and AFLFast with the deterministic stage (w/o "-d"). However, as Michal Zalewski pointed out in a discussion on AFLFast, he suggested future experiments be conducted with "-d". Hence, we opted to executed both AFL and AFLGo without the deterministic stage (w/ "-d").
Third, we made sure that there is no disadvantage in the comparison of AFLGo with AFL or AFLFast with AFL. In both papers, all fuzzers are started on the same seed corpus, using the same command line parameters, and given the same time budget. In the case of AFL, AFLFast, AFLGo on binutils, the "seed corpus" was the empty file:
mkdir in
echo "" > in/in

Fourth, in order to allow other researchers (including you) to reproduce our results, we made our tools publicly available. The experimental infrastructure is discussed in the paper. I hope this has answered your questions.
Since there is no issue with the tool per-se, I am closing this issue.

@mboehme I tried AFL 1.94b (I think it was the version AFLFast paper) with c++filt version before your fix. CVE-2016-4487/CVE-2016-4488 took a little long time to be found, but it reported all 46 unique crashes within 1 hour. I checked gdb+valgrind backtrace and some are relevant to CVE-2016-4492/CVE-2016-4493, CVE-2016-4490 and some other crashes. But I don't know how to use AFLGo to get the distance for binutils; can you open source the scripts? (I had some questions in #19 and hope you answer)
Also, I'm also interesting in that said by @karl-fuzznoob why AFLFast can be better than AFLGo sometimes?

@fuseproj Interesting observations. @mboehme Can you share the details?

Also, 4487/4488 and 4492/4493 are similar crashes, but in terms of time fuzzing results are different.


Added latam currency

Added latam currencies

Have you tested to make sure all of these work in all of the rates APIs integrated?

Just finalized moving around the exchange rates for the different currencies.

Thank you! And the rates API has all of these available?

@merc1er does this look okay to you?

Thank you @Jzarecta and @merc1er!


cigien: Watch samptelenergy.com

cigien requests the watch of the watch_keyword samptelenergy\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

samptelenergy\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

samptelenergy\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by tripleee in Charcoal HQ

Approved by tripleee in Charcoal HQ


Request 144

[ ] Testing input from #144

@Timh37 it seems like pangeo-forge-esgf is not finding urls for any of the requested datasets.
I am hopeful that https://github.com/jbusecke/pangeo-forge-esgf/issues/42 will help with this issue.

Oh this is looking very promising @Timh37 :


HDDS-8977. Ratis crash if a lot of directories deleted at once

What changes were proposed in this pull request?

change max limit for DirectoryDeletingService to 6000 (from 10000), considering, 4KB size (considering acls, key/file name, and other meata)  * 6000, resulting 24MB
If overall request size crosses the size limit, and excluding the latest request do not have any information to delete, then consider limit as 1000 for the task for deletion and retry with same.

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8977
How was this patch tested?

Testcase is added to verify the case of crossing limit size and retrying with 1000 limit for the task.

Thanks for working on this @sumitagrawl. I did not get a chance to thoroughly review the changes, but checking the size of the proto is a good idea. Looks like there should not be any performance issues with getSerializedSize() according to this old mail thread I dug up. Would it make sense to increase the Ratis buffer limit in the PR as well? After RATIS-1783 that change should be effective.
cc @kerneltime @szetszwo who also looked at this issue in the past.

Would it make sense to increase the Ratis buffer limit in the PR as well? After RATIS-1783 that change should be effective.

@errose28
IMO, increase ratis buffer size needs to be handled separately, as it involves,

performance impact verification
compatibility if rolling upgrade supported
involves changes for OM, SCM, DN as common way of handling.

Thanks @sumitagrawl for the patch, @errose28 for the review/comments.


In the readme suggested action, use checkout and upload-artifact v4.

v3 has been deprecated for a while, and the bump to v4 is painless (and has been done on the action itself already).

Thanks!


Error when rendering

Hello.
I come across an error when trying to render a single view point cloud using render_depth.py in ./render.
`../shapenet_v1/ShapeNetCore.v1/04530566/f1800fb3a9873af1e64f09105f9c8a56/model.obj
Traceback (most recent call last):
File "/snap/blender/20/2.79/scripts/addons/io_scene_obj/init.py", line 148, in execute
return import_obj.load(context, **keywords)
File "/snap/blender/20/2.79/scripts/addons/io_scene_obj/import_obj.py", line 1096, in load
idx = int(obj_vert[0]) - 1
ValueError: invalid literal for int() with base 10: b'8f'
location: /snap/blender/20/2.79/scripts/modules/bpy/ops.py:189
location: /snap/blender/20/2.79/scripts/modules/bpy/ops.py:189
Traceback (most recent call last):
File "/snap/blender/20/2.79/scripts/addons/io_scene_obj/init.py", line 148, in execute
return import_obj.load(context, **keywords)
File "/snap/blender/20/2.79/scripts/addons/io_scene_obj/import_obj.py", line 1096, in load
idx = int(obj_vert[0]) - 1
ValueError: invalid literal for int() with base 10: b'f'
location: /snap/blender/20/2.79/scripts/modules/bpy/ops.py:189
location: /snap/blender/20/2.79/scripts/modules/bpy/ops.py:189
Traceback (most recent call last):
File "/snap/blender/20/2.79/scripts/addons/io_scene_obj/init.py", line 148, in execute
return import_obj.load(context, **keywords)
File "/snap/blender/20/2.79/scripts/addons/io_scene_obj/import_obj.py", line 1096, in load
idx = int(obj_vert[0]) - 1
ValueError: invalid literal for int() with base 10: b'20.646'
location: /snap/blender/20/2.79/scripts/modules/bpy/ops.py:189
location: /snap/blender/20/2.79/scripts/modules/bpy/ops.py:189
`
I use blender 2.79, do I need to update or do something?
It renders successfully first 6 .obj, and fails from then on.
Best
Cheng

Looks like there is a problem with the obj file. There's a float value where an int is expected. You can skip these files if there are not too many of them.

Thanks! I'll try it.
I'm closing this issue.


[Bug]: Not working with Base64

Summary
My source is a Base64 image and i get this error when trying to load it:

Environment
"expo": "~51.0.31",
"react-native": "0.74.5",
"react-native-zoom-toolkit": "^3.1.0",
"react-native-gesture-handler": "~2.16.1",
"react-native-reanimated": "~3.10.1",

Steps to Reproduce
No response

I do not really get what's your issue, what's that you're doing in order for that error to pop up?

The image that im fetching from the API is not an image url but a Base64 string.
example:
"data:image/jpg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgICAgMCAgIDAwMDBAYEBAQEBAgGBgUGCQgKCgkICQkKDA8MCgsOCwkJDRENDg8QEBEQCgwSExIQEw8QEBD/wAALCAGQAtABAREA/8QAFwABAQEBAAAAAAAAAAAAAAAAAAcECf/EACQQAQACAQIEBwAAAAAAAAAAAAABAgUDBAYIERITFBghV6XT/9oACAEBAAA/AOVQAAAAAAAAAAAAAAAAAAAAAAAAAAAAALh6KOZv4z+5x/7o7ncJk+Gs3kOHM3tvLZHFbvW2W70e+t/D1tK80vXurM1npasx1iZientMsIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/2Q=="

I'm closing the issue as it's clearly not related to the library, I tried on both platforms and no of them proved to be related to your problem, by the looks of the of error logs it's a problem with the base64 image data being interpreted as a null value, see  a similar issue

@IlirBajrami My bad and sorry for the hurry, I just made several tests to render images as base64, passing null or undefined, same to the function that gets the resolution of an image, nothing proved to cause the issue you have, you could make some testing with RN Image's getSize and getSizeWithHeaders functions and see if they cause this problem to you, something like this:
import {Image} from 'react-native';

Image.getSize('base64 image', (w,h) => console.log(w, h), () => console.log('failed'));


[Accessibility] Add a command to focus Python Native REPL Input

Type: Feature Request
CC @amunger
ctrl+DownArrow (focus REPL input) works only when user stays in the interactive REPL editor group. There is no direct way to focus the REPL input area when keyboard-only users' cursor is outside the REPL.
Currently, they need to run "REPL: Focus Most Recent REPL Execution" and then press Ctrl+DownArrow.
Please add a command to focus the REPL input as well.
VS Code version: Code - Insiders 1.96.0-insider (5690e16478e0d893f5649b87e842faa82965b738, 2024-11-12T05:04:13.942Z)
OS version: Windows_NT x64 10.0.22631
Modes:

@jooyoungseo - do you have any thoughts on default keybindings to use?
perhaps:

Read last execution output - alt+home | alt+R
Focus last item executed - alt+home | alt+end
Focus REPL input - alt+home | alt+home

@amunger May I ask why we need "Read last execution output" command? Copilot and Notebooks don't have this command, either. What's not working in the Native REPL though is the Accessible View (alt+F2) in the input. If you press alt+F2 in Copilot Chat after you receive a response, you can open the received response in the Accessible View. This is not supported in the Native REPL, and I don't think we need to add the read the last executed command.

I think the following commands suffice:

Focus last item executed - alt+home, alt+home

Focus REPL input - alt+end, alt+end

sounds good, I'll just add those then


TsType of properties should be intelligent

Right now, if a property doesn't explicitly specify a type ex: ```const var: Type = ..., deno_docdoesn't populate thetsType` field for it.
Sometimes, type is not needed as it can be guessed by the IDE and the typescript compiler as well
const type = "it's a string";

in that scenario, it's known that the default type of that variable is string, thus tsType should use such.

This is really broad issue, and would be hard to tackle as a whole.
Actually your example works already:
export const a = "a";

Will get you a variable named a with a type of "a".  The tests for it are here: https://github.com/denoland/deno_doc/blob/main/src/tests.rs#L1317
Non-simple types are not currently supported.  I was going to open up some issues to infer types for specific structures.  There is already a request at #92 for object literals.
So my opinion is that this issue is to broad and unactionable.

As mentioned, I have opened up other issues that are more narrow with specific cases we can try to tackle.  I am going to close this in lieu of those other issues.


Fix --int-arg, --bool-arg, etc options parsing

Problem
Multi-argument options are non-intuitive (even our manual did not document it right: https://github.com/project-chip/connectedhomeip/blob/master/docs/testing/python.md#running-tests). Instead of that, use multi-options (which aligns with the manual).
Testing
CI will verify potential failures.

Please fix after SVE since the test_parameters config in TH is an arguments dict which means if --string-arg a:foo b:far shows as:
{
   "string-args": "a:foo b:bar"
}

Therefore:
{
   "string-arg": "a:foo",
   "string-arg": "b:bar"
}

would lead to only b:bar existing :(
if we split, then TH can no longer support any tests that has multiple options of same type :(. This is horrendous, but it's more distruptive to change it right now than to leave it.

This is correct. We can add the changes needed in TH to our backlog, but I defer to @raju-apple regarding the priority.

@tcarmelveilleux I've updated this PR little bit. Now it will support both option types: multi-argument and multi-option. Also, I've added a unit test to verify that both these behaviors work as expected. What do you think about such compromise?


Update README.md

This should do it.
Will close #129.
I think it will be worthwhile doing an immediate v4.1.2 for this, but replicating the v4.1.1 release description, adding the image fix. Set full changelog to v4.1.0..v4.1.2.

Draft v4.1.2 is prepped and ready to go on your review.

Holy heck. Fifty have already downloaded v4.1.1.

Some of those images would never have shown up in HACS - I used the existing links as the template for the replacement of the pasted images.

So we're good to go? Looks fine to me...

Yep


Questions: posting json objects & using angularjs

Hello Fredrik,
I'd like to be able to create a simple single page application using angular and D3 on a netduino 2 plus.
These libraries are somewhat large, so I was originally planning on pointing to CDNs.  However, it would be nice if they could be located on an SD card on the device.  If this is supported, is it necessary to create a controller for the routes to the pages or is it based on the file name only?
Also, to get data back and forth between the client and the device, I was going to use json data.  I saw your example that handles the GET, but was wondering how a POST would work.  Does everything in the json object show up in the form collection?
If you are interested (and I'm able to get my project working), I'd be happy to share the results as an example for the project.
Thanks,
Andy

Hi!
Static files from SD card is supported, they are streamed so you will not have any size problems. I have done benchmarks with large files and it works without any problem.
You add the static file handler via this line:
webServer.SetFileAndDirectoryService(new FileAndDirectoryService());
And you need to add a directory called WWW to your SD card where you place all static files, this will also work in the emulator but you need a folder on your drive called WWW in local resources folder instead.
I have started building a JSON deserializer but it never made it from prototype state, i could look at how much work remains to get it in a working state, it is a good thing to have! But right now you will only have GET/POST support with forms key/value pairs as in the examples. This will work from both from query string and POST data.
It is also possible to get the data stream and deserialize it in another way in your Controllers and this is located in HttpContext.Socket. But this will only work one time and only if you don´t call the build in handlers as it is from the Socket directly with no server buffers in between.
But i can check the state of my json deserializer and get back here as it will give you the easiest solution.
And i would love to see some code, Angular is awesome! :)
/Fredrik

Thanks for the quick reply.
I wonder if a library like json.net would work to deserialize the posts.  Not sure if it has dependencies that would keep it from running on the micro framework.  In the meantime, I think I can get data to the device using the forms data query screen, but I was hoping to not need to reload the page on each update.
Have you had a chance to look into D3 (http://d3js.org/)?  Since all the work is being done on the client, I think your web server on the netduino could host some pretty interesting visualizations.  AngularJS data-binding to SVGs has gotten a lot better, so for some visualizations, D3 isn’t necessary (though still very helpful!).
Thanks,
Andy
From: Fredrik Rinsén<EMAIL_ADDRESS>Sent: Monday, April 20, 2015 1:52 PM
To: Rinsen/WebServer
Cc: Andy Allman
Subject: Re: [WebServer] Questions: posting json objects & using angularjs (#7)
Hi!
Static files from SD card is supported, they are streamed so you will not have any size problems. I have done benchmarks with large files and it works without any problem.
You add the static file handler via this line:
webServer.SetFileAndDirectoryService(new FileAndDirectoryService());
And you need to add a directory called WWW to your SD card where you place all static files, this will also work in the emulator but you need a folder on your drive called WWW in local resources folder instead.
I have started building a JSON deserializer but it never made it from prototype state, i could look at how much work remains to get it in a working state, it is a good thing to have! But right now you will only have GET/POST support with forms key/value pairs as in the examples. This will work from both from query string and POST data.
It is also possible to get the data stream and deserialize it in another way in your Controllers and this is located in HttpContext.Socket. But this will only work one time and only if you don´t call the build in handlers as it is from the Socket directly with no server buffers in between.
But i can check the state of my json deserializer and get back here as it will give you the easiest solution.
And i would love to see some code, Angular is awesome! :)
/Fredrik
—
Reply to this email directly or view it on GitHubhttps://github.com/Rinsen/WebServer/issues/7#issuecomment-94538894.

Hi!
You will absolutely not need to reload the whole page, you can collect data via GET or POST via Angular data service and send any parameters you need, if you need any, in the query string or as form data, and then return these as json data from the Controller. The deserialization is only needed if you have to post in json data objects for some reason but this can be avoided in most cases.
There is probably no way that json.net will work on NETMF. The metadata support is a bit limited in NETMF and a lot of the dependencies is probably not there. But if you only need to return json data which is what i think you want my serializer should be good enough.
I have not worked with D3 but i have seen some things done in i and they look great so i think that is a good choice. We also looked at Highcharts and i think that it was easier to get started and make charts in that but there may have been a license price tag on that one.
This web server also have support for CORS if you would like to get data from multiple devices and show them at the same time in one browser from one "head"-device. CORS will avoid XSS problems.
http://en.wikipedia.org/wiki/Cross-origin_resource_sharing
I have made a example for this but don´t know where that one is right now and i don´t have any documentation on my implementation more than that you call a method in your Controller called AllowCors() and then it will add the required header and all the other details of how it works is in the wikipedia page :) But it is a pretty basic implementation that only makes it work and does not fully support the standard.
/Fredrik


Keystore and Crypto host functions discussion

While thinking about how to introduce new crypto types required by the Sassafras protocol, I ended up thinking that maybe, before blindly start writing code, it may be a better idea to first open a discussion about how cryptographic stuff is used by Substrate in order to maybe reduce some technical debt.
Things to know:

KeyTypeId: identifies the context of application of a key (e.g. babe or grandpa)
CryptoTypeId: indetifies the cryptographic scheme (e.g. sr25519 or ed25519)
Keystore: an object used to work with secrets persisted somewhere (typically in the local filesystem)
raw public key: an opaque byte array
constructed public key: a key of one particular scheme (e.g. sr25519::Public)

Keystore
Currently, there are some inconsistencies in the keystore API.
I'm going to refer to the Local keystore implementation (currently the only implementation provided by the Substrate if we are not considering the testing one).
Signing
To sign a message there is the sign_with method which takes KeyTypeId, CryptoTypeId and a raw public key.
The public key and key-type-id are used to lookup the secret while the crypto-type-id is used to apply the proper scheme to sign the message. A constructed public key is built within the method after a match over all the supported CryptoTypeId.
The interface also provides sign_with_any and sign_with_all with an interface similar to sign_with.
The former is not used anywhere BTW and the second can be replaced by a loop in the caller (IMO it is not called so often to justify the requirement of an extra method).
Isolated there is a special ecdsa_sign_prehashed. It takes instead a constructed ecdsa::Public key
Lookup
To lookup keys there is one stand-alone method for each crypto scheme (e.g. one for sr25519 and one for ed25519).
These returns a typed Public key.
This API design is not consistent with the one used by sign_with that instead:

deals with raw public keys in the interface
is a catch-all for all crypto schemes (more crypto agile)

Key generation
The API exposes methods to generate keys and these are designed in the same way of lookup methods.
I.e. one stand-alone method for each scheme.
VRF
There is a sr25519_vrf_sign, stand-alone method for VRF signatures using sr25519. This uses constructed keys a well.
Rework
In the prospective of introducing new crypto schemes to support new protocols, I would like to discuss the interface of the keystore first.
As WiP extensions there is already:

BLS12-381 signatures introduced by this PR from @drskalman.
Here the bls_sign method is added as  a stand-alone method, i.e. not part of sign_with:

it doesn't follow the same pattern of the others (i.e. is not part of sign_with).
what if we will introduce the usage of other BLS curves in the future?

Sassafras will make use of ed_on_bls12_381 or ed_on_bls12_381_bandersnatch. A primitive that is related but not the same bls12-381. This will require its methods as well.

Just look at how many possible future curves just from arkworks we may require one day here

Option 1
One option is to provide a fully agile keystore and be less bound (at least in the interface) to the currently provided primitives.
This also allows users of Substrate that in their client are using protocols with crypto schemes not already supported by us.
To do this we may think of removing specialized methods from the interface, i.e. provide methods in a form similar to sign_with. And thus:

sign(app_id: KeyTypeId, crypto_id: CryptoTypeId, public: &[u8], msg: &[u8]) -> Result<Vec<u8>>
sign_prehashed(app_id: KeyTypeId, crypto_id: CryptoTypeId, public: &[u8], hash: &[u8]) -> Result<Vec<u8>>
vrf_sign(app_id: KeyTypeId, crypto_id: CryptoTypeId, public: &[u8], msg: &[u8]) -> Result<Vec<u8>>
public_keys(app_id: KeyTypeId, crypto_id: CryptoTypeId) -> Vec<Vec<u8>>
etc

This interface prevents the explosion of methods and thus is more sustainable.
Obviously, for example, the usage of vrf_sign with a crypto scheme that doesn't support it will return a "not supported" error...
However... should also be considered that we still provide one method per primitive in the
Crypto runtime interface.
So maybe, if a good part of the primitives, are exposed to the runtime as well the explosion will be observed there at some point
(but this may be considered as an independent problem... more below).
Option 2
Remove the generic sign_with method from the keystore and provide only  specialized stand-alone methods for all the schemes we support. We already do this for all the operations except for "signing". Thus provide instead  sr25519_sign, ed25519_sign, ecdsa_sign.
My only concern is that we will also have to introduce stuff like:

bls12_381_sign
bls12_377_sign
ed_over_bls12_381_sign
ed_over_bls12_377_sign
and there are others... who know what we're going to support in the future

Host Functions
If we decide to go with Option 1 maybe we should also consider a refactory in the sp_io crate in order to provide more fine-grained capabilities. For example @achimcc  is already adopting this approach to define all the host functions he's requiring to work with a bunch of "special" elliptic curves operations.
Maybe we should start grouping these host functions in separate files and better logically organize them.
For example instead of a giant group containing crypto "stuff" divide these in a more fine grained way as:

Bls12381
Bls12377
EdOnBls12381
EdOnBls12377
Sp25519
Ed25519
Ecdsa
...

Strict adoption of this approach also requires to extract the methods from the ever-growing sp_io::crypto module.
The ideal would be having a module like sp_io::crypto::sp25519 instead of embedding all the crypto primitives in one single crypto module.
But maybe is not backward compatible... But is this even possible (having submodules of sp_io::crypto for each crypto scheme)? Maybe we can do this at least for new stuff?

Final Considerations
All in all, I wanted to highlight and open a discussion about possible issues that I have noticed before introduce even more new stuff.
The end goal is to introduce new things in a more organized and carefully designed way before is "too late" to rethinking about it (especially for the host functions part) and accumulate technical debt.

cc @burdges @paritytech/sdk-node

The entire keystore contains currently code for stuff that was never implemented (remote signing). CryptoTypeId alongside the CryptoStore (the async interface) trait are prominent changes that were introduced because of this. We can properly do a clean up of all this stuff.

Option 2
Remove the generic sign_with method from the keystore and provide only specialized stand-alone methods for all the schemes we support. We already do this for all the operations except for "signing". Thus provide instead sr25519_sign, ed25519_sign, ecdsa_sign.

As the host functions need to be explicit any way, I would also go here the explicit way. It is not that we introduce every day a new crypto scheme. I think the current interface is basically the same since its initial creation (besides the split in sync/async).

Maybe we should start grouping these host functions in separate files and better logically organize them. For example instead of a giant group containing crypto "stuff" divide these in a more fine grained way as:
* `Bls12381`

* `Bls12377`

* `EdOnBls12381`

* `EdOnBls12377`

* `Sp25519`

* `Ed25519`

* `Ecdsa`
  ...

Strict adoption of this approach also requires to extract the methods from the ever-growing sp_io::crypto module.
The ideal would be having a module like sp_io::crypto::sp25519 instead of embedding all the crypto primitives in one single crypto module.
But maybe is not backward compatible... But is this even possible (having submodules of sp_io::crypto for each crypto scheme)? Maybe we can do this at least for new stuff?

Yes for sure! We can add these new host functions to new "modules".

BLS12-381 hash-to-curve-g2 is extremely slow, but it maybe makes more sense for Ethereum light client.
Is there a plan to add host-to-curve-g2 as a host function in substrate?

As discussed in https://eprint.iacr.org/2022/1611 we'll hash-to-G1 but have equivalent public keys on G1 and G2, so then verifiers always sum public keys on G1.  It's silly to have any verifier operations on G2 like ethereum does.

As discussed in https://eprint.iacr.org/2022/1611 we'll hash-to-G1 but have equivalent public keys on G1 and G2, so then verifiers always sum public keys on G1. It's silly to have any verifier operations on G2 like ethereum does.

I see the performance improvement to use hash-to-G1 in that paper,  but Ethereum use hash-to-G2.
It has no choice for bridges based on Ethereum beacon light client, such as Snowfork and Darwinia network.
If there is a hash-to-G2 host function in substrate, will be extremely helpful for truthless bridges.
cc @wuminzhe @AlistairStewart

@hujw77 maybe this is a topic for a separate issue.

Indeed it is. We may well have a hash to G_2 host function in the future but we don't envision Substrate nodes BLS signatures to ever need to be in G_2.


"Single source of truth" documentation

Is the documentation currently maintained/updated?
https://developers.google.com/closure/library/docs/overview
https://google.github.io/closure-library
Is there interest in having a single source of truth that people can point to? There are a few issues open regarding documentation: #968, #963, #856, #861

Neither is particularly relevant.  The developers.google.com documentation was written by someone outside our team years ago and we don't own it.  The github.io documentation was supposed to be a version that we owned, and we got a tech writer to work with us, but he bailed out before we really got it off the ground.  The idea was to scrub and mirror a bunch of our internal docs, but that has not actually happened.
At this point, it's not clear what makes the most sense w.r.t. documenting Closure, if ultimately the thing we want to recommend going forward is a new "Closure Good Parts" library that we extract out of it - it's going to look different enough that the docs should probably be started fresh for that, specifically.

For reference, which are these "Good Parts", @shicks?

This issue is being closed as a preparation step before archiving the repository.  See the README for more detail.


[BUG] UI/UX Issue (Responsive)

Description
Issue: Footer section overlap with the page content.
Solution suggestion: Set some background color to the footer.
Screenshots

Thank you for creating this issue. Wait until you are allowed to work on this before you start, and do not forget to include this issue in your pull request. While you wait, don't forget to join the community - [Automated].

Thank you for creating this issue. Wait until you are allowed to work on this before you start, and do not forget to include this issue in your pull request. While you wait, don't forget to join the community - [Automated].

Wow, you just created your first issue on this project, that's so cool. Make sure to wait till you are allowed to work on this before you do and when you are allowed to, don't forget to mention this issue in your pull request. Also join the community - [Automated]

Do you want to work on this?

Do you want to work on this?

I can work on it!

Do you want to work on this?

I can work on it!
Should I keep The position of the text fixed or should I place it at the bottom of the page, to make it responsive?

Do you want to work on this?

I can work on it!
Should I keep The position of the text fixed or should I place it at the bottom of the page, to make it responsive?

Place it at the bottom and make it responsive using clamp or min-max because on mobile the text is too big

Do you want to work on this?

I can work on it!
Should I keep The position of the text fixed or should I place it at the bottom of the page, to make it responsive?

Place it at the bottom and make it responsive using clamp or min-max because on mobile the text is too big

Sure! working on it.

Thank you for creating this issue. Wait until you are allowed to work on this before you start, and do not forget to include this issue in your pull request. While you wait, don't forget to join the community - [Automated].

:tada: This issue has been resolved in version 1.9.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

Thank you for creating this issue. Wait until you are allowed to work on this before you start, and do not forget to include this issue in your pull request. While you wait, don't forget to join the community - [Automated].


Tab autocomplete not working

Describe the bug
Since the new release of the sandbox, the tab autocomplete function within juptyer notebooks no longer works (nothing happens when you hit tab)
Expected behavior
Variables and methods should autocomplete when hitting the tab key

fixed


Generate starknet_artifacts.json file

Closes #369

Stack:

#375
#374
#373 ⬅
#372

⚠ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

✓ Commit merged in pull request #375


Simplify & streamline logging

Normally I'd lean toward minimal logging, but since this is designed to embed JS into unknown third-party websites, it really needs helpful logging to assist in support & troubleshooting. It currently has a bunch of console.log() calls littered throughout it, but it really could use some consolidation & support for log levels.
I don't really want to create too much bloat, nor rely on too much third party code (due to potential conflicts), but improving this situation should make it easier to rely on better logging solutions in the future (if needed).

PR #5 includes new _info(), _warn(), _error(), and _log() private methods, plus log_level option. Also automatically determines name of function called from and UMbed class name (for easier modification & renaming, e.g. as Discovery Map does).


AddGradleEnterprise recipe issue with existing settings plugin

Given a settings file containing a plugins block, for example:
plugins {
    id("org.gradle.test-retry") version("1.5.0") apply(false)
}

When running org.openrewrite.gradle.plugins.AddGradleEnterprise no change happens, instead is expected:
plugins {
    id("org.gradle.test-retry") version("1.5.0") apply(false)
    id 'com.gradle.enterprise' version '3.12.2'
}

This should be fixed already with the latest snapshot builds via https://github.com/openrewrite/rewrite/commit/ca5706c63eb9fd22b95e69441771406201fc1660.

Confirmed fixed with ca5706c.

Thanks, I confirm that fixes it with latest SNAPSHOT


[SUPPORT] OutOfMemory on upserting into MOR dataset

Describe the problem you faced
We have a Spark Streaming application running on EMR 5.31.0 that reads from a Kinesis stream (batch interval of 30 minutes) and upserts to a MOR dataset that is partitioned by date. This dataset is ~ 2.2 TB in size and ~ 6 billion records. Our problem is the application is now persistently crashing on an OutOfMemory error regardless of the batch input size (stacktrace attached below is for an input of ~ 1 million records and size ~ 250 mb). For debugging we've tried replacing the Hudi upsert with a simple count and afterwards there seems to be minimal memory usage by the application based on the Spark UI.
 df.write.format("org.apache.hudi").options(**hudi_options).mode("append").save(OUTPUT_PATH)
This seems similar to https://github.com/apache/hudi/issues/1491#issuecomment-610626104 since it seems to be running out of memory on reading old records based on the stacktrace. There are a lot of large log files in the dataset:
...
2021-01-01 00:28:08  910487106 hudi/production/transactions_v9/2020/12/30/.0ef05182-e0ad-44b5-b52d-0412a6b44a01-1_20210101033018.log.1_3774-34-62086
2021-01-01 21:03:26  910490317 hudi/production/transactions_v9/2020/12/30/.0ef05182-e0ad-44b5-b52d-0412a6b44a01-1_20210101033018.log.11_3774-34-62109
2021-01-01 16:52:39  910495970 hudi/production/transactions_v9/2020/12/30/.0ef05182-e0ad-44b5-b52d-0412a6b44a01-1_20210101033018.log.9_3774-34-62083

Our Hudi configs are:
hudi_options = {
        "hoodie.table.name": "transactions",
        "hoodie.datasource.write.recordkey.field": "id.value",
        "hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
        "hoodie.datasource.write.partitionpath.field": "year,month,day",
        "hoodie.datasource.write.table.name": "transactions",
        "hoodie.datasource.write.table.type": "MERGE_ON_READ",
        "hoodie.datasource.write.operation": "upsert",
        "hoodie.consistency.check.enabled": "true",
        "hoodie.datasource.write.precombine.field": "publishedAtUnixNano",
        "hoodie.compact.inline": True,
        "hoodie.compact.inline.max.delta.commits": 10,
        "hoodie.cleaner.commits.retained": 1,
}

Our Spark configs are (CORES_PER_EXECUTOR = 5, NUM_EXECUTORS = 30) for a cluster running on 10 r5.4xlarge instances:
            "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
            "spark.sql.hive.convertMetastoreParquet": "false",
            # Recommended from https://aws.amazon.com/blogs/big-data/best-practices-for-successfully-managing-memory-for-apache-spark-applications-on-amazon-emr/
            "spark.executor.cores": CORES_PER_EXECUTOR,
            "spark.executor.memory": "36g",
            "spark.yarn.executor.memoryOverhead": "4g",
            "spark.driver.cores": CORES_PER_EXECUTOR,
            "spark.driver.memory": "36g",
            "spark.executor.instances": NUM_EXECUTORS,
            "spark.default.parallelism": NUM_EXECUTORS * CORES_PER_EXECUTOR * 2,
            "spark.dynamicAllocation.enabled": "false",
            "spark.streaming.dynamicAllocation.enabled": "false",
            "spark.streaming.backpressure.enabled": "true",
            # Set max rate limit per receiver to limit memory usage
            "spark.streaming.receiver.maxRate": "10",
            # Shutdown gracefully on JVM shutdown
            "spark.streaming.stopGracefullyOnShutdown": "true",
            # GC options (https://databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html)
            "spark.executor.defaultJavaOptions": "-XX:+UseG1GC -XX:+PrintFlagsFinal -XX:+PrintReferenceGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12",
            "spark.driver.defaultJavaOptions": "-XX:+UseG1GC -XX:+PrintFlagsFinal -XX:+PrintReferenceGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12",
            "spark.streaming.kinesis.retry.waitTime": "1000ms",  # default is 100 ms
            "spark.streaming.kinesis.retry.maxAttempts": "10",  # default is 10
            # Set max retries on S3 rate limits
            "spark.hadoop.fs.s3.maxRetries": "20",

Environment Description

Hudi version : 0.6.0

Spark version : 2.4.6 (EMR 5.31.0)

Hive version : Hive 2.3.7

Hadoop version : Amazon 2.10.0

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : no

Additional context
Although the records are partitioned by date the Kinesis stream contains many backfill records as well so many partitions may be upserted to in a single batch.
Please let me know what other context I can provide that would be useful. Thank you!
Stacktrace
  File "stream_transactions.py", line 965, in <module>
    ssc.awaitTermination()
  File "/mnt2/yarn/usercache/hadoop/appcache/application_1609869637845_0001/container_1609869637845_0001_01_000001/pyspark.zip/pyspark/streaming/context.py", line 192, in awaitTermination
  File "/mnt2/yarn/usercache/hadoop/appcache/application_1609869637845_0001/container_1609869637845_0001_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/mnt2/yarn/usercache/hadoop/appcache/application_1609869637845_0001/container_1609869637845_0001_01_000001/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "/mnt2/yarn/usercache/hadoop/appcache/application_1609869637845_0001/container_1609869637845_0001_01_000001/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o89.awaitTermination.
: org.apache.spark.SparkException: An exception was raised by Python:
Traceback (most recent call last):
  File "/mnt2/yarn/usercache/hadoop/appcache/application_1609869637845_0001/container_1609869637845_0001_01_000001/pyspark.zip/pyspark/streaming/util.py", line 68, in call
    r = self.func(t, *rdds)
  File "stream_transactions.py", line 954, in process
    df.write.format("org.apache.hudi").options(**hudi_options).mode("append").save(OUTPUT_PATH)
  File "/mnt2/yarn/usercache/hadoop/appcache/application_1609869637845_0001/container_1609869637845_0001_01_000001/pyspark.zip/pyspark/sql/readwriter.py", line 739, in save
    self._jwrite.save(path)
  File "/mnt2/yarn/usercache/hadoop/appcache/application_1609869637845_0001/container_1609869637845_0001_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/mnt2/yarn/usercache/hadoop/appcache/application_1609869637845_0001/container_1609869637845_0001_01_000001/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/mnt2/yarn/usercache/hadoop/appcache/application_1609869637845_0001/container_1609869637845_0001_01_000001/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o1902.save.
: java.lang.OutOfMemoryError
	at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
	at java.io.DataOutputStream.write(DataOutputStream.java:107)
	at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
	at org.apache.hudi.common.table.log.block.HoodieAvroDataBlock.serializeRecords(HoodieAvroDataBlock.java:114)
	at org.apache.hudi.common.table.log.block.HoodieDataBlock.getContentBytes(HoodieDataBlock.java:95)
	at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlock(HoodieLogFormatWriter.java:139)
	at org.apache.hudi.table.HoodieTimelineArchiveLog.writeToFile(HoodieTimelineArchiveLog.java:309)
	at org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:282)
	at org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:133)
	at org.apache.hudi.client.HoodieWriteClient.postCommit(HoodieWriteClient.java:381)
	at org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:126)
	at org.apache.hudi.client.AbstractHoodieWriteClient.commit(AbstractHoodieWriteClient.java:99)
	at org.apache.hudi.client.AbstractHoodieWriteClient.commit(AbstractHoodieWriteClient.java:90)
	at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:395)
	at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:205)
	at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:125)
	at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:173)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:169)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:197)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:194)
	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:169)
	at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:114)
	at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:112)
	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:677)
	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:677)
	at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$executeQuery$1(SQLExecution.scala:83)
	at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1$$anonfun$apply$1.apply(SQLExecution.scala:94)
	at org.apache.spark.sql.execution.QueryExecutionMetrics$.withMetrics(QueryExecutionMetrics.scala:141)
	at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$withMetrics(SQLExecution.scala:178)
	at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:93)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:200)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:92)
	at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:677)
	at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:286)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:272)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:230)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)
	at org.apache.spark.streaming.api.python.TransformFunction.callPythonTransformFunction(PythonDStream.scala:95)
	at org.apache.spark.streaming.api.python.TransformFunction.apply(PythonDStream.scala:78)
	at org.apache.spark.streaming.api.python.PythonDStream$$anonfun$callForeachRDD$1.apply(PythonDStream.scala:179)
	at org.apache.spark.streaming.api.python.PythonDStream$$anonfun$callForeachRDD$1.apply(PythonDStream.scala:179)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)```

@umehrot2 : Can you take a look at this ?

@umehrot2 : Can you take a look at this ?

Thank you, I have also tried setting hoodie.write.buffer.limit.bytes as per https://github.com/apache/hudi/issues/1491#issuecomment-610626104 but it still OOMs in same way (this config seems undocumented and defaults to 410241024 bytes, not sure the rationale on changing it to 131072 as per that comment)

Thank you, I have also tried setting hoodie.write.buffer.limit.bytes as per https://github.com/apache/hudi/issues/1491#issuecomment-610626104 but it still OOMs in same way (this config seems undocumented and defaults to 410241024 bytes, not sure the rationale on changing it to 131072 as per that comment)

I took a deeper look at this. For you this seems to be happening in the archival code path:
 at org.apache.hudi.table.HoodieTimelineArchiveLog.writeToFile(HoodieTimelineArchiveLog.java:309)
 at org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:282)
 at org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:133)
 at org.apache.hudi.client.HoodieWriteClient.postCommit(HoodieWriteClient.java:381)

In HoodieTimelineArchiveLog where it needs to write log files with commit record, similar to how log files are written for MOR tables. However, in this code I notice a couple of issues:

The default maximum log block size of 256 MB defined here, is not utilized for this class and is only used for the MOR log blocks writing case. As a result, there is no real control over the block size that it can end up writing which can potentially overflow ByteArrayOutputStream whose maximum size is Integer.MAX_VALE - 8. That is what seems to be happening in this scenario here because of an integer overflow following that code path inside ByteArrayOutputStream. So we need to use the maximum block size concept here as well.
In addition I see a bug in code here where even after flushing out the records into a file after a batch size of 10 (default) it is not clearing the list and just goes on accumulating the records. This seems logically wrong as well (duplication), apart from the fact that it would keep increasing the log file blocks size it is writing.

I will open a jira for this issue to track this.

I took a deeper look at this. For you this seems to be happening in the archival code path:
 at org.apache.hudi.table.HoodieTimelineArchiveLog.writeToFile(HoodieTimelineArchiveLog.java:309)
 at org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:282)
 at org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:133)
 at org.apache.hudi.client.HoodieWriteClient.postCommit(HoodieWriteClient.java:381)

In HoodieTimelineArchiveLog where it needs to write log files with commit record, similar to how log files are written for MOR tables. However, in this code I notice a couple of issues:

The default maximum log block size of 256 MB defined here, is not utilized for this class and is only used for the MOR log blocks writing case. As a result, there is no real control over the block size that it can end up writing which can potentially overflow ByteArrayOutputStream whose maximum size is Integer.MAX_VALE - 8. That is what seems to be happening in this scenario here because of an integer overflow following that code path inside ByteArrayOutputStream. So we need to use the maximum block size concept here as well.
In addition I see a bug in code here where even after flushing out the records into a file after a batch size of 10 (default) it is not clearing the list and just goes on accumulating the records. This seems logically wrong as well (duplication), apart from the fact that it would keep increasing the log file blocks size it is writing.

I will open a jira for this issue to track this.

For now, I would suggest to archive at smaller intervals. May be try out something like:

hoodie.keep.max.commits: 10
hoodie.keep.min.commits: 10

For now, I would suggest to archive at smaller intervals. May be try out something like:

hoodie.keep.max.commits: 10
hoodie.keep.min.commits: 10

Thanks Udit! I'd tried setting hoodie.commits.archival.batch to 5 earlier today after going through the source code - that got my application back and running again.
The first bug definitely seems like the root cause, after turning on more verbose logging I found several 300mb commit files being loaded in for archival before the crash (re: the second bug https://github.com/apache/hudi/blob/e3d3677b7e7899705b624925666317f0c074f7c7/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java#L353 clears the list, which isn't the most intuitive). It seems like these large commit files were generated when I set hoodie.cleaner.commits.retained to 1.
What is the trade-off in lowering hoodie.keep.max.commits and hoodie.keep.min.commits? I couldn't find much good documentation on the archival process/configs.

Thanks Udit! I'd tried setting hoodie.commits.archival.batch to 5 earlier today after going through the source code - that got my application back and running again.
The first bug definitely seems like the root cause, after turning on more verbose logging I found several 300mb commit files being loaded in for archival before the crash (re: the second bug https://github.com/apache/hudi/blob/e3d3677b7e7899705b624925666317f0c074f7c7/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java#L353 clears the list, which isn't the most intuitive). It seems like these large commit files were generated when I set hoodie.cleaner.commits.retained to 1.
What is the trade-off in lowering hoodie.keep.max.commits and hoodie.keep.min.commits? I couldn't find much good documentation on the archival process/configs.

@umehrot2 : would you mind following up on this issue

@jtmzheng : few quick questions as we triage the issue.

Were you running older version of Hudi and encountered this trying to upgrade to a latest version?
Is this affecting your production? trying to gauge the severity.
Or you are trying out a POC ? and this is the first time trying out Hudi.

@nsivabalan I have not encountered the issue again after temporarily lowering hoodie.commits.archival.batch which cleared out the large commit files being loaded for archival. I believe @umehrot2 identified the right root cause/bug in https://github.com/apache/hudi/issues/2408#issuecomment-758320870 (first one). I think these large commits were generated after I added the option hoodie.cleaner.commits.retained:1 but I'm not sure (it lined up timeline-wise and that change caused the dataset size to shrink drastically)
Some context:

dataset was always indexed with 0.6.0 (no upgrade)
we are trying to productionize a dataset in Hudi data lake, but it is not there yet.
this is also our first time working with Hudi

I think this issue can be closed as a support request though would be great to understand the different archival configs better (couldn't find good documentation on these)

thanks a lot @jtmzheng . Yeah, I have filed two tickets on this end.
https://issues.apache.org/jira/browse/HUDI-1605
https://issues.apache.org/jira/browse/HUDI-1604


firebase_auth error message getters differ across platforms

when using a standard .catchError((e) { the human readable message is contained in e.details if it's iOS but is in e.message when on Android.

@blaneyneil
The issue at https://github.com/flutter/flutter/issues/20658 has been closed and moved here. Future collaboration on this issue will be done here.

Consolidating under #1456 - we'll be looking at parsing native error messages & codes and bubbling these up to the Dart layer in a consistent cross-platform format.


Automated cherry pick of #8551: fix: make test go vet error

Cherry pick of #8551 on release/3.5.
#8551: fix: make test go vet error

Codecov Report

Merging #8552 into release/3.5 will decrease coverage by 0.05%.
The diff coverage is 0.06%.

@@               Coverage Diff               @@
##           release/3.5    #8552      +/-   ##
===============================================
- Coverage         6.14%    6.08%   -0.06%
===============================================
  Files              761      769       +8
  Lines           118373   119571    +1198
===============================================
+ Hits              7269     7278       +9
- Misses          110445   111630    +1185
- Partials           659      663       +4

Flag
Coverage Δ

#aFlag
6.08% <0.06%> (-0.06%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/apigateway/handler/init.go
100.00% <ø> (ø)

pkg/apis/list.go
0.00% <0.00%> (ø)

pkg/cloudcommon/db/project_resources.go
0.00% <0.00%> (ø)

pkg/cloudcommon/validators/validators.go
63.08% <0.00%> (-1.43%)
:arrow_down:

pkg/cloudprovider/images.go
0.00% <0.00%> (ø)

pkg/compute/guestdrivers/esxi.go
0.94% <0.00%> (-0.07%)
:arrow_down:

pkg/compute/models/cachedimages.go
1.73% <0.00%> (-0.04%)
:arrow_down:

pkg/compute/models/cloudregions.go
1.54% <0.00%> (-0.04%)
:arrow_down:

pkg/compute/models/cloudsync.go
0.00% <0.00%> (ø)

pkg/compute/models/cloudsyncelasticcache.go
0.00% <0.00%> (ø)

... and 47 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f6ad813...f896e02. Read the comment docs.


Chore: vela delete doc

Signed-off-by: Somefive<EMAIL_ADDRESS>Description of your changes

Refactor vela delete doc to let auto-doc-gen work properly. ( is treated as a html tag and will not be properly printed.)
How has this code been tested

Special notes for your reviewer

/backport 1.7

/backport 1.7


I know I can install the prepare-commit-hook to the project. But I don't want to use codegpt for every commit.
So I prefer to use codegpt commit when it's needed.
感謝!

@k0286 CodeGPT already have git prepare-commit-msg hook built-in.
Just try codegpt hook install to install the hook.
Can you describe more about your requirements? I don't really get what you want.

Thx for your replay.
E.g. I don't want to codegpt been triggered when running git rebase.
But If the prepare-commit-msg hook is intsalled, I don't have any option to disable it except uninstall the hook.

Reference from this: https://stackoverflow.com/a/57590865/910074
git rebase runs pre-rebase and post-rewrite hooks. Are you sure your git rebase will trigger prepare-commit-msg hook?

Reference from this: https://stackoverflow.com/a/57590865/910074
git rebase runs pre-rebase and post-rewrite hooks. Are you sure your git rebase will trigger prepare-commit-msg hook?

Of course, if you met the conflicts when doing rebase process.

Yeah, you are right. I actually don't want to codegpt commit been triggered during I'm solving the conflicts. But It triggers if the codegpt commit hook is installed. There are no options to bypass the prepre-commit-message hook, right?

@appleboy Do you have any idea?


TensorFlowOnSpark deadlock on yarn client

I run the mnist example on examples/mnist/spark/mnist_spark.py with this command line:
/Users/alessandro/spark-2.2.0-bin-hadoop2.7/bin/spark-submit --master yarn --deploy-mode cluster --num-executors 4 --executor-memory 10G --total-executor-cores 4 --conf spark.dynamicAllocation.enabled=false --conf spark.yarn.maxAppAttempts=1 --py-files tfspark.zip --archives hdfs:///user/alessandro/Python.zip#Python examples/mnist/spark/mnist_spark.py --images hdfs:///user/alessandro/mnist/csv/images --labels hdfs:///user/alessandro/mnist/csv/labels --model hdfs:///user/alessandro/mnist/mnist_model --output hdfs:///user/alessandro/mnist/output
The execution RUNNING:

The execution remains in running without giving any progress, also displaying spark jobs for current application. Thanks in advance.
*Everything was launched in localhost

Please look for any errors in the executor logs via the Spark UI (typically at http://localhost:8080)

Hi @leewyang I solved the issue removed line logdir = TFNode.hdfs_path(ctx, args.model)  from mnist_dist.py, but now if I want see the output of the trained procedure how can I do? Because the hdfs_path() method should stores model state into HDFS.

I think you need to add the path to the libhdfs.so file per the note here.

I work in localhost installation of Spark and Hadoop. I can't use cuda on MacOs :(

Sorry, you only need the LIB_HDFS part, i.e. find the location of the libhdfs.so file in your hadoop installation, and export LIB_HDFS only.  (You won't need LIB_CUDA, since you can't use the GPU on your mac).

I solved this issue by switching to a machine with CUDA and increase the spark.yarn.executor.memoryOverhead (set in spark-deafults configuration file)


Fix Duplicate Library from CesiumRuntime.Build.cs

Fix the duplicate "ktx_read", in library list.

Thank you so much for the pull request @VintageDeveloper! I noticed this is your first pull request and I wanted to say welcome to the Cesium community!
The Pull Request Guidelines is a handy reference for making sure your PR gets accepted quickly, so make sure to skim that.

:x: Missing CLA.

Please send in a Contributor License Agreement (CLA) and comment back here to let us know to check this!
Maintainers, here's how to check for a submitted CLA.

Reviewers, don't forget to make sure that:

[ ] Cesium for Unreal Samples works.


add F-P link between GO:0016926 protein desumoylation and GO:0016929	SUMO-specific protease activity

GO:0016926 protein desumoylation
--part_of SUMO-specific protease activity

really ?

SUMO-specific protease activity is part of  GO:0016926 protein desumoylation
(it removes the SUMO from proteins)?
I would say its the ONLY part_of GO:0016926 protein desumoylation because it's a single step process.

As I was curating some SUMO protease papers I noticed there is a slight problem with these terms.
SUMO-specific protease activity
defined
Catalysis of the hydrolysis of SUMO, a small ubiquitin-related modifier, from previously sumoylated substrates.
This is correct,  when people refer to SUMO-specific protease activity, they mena the activity that cleaves sumo conjugates from proteins.
However this term has 2 children.
GO:0070139 Name | SUMO-specific endopeptidase activity
Catalysis of the hydrolysis of peptide bonds between an alpha-carboxyl group and an alpha-amino group within the small conjugating protein SUMO.
This is describing the  C terminal cleavage of SUMO itself, and is a maturation step of SUMO.
I don't think

this should be represented as a type of  "SUMO-specific protease activity"
because it is a different context.
I captured the difference between the 2 as follows:
The other child
GO:0070140
Name | SUMO-specific isopeptidase activity
Ontology | Molecular Function
Definition | Catalysis of the hydrolysis of an isopeptide bond between the small conjugating protein SUMO and a protein to which SUMO has been conjugated.
is equivalent to the parent.
I think  GO:0070139 Name | SUMO-specific endopeptidase activity
should be obsoleted, becasue it doesn't fit the parent definition (it isn't deconjugating SUMO), and it's a gene product specific cleavage event (should be captured with a generic protease has_substrate SUMO).
(7 experimental annotation 6 reactome, 1 TAIR)

I think the issue here lies in the definition of the parent term. It should be defined like its parent, GO:0019783--Catalysis of the hydrolysis of peptide or isopeptide bonds within small proteins such as ubiquitin or ubiquitin-like proteins (e.g. APG8, ISG15, NEDD8, SUMO), or between the small protein and a larger protein to which it has been conjugated. Source: GOC:ma, GOC:mah
I suggest we loosen the definition of GO:0016929 from--Catalysis of the hydrolysis of SUMO, a small ubiquitin-related modifier, from previously sumoylated substrates.
to--Catalysis of the hydrolysis of peptide or isopeptide bonds within SUMO) or between SUMO and a larger protein to which it has been conjugated.
@mah11  many of these these terms have your xref, do you recall their original intent?

@mah11 many of these these terms have your xref, do you recall their original intent?

Sorry, no, I don't recall much about this branch, and I suspect I didn't do much besides plug text definitions in. It looks like the "[small thing]-specific protease activity" terms all follow the same definition pattern, and I don't remember any conscious effort to do otherwise.
The whole branch of MF terms for peptidase activities affecting small Ub-like modifiers looks a bit dodgy, which I strongly doubt was intentional. Broadening the definition of GO:0016929 as you suggest will help, and the same should be done for the sibling terms for other small modifying proteins.
But I also think there's a problem with the parts of the definitions that mention which bonds can be cleaved. I don't remember whether isopeptide bond cleavage should fit the definition of "peptidase activity", but at present the Ub-like yada terms are the only ones that mention anything other than peptide bonds. And I'm reasonably confident that ester, thioester, and amide bond cleavage do not fit into the scope of "hydrolysis of peptide bonds", so the GO:0036459 is_a GO:0019783 (which in turn is_a GO:0008234, and so on) is a howler of a TPV. I'm not even sure "thiol-dependent" fits under "cysteine-type peptidase". I hope I didn't create that mess!

Thanks @mah11
I don't think it is as bad as it might seem. I assume that the peptidase working group organized the top levels of the ontology correctly and isopeptidase is a type of peptidase. So with the defintion tweaks that I suggested to all of the Ub-like yada terms, things should be in order at least for those. Then the children would fall into place with respect to being appropriate subtypes of the parent.
Looking at the EC link from GO:0036459 suggests that the thiol-dependent paths are ok.

I still think its odd to combine  removal of a protein modifier from a protein with the processing(maturation)  of the modifier.  The first activity is part of desumoylation and  the second is part of protein maturation (although its a particular protein maturation that is required  for "sumoylation"  and indirectly  "desumoylation" to occur in the first place.
I can see how these were lumped historically (and this is old GOC:ma),  because the same enzyme can perform the  deconjugation step, and the processing step.
When biologist refer to SUMO specific proteases, they are referring to SUMO isopeptidases.
For example:
https://genomebiology.biomedcentral.com/articles/10.1186/s13059-014-0422-2
SUMOylation is a highly dynamic process, where deconjugation (deSUMOylation) is catalyzed by a family of cysteine proteases, termed SUMO-specific proteases or SUMO isopeptidases. A subset of these enzymes also functions in processing of the SUMO precursor proteins, which is a prerequisite for their conjugation. In this review, we summarize the current view of SUMO deconjugating enzymes.
I really think we need to  remove the grouping term SUMO-specific proteases  because it is just a grouping term for 2 activities often performed by the same enzyme.
SUMO-specific protease  should then be an exact synonym for  "SUMO isopeptidase"
SUMO isopeptidase can have the f-p links to deconjugation.
The processing of SUMO itself is a gene product specific proteolysis activity (not a process in itself like desumoylation of multiple target proteins),  it's gene product specific so it should not  really have a term SUMO-specific endopeptidase activity in modern GO. It can easily be captured with a "endopeptidase" has_substrate SUMO
annotation.

PMID:28284030 actually calls both SUMO-specific proteases (so its more of a "gene product name". But I still think  my argument for not having a grouping term stands. Different activities, same gene product "SUMO specific protease". In this case, I think "SUMO specific protease" should be a related synonym of both terms rather than a parent. The grouping term should be obsoleted and people should reannotate to the isopeptidase, the endopeptidase or both depending on what is appropriate for their data. There is no reason to make direct annotation to the  existing grouping term. It is only making the annotation less precise than it could be.

some replies to David ...

I don't think it is as bad as it might seem. I assume that the peptidase working group organized the top levels of the ontology correctly and isopeptidase is a type of peptidase.

I was in the peptidase working group, and I do remember that work fairly well. We didn't consider isopeptide bonds explicitly, but I've had another look now and learned that a peptide and isopeptide bonds are chemically the same bond, and the difference is where they are in proteins. So now I think distinguishing them would fall under the aim to get rid of substrate specificity in the peptidase branch, except where annotators clamored for, well, exceptions. Ubiquitin and similar modifiers were granted exceptions, so I don't have a problem with continuing to do so.
tl;dr: sure, go ahead as you suggest

Looking at the EC link from GO:0036459 suggests that the thiol-dependent paths are ok.

One thing I remember particularly clearly from the peptidase work is that Neil and Alan both regarded the EC peptidase classification as unreliable, and they actively encouraged us to make GO MF diverge from EC specifically in the peptidase branch.
My problem is that GO has defined the root-most "peptidase activity" term (GO:0008233) as "Catalysis of the hydrolysis of a peptide bond." Ester and thioester bonds are not peptide bonds.
GO:0036459 post-dates the peptidase work by some years and may have been unintentionally misplaced.
(I'm neutral on the question of grouping vs. not grouping "any peptidase that has anything to do with Ub/SUMO/othersmallthing"; any of 'em violate the substrate-specificity purge, but in a way I've been willing to tolerate.)

I still think the gouping is ok and annotators should simply annotate to the most specific term possible.
As far as the divergence from EC, I don't really have the expertise to do that without a great deal of research and help from experts, or at least the people who were in the peptidase working group. To close this ticket in a timely manner, I suggest that I make the tweaks to the definitions that I suggested above and solve the Ub-like blah issue. If we want to rework the peptidases again, we should open another ticket and tag it as a miniproject..

I really think we need to remove the grouping term SUMO-specific proteases because it is just a grouping term for 2 activities often performed by the same enzyme.
SUMO-specific protease should then be an exact synonym for "SUMO isopeptidase"

No, it's a grouping term based on similar molecular functions/biochemical activities.

OK good plan, too much work for a small gain. I will make a note to move our modifiers down to the specific terms instead. A good compromise would also be to flag as not for direct annotation. I will do this for us internally.

No, it's a grouping term based on similar molecular functions/biochemical activities.

Doesn't the commonality has more to do with the target molecule  but in different context ?

I think the flag is a good idea! Let me check existing annotations. We can bring this up on an annotation call and allow people to move their annotations as well. I suspect that a lot of people annotated just based on the term name and never investigated the children. @vanaukenk  can we add this to an agenda for a call?

yeah ignore me, they are both isopeptide bonds ...

PS> I think the flag is only a good idea when we haven't enumerated the possible children. It looks like they don't exist for all the terms. I'm not sure if all of these molecules are process in the same manner, do you know?
Maybe that's why some of the Ub-like blah protease terms don't have children and are defined more precisely. Do you know off the top of your head?

This might be the case. Ubiquitin is clearly processes but thats because it is  sometimes transcribed as tandem copies. I don't know if we know what does the processing (I'm sure we must). For the other modifiers I don't think they have been studied in so much depth that we know about any processing....
I have a vague inckling that I might have requested the parentage for the SUMO term in the past. I really thought I had, but then changed my mind when I looked and we had not annotated to the children....
But I bet this is the reason that this structure only exists for SUO....

But the original parent term is defined as though there are two flavors, so I suspect it wasn't just you. Maybe the best thing for me to do now is stick to the SUMO term and any others that have two flavors. I will leave the others as they are until we find that they also use the processing enzymes.

Val said, "Ubiquitin is clearly processed but that's because it is sometimes transcribed as tandem copies." In humans, it's always transcribed as a tandem polyprotein, sometimes poly-Ub and sometimes a ribosomal protein and Ub. A recent review argues that the same deubiquitinases responsible for cleaving the isopeptide bonds that attach ubiquitin to target proteins, also cleave the peptide bonds in the tandem proteins to release ubiquitin monomers (PMID: 26235645). The same group also appears to argue (guessing from the abstract) that there is a similar overlap for SUMO processing and de-SUMOylation (PMID: 26522917).

@ukemi  - Yes, we can certainly add this item to an annotation call, if needed.  The next call is not scheduled until August 8th, though.

For today's annotation call, these are the existing experimental annotations in AmiGO:
GO:0016929 SUMO-specific protease activity

uspl1, ULP1D, ESD4, OTS2 - UniProtKB

SENP2, senp2 - UniProt

ULP1A, ULP1D, ESD4, OTS2 - TAIR

ULP1, ULP2 - SGD

Ulp1 - FlyBase

PF3D7_1233900, SENP1 - GeneDB

Senp1, Senp2 - MGI

ulp1 - PomBase

GO:0070139 SUMO-specific endopeptidase activity

SENP1, SENP2, SENP5 - Reactome

ASP1 - TAIR

GO:0070140 SUMO-specific isopeptidase activity

ULP1, ULP2, ULP3 - CGD

Uspl1 - MGI

Uspl1 - RGD

WSS1 - SGD (a NOT annotation)

USPL1 - UniProt

uspl1 - UniProtKB


format:json causes browser to close immediately

I am running a Protractor cucumber test and trying to generate report using protractor-multiple-cucumber-html-reporter-plugin.
But when I am using format:json:result.json in the config file the browser(chrome) immediately closes as soon as the test starts running and it shows all the test cases passed in the report.
But I wrote the scenarios in such a way that some test cases should fail.This only happens when I using format:json:result.json in cucumberOpts.
When I am using format:'pretty', browser works fine and it shows running of all the test cases and also it shows correct number of test cases passed and failed.
Please find my config file
const path = require('path');

exports.config = {
	directConnect: true,
    framework: 'custom',
    frameworkPath: require.resolve('protractor-cucumber-framework'),
    cucumberOpts: {
        require: [
        	 'maths.js',
        ],
        // Tell CucumberJS to save the JSON report
        format: 'json:.tmp/results.json',
        strict: true
    },

    specs: [
        '*.feature'
    ],

    multiCapabilities: [{
        browserName: 'chrome',
        shardTestFiles: true,
        maxInstances: 2,
        chromeOptions: {
            args: ['disable-infobars']
        }
    }],

    // Here the magic happens
    plugins: [{
        package: 'protractor-multiple-cucumber-html-reporter-plugin',
        options:{
        	 automaticallyGenerateReport: true,
             removeExistingJsonReportFile: true
        }
    }]
};

package.json
{
  "name": "protractor-cucumber-example",
  "description": "Example of using Protractor with Cucumber and Page Objects",
  "repository": "https://github.com/mlev/protractor-cucumber-example",
  "license": "Apache-2.0",
  "devDependencies": {
    "chai": "~3.5.0",
    "chai-as-promised": "~6.0.0",
    "cucumber": "~1.3.2",
    "multiple-cucumber-html-reporter": "^1.0.1",
    "protractor": "^5.1.1",
    "protractor-beautiful-reporter": "^0.5.0",
    "protractor-cucumber-framework": "^3.0.0",
    "protractor-multiple-cucumber-html-reporter-plugin": "^1.1.0",
    "protractor-pretty-html-reporter": "0.0.14"
  },
  "scripts": {
    "prestart-webdriver": "webdriver-manager update",
    "start-webdriver": "webdriver-manager start",
    "protractor": "protractor protractor.conf.js"
  },
  "dependencies": {
    "pretty": "^2.0.0"
  }
}

Hi @doe-d
Can you place some console log here to see what happens. This is a strange behaviour. I also use it for testing in the project for my customer and I'm not experiencing this behaviour.

Hi @wswebcreation Console log shows all test passed.
Console log
[13:52:22] I/launcher - Running 1 instances of WebDriver
[13:52:22] I/direct - Using ChromeDriver directly...
Feature: test page
Scenario: Verify tilte contains Angular Js
√ Given I visit angularjs.org
√ When I check the title
√ Then I title contains angularjs
1 scenario (1 passed)
3 steps (3 passed)
0m00.004s
===============================================================================
Multiple Cucumber HTML report generated in:

C:\Users\user\eclipse-workspace\test_7\.tmp\report\index.html

Tnx for using Multiple Cucumber HTML report

Grtz wswebcreation

===============================================================================
[13:52:35] I/launcher - 0 instance(s) of WebDriver still running
[13:52:35] I/launcher - chrome #01 passed
Please find below the config file
conf.js
// An example configuration file.

exports.config = {
  directConnect: true,

  // Capabilities to be passed to the webdriver instance.

  // Framework to use. Jasmine is recommended.
  framework: 'custom',
  frameworkPath: require.resolve('protractor-cucumber-framework'),
  // Spec patterns are relative to the current working directory when
  // protractor is called.
  specs: ['angular.feature'],

   cucumberOpts: {

      require: [
          'angular_steps.js',

            ],

       format: 'json:.tmp/results.json',
        strict: true
      },
  multiCapabilities: [{
      browserName: 'chrome',
      shardTestFiles: true,
      maxInstances: 2,
      chromeOptions: {
          args: ['disable-infobars']
      }
  }],

  // Here the magic happens
  plugins: [{
      package: 'protractor-multiple-cucumber-html-reporter-plugin',
      options:{
       	 automaticallyGenerateReport: true,
           removeExistingJsonReportFile: true,
           openReportInBrowser: true
      }
  }]
};

'angular.feature'
Feature: test page
Scenario: Verify tilte contains Angular Js

Given I visit angularjs.org
When I check the title
Then I title contains angularjs

angular_steps.js
var chai = require('chai').use(require('chai-as-promised'));
var expect = chai.expect;
var angularSteps = function() {

  var angularPage = require("C:\\Users\\user\\eclipse-workspace\\test_7\\angular_page.js");

  this.World = function MyWorld() {
   this.page = new angularPage();
  };
  this.Given(/^I visit angularjs\.org$/, function (callback) {
      // Write code here that turns the phrase above into concrete actions
      this.page.get();
      callback();
    });
  this.When(/^I check the title$/, function (callback) {
      // Write code here that turns the phrase above into concrete actions

      callback();
    });
  this.Then(/^I title contains angularjs$/, function (callback) {
      // Write code here that turns the phrase above into concrete actions
      expect(browser.getTitle()).to.eventually.contains('nonAngular');
      callback();
    });
}
module.exports = angularSteps;

angular_page.js
var angularPage = function() {
  this.get = function() {
    browser.get('http://www.angularjs.org');
  };
 }
module.exports = angularPage;

package.json
{
"name": "protractor-cucumber-example",
"description": ,
"repository": ,
"license": "Apache-2.0",
"devDependencies": {
"chai": "~3.5.0",
"chai-as-promised": "~6.0.0",
"cucumber": "~1.3.2",
"cucumber-html-report": "^0.6.1",
"cucumberjs-junitxml": "^1.0.0",
"protractor": "^5.1.1",
"protractor-cucumber-framework": "^3.0.0",
"protractor-multiple-cucumber-html-reporter-plugin": "^1.1.0",
"protractor-pretty-html-reporter": "0.0.14"
},
"scripts": {
"prestart-webdriver": "webdriver-manager update",
"start-webdriver": "webdriver-manager start",
"protractor": "protractor protractor.conf.js"
},
"dependencies": {
"cucumber-junit-reporter": "0.0.1",
"pretty": "^2.0.0"
}
}

Ok, tnx, will dive into it this weekend.

can anyone help me . I am also facing the same issue.

Your steps aren’t waiting for the asynchronous actions. See this

Hi @darrinholst
Tnx for referring! Didn't saw that, sorry that you have been bothered about it in your project.
Going to close the issue now


⚠ ClamAV has degraded performance

In c6add2c, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 69 ms

Resolved: ClamAV performance has improved in 5659e63.


cannot glfw-imgui.lib

Required Info

Camera Model
D435

Firmware Version
<IP_ADDRESS>

Operating System & Version
Windows 10

Platform
PC

SDK Version
2.0

Language
{C++ }

Segment
Gait analysis}

Issue Description
I ran "multicam" in the sample code. There was a link error. The contents are as follows.
"LINK : fatal error LNK1104: 'C:\Program Files (x86)\Intel RealSense SDK 2.0\samples\multicam\Debug\glfw-imgui.lib' cannot open file."
The file doesn't really exist.
My actions to fix the error are:

vc++ directory -> include directory ==> C:\Program Files (x86)\Intel RealSense SDK 2.0\third-party\glfw-imgui\include
linker->general ==>C:\Program Files (x86)\Intel RealSense SDK 2.0\lib\x86;C:\Program Files (x86)\Intel RealSense SDK 2.0\lib\x64
linker -> input -> Additional dependencies ==> realsense2.lib

No source code modifications.
That's all. somebody help me!!

Hi @devLupin,
Please make sure that you follow Windows 10 Installation instructions that described here:
https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_windows.md
After completion of this process, please run "multicam" example and let us know if you still
experience the same issue.
Thank you!

@devLupin,
Do you still need help with this question?

@RealSenseCustomerSupport
I solved it in a different way. thank you!


Fix to issue 23326 for completions on new.target

Fixes #23326

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: elizabethdinella sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

(thought I had a suggestion but I didn't)


Scroll bar not showing

Describe the bug
There is no scroll bar in any windows with this theme. This makes it difficult to scroll long documents, since you must use the mouse scroll all the time.
To Reproduce
Steps to reproduce the behavior:

Install theme
Notice that the scroll bar has vanished

Expected behavior
The scroll bar shouldn't disappear.
Screenshots

Desktop (please complete the following information):

OS: Ubuntu 20.04, Kali Linux 2021

Hi there,
yeah that was an intentional choice, but I will disable this Setting for now and want to make it an optional setting in the future which I mentioned here: #67
Until then, if somebody is stumbling over this thread in the future, that the scrollbar is visible again, please use the "Hider" plugin from the MarketPlace. :)


docs: add interactive game blog in CN

Summary
translate US -> CN
@geometryolife could u help me to finish the translation?

The translation has been completed, please take a look @templexxx


Move userdata creation to separate script

to make it easier to differentiaate from deployment
scripts.

please test this

please test this

for some reason the builds are getting failed. need to look into it.

this build was in a passed state. even though it times out...

Nops, I see contrail schema transformer was down. I had to restart it to fix that problem. Just mins before the system got in good state.

ah. I just saw that it came up.

is it an issue? I've never seen that before...

b/c it probably has nothing to do with this patch

I dont know but for last few tests failed today. just looking at them to see any pattern.

Also triggering another test

please test this

Weird! I see consul of 513 has the node entries for build #512 also. This was causing failures for 513. Need to check more on this, please dont delete #513 (512 is already got deleted).

There is some issues (some change had happen) there with the way consul membership happens. Now consul node is getting membership from the nodes of other builds also, which is causing failure of all these builds. Investigating.

please test this

@hkumarmk with the patch I merged earlier, I got consul working again. I reran tests to validate this patch and it looks fine now. Please review and merge.

Done!!


Synchronize page numbering in outline with page counter

This PR addresses issue #275.
Description
Instead of inserting the number of the page where a heading is located, we now take into account the state of the page counter and therefore query the page counter at the location of the heading.
Discussion

Currently (with the changes in this PR), when we have
#outline()

#pagebreak()
#counter(page).update(1)
= heading
#counter(page).step()
...

the heading will be listed with page number 1 instead of 2 in the outline, that is, if the page counter is altered after the heading, the page number in the outline will not match the page number appearing in the footer.
I will be happy to change that, however I am unsure how to specifically query the state of the page counter at the end of a page.

I tested what happens, if we have no numbering or footer set, that is, #set page(numbering: none, footer: none). It seems to behave correctly, however as this is my first time working with this codebase I cannot fully understand why this is the case. I'd recommend that the reviewer double checks the correctness.

I'll see whether I can handle the edge case with the page counter update in a general way.


Messages bug

Do not believe anything that gets sent on the tucker account, i am not sending those messages.
Ask me here if i send a message just to be sure.

Also if anything wierd gets posted on BiblitoecaOvidiu, still not me.
Ask me here!

It was me all the time, some may be others but i dont know :D

Fixed on 276cc4ef864ed424037c3180ef95f6d50318f381

unpin this

sure but I fixed it! lets go!