Behavior when token is not found

When aggregate_pullreq.py is executed without token and data folder is empty, pr.db was created and exit with error.
$ python aggregate_pullreq.py <repository_url>
Warning: Token file does not exist. path: /home/user/Programs/Github/aggregate_pr/.token
"/home/user/Programs/Github/aggregate_pr/data/pr.db" does not exist. Create a new DB file and continue.
Error Status: 401

In the above state, aggregate_pullreq.py doesn't work correctly although token exists.
$ python aggregate_pullreq.py <repository_url>
Traceback (most recent call last):
  File "/home/user/Programs/Github/aggregate_pr/aggregate_pullreq.py", line 64, in <module>
    main()
  File "/home/user/Programs/Github/aggregate_pr/aggregate_pullreq.py", line 43, in main
    db.write_data(all_pr)
  File "/home/user/Programs/Github/aggregate_pr/modules/database.py", line 51, in write_data
    only_add = get_only_additional(ldf)
  File "/home/user/Programs/Github/aggregate_pr/modules/database.py", line 44, in get_only_additional
    original = self.to_ldf()
  File "/home/user/Programs/Github/aggregate_pr/modules/database.py", line 56, in to_ldf
    return self.conn.sql(f"{ALL_SELECT}").pl().lazy()
duckdb.CatalogException: Catalog Error: Table with name merged_pr does not exist!
Did you mean "pg_am"?

If I do one of the following, it works correctly.

remove data/pr.db
run with --init option

It would be better to exit immediately if token doesn't exist.

Thanks!
Permanent measures will be addressed in #24.


BUG: SIGSEGV in qhull when array type is wrong

Describe your issue.
Received a SIGSEGV, when creating a scipy.spatial.HalfspaceIntersection with an integer array instead of a float array.
Downgrading scipy to an earlier version (1.6.0 but did not try in-between), fixed the issue:
I got an error message telling me the input array has wrong type.
Adding dtype=float indeed solved this usage issue.
Execution environment:
ubuntu 20.04
Reproducing Code Example
from scipy.spatial import HalfspaceIntersection
import numpy as np

intersection = HalfspaceIntersection(
    halfspaces=np.array([[0, 1, -1],
                         [0, -1, 1],
                         [1, 0, -1],
                         [-1, 0, 1]]),
    interior_point=np.array([0, 0])
)

Error message
Segmentation fault (core dumped)

SciPy/NumPy/Python version information
1.8.1 1.22.4 sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)

I just tried this with the main development branch ('1.10.0.dev0+1272.31013c0'), and I got the segmentation fault.


Fix broken RecvMsgOut parsing

The current parsing doesn't actually let the buffer ever be truncated and instead errors out immediately. This PR fixes it to only ever out when the buffer is smaller than the header. This implementation matches liburing's validation: https://github.com/axboe/liburing/blob/c44319a6bfb9953b430325227a94136690c19c33/src/include/liburing.h#L831-L834

This looks good, is there a regression test possible?

I added a test, but it's pretty terrible as I didn't want to deal with cancelling the recvmulti to change its header length or add ancillary data. That said, it does at least test for payload and namelen truncation.

@quininer any chance you can run the test on your machine? It works fine on mine, so I'm not sure what's going on in CI. Maybe old kernels didn't set the trunc flag?

You can try check some flag to skip it on ci.

Sounds good, I added a kernel version check for 6.6 (which is what I'm running), but it might work on earlier kernels.

maybe you are interested review this @lucab :)

Ok, I think I addressed all the feedback.

Thank you!


新包没有打Jenkins镜像

In GitLab by @cuiaq123 on May 13, 2020, 07:13

In GitLab by @cuiaq123 on May 13, 2020, 09:16
closed


[2.0.1] Native scrollbar is flashed when the view first starts overflowing

Simplebar 2.0.1. Chrome 55. Windows 10.
When a view goes from not needing a scrollbar to needing a scrollbar, the native scrollbar will flash for a brief moment of time. Probably due to debouncing and/or initialization times.
I wonder if there are any ways to avoid the flashing?

Hi,
I can't reproduce this issue on my testing page. How can I reproduce?
The debounce shouldn't be the problem as we are using the leading option (so calculation got called immediately anyway, only subsequent calls are debounced).

Hey!
I tried to create a test to reproduce the issue, but failed to create a minimalistic example.
I think the scroll was flashing cause of the complexity of my view rendering - it added a lot of things to the browser main queue and by the time the queue got to the SimpleBar, the browser had already decided to render the default scrollbar.
I cannot give you a reproducible testcase right now, sorry. I'll look into it a bit more on my own for now, Currently it seems it's entirely my fault. 😐
Great library, thank you for your efforts!

Ok thanks for this. I hope it's not caused by the Mutation Observer, I didn't test it in situation with a big DOM and lot of manipulations.
If you come back with some performances issues I'll try to take a look at that as it may be concerning.

While trying to create a testcase to replicate the issue at hand, I ran across another issue: scrollbar slider position calculations. In scrollable areas with extremely long content, the scrollbar slider position seems to be calculated a bit wrong, causing the native scrollbar to appear as well.
See the fiddle below. Scroll to the bottom to see the issue. (Windows 10, Chrome 55, Simplebar 2.0.1).
https://gist.run/?id=1b4a0c88d0a5772ef26248ca22e87a2e

Oh that's a weird one!
Apparently it's happening no matter the number of item, when you reach the end of the list!

Hmm, true. Could be related to the fact that it is a fragment inside an iframe, not in an actual website.


cache loaded images

looks like ggez does a full load/decode on every Sprite::from_file. This slows down debug builds a lot.
Also, check text caching.

Working on this a part of macroquad rewrite


Add yaml extension to default

Both .yml and .yaml extensions should be used be default

Thank you!

Released in 5.1.0


Svg badges are displayed incorrectly in the markdown card after post publication

Issue Summary
A set of svg badges on one line are displayed on separate lines in the markdown card after publication.
However, inside the editor the badges are displayed correctly (on one line).
To Reproduce

Create a new post
Select a Markdown card from the "+" menu
Add multiple svg badges on one line, like this:

[![DeviantArt](https://img.shields.io/static/v1?message=DeviantArt&logo=deviantart&labelColor=5c5c5c&color=00DC96&logoColor=white&label=%20)]() [![Steam](https://img.shields.io/static/v1?message=Steam&logo=steam&labelColor=5c5c5c&color=gray&logoColor=white&label=%20)]() [![Shikimori](https://img.shields.io/static/v1?message=Shikimori&logo=shikimori&labelColor=gray&color=lightgray&logoColor=white&label=%20)](https://genzmey.su) [![PlayStation](https://img.shields.io/static/v1?message=PlayStation&logo=playstation&labelColor=gray&color=blue&logoColor=white&label=%20)]() [![GitHub](https://img.shields.io/static/v1?message=GitHub&logo=github&labelColor=gray&color=green&logoColor=white&label=%20)]() [![LiveLib](https://img.shields.io/static/v1?message=LiveLib&logo=libreoffice&labelColor=gray&color=53BDFB&logoColor=white&label=%20)]() [![YouTube](https://img.shields.io/static/v1?message=YouTube&logo=youtube&labelColor=gray&color=red&logoColor=white&label=%20)]()

In the editor preview you will see the badges on one line (as expected):

Publish the post and see the result:

Badges are placed under each other

Why do you consider this to be a bug?

Other services using markdown put badges on one line in this case
The editor preview is different from the final result

What did you expect to happen instead?
All badges on one line, just like the editor preview
Technical details:

Ghost Version: 3.39.0 (Ubuntu 20.04.1 LTS)
Node Version: v12.20.0
Browser/OS: Mozilla Firefox 83.0 (64-bit) / Windows 10
Database: MySQL 8.0.22

Uh, I'm sorry but I didn't think to check it out with different themes right away
Looks like this is a casper theme issue (I tried the theme Massively and everything is displayed correctly)
Should I close this issue and open the same one in the casper repository?

@GenZmeY yes, please reopen in theme repositories you think are appropriate for this issue.


Remove AWS environment from component and use provisioner params instead

What does this PR do?
Remove AWS environment from Kubernetes and ECS environment so they remain "cloud agnostic" and rather use a provider parameter to provide an existing environment

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

/merge

/merge


Set default witnesses in case of collision

This PR changes the behavior of CD_Pair::getClosestPoints, in order to provide a default value of the witness points in case of collision when the penetration depth computation is disabled.
The new behavior will provide the last computed support points that were computed before the collision was detected. It should provide a better direction to separate the objects than a random set of points.

The unit test does not pass because we need to update the test in the submodule of shared tests (merge this PR https://github.com/jrl-umi3218/sch-shared-tests/pull/8)

It was a draft because the tests were not supposed to pass yet.


DADI image support for "No image conversion"

TLDR: we'd love to request for feasibility of adding DADI blobs as part of the original OCIV1 image
Use cases:

Easier management of a single image for both accelerated and original
More transparent onboarding since it does not require workload to change the deployment config, workloads could use the same image as OCIV1 and later the node could decide when to start switching to accelerated format for a seemless user experience.
Quick rollback: similar to 2, in case we encountered issue for accl image, we can switch containerd config snapshotter to overlayfs for a quick rollback.
Easier security scanning and auditing

AFAIK, there was previously a gap in image spec to achieve this state. I'm wondering if DADI could leverage on the new addition into the image spec for reference types https://github.com/opencontainers/wg-reference-types
An example can be found here https://github.com/awslabs/soci-snapshotter#no-image-conversion
Just a random thought, please feel free to correct me:

Pull image, the image is an OCI V1 image with DADI blobs as its additional artifacts.
Containerd calls overlaybd-snapshotter.Prepare
overlaybd-snapshotter checks the image, if it has DADI blobs, then stream with overlaybd. Otherwise fall back to overlayfs.

You mean referencing DADI image blobs in OCIv1's metadata, and detect that reference in overlaybd-snapshotter?

@lihuiba that's correct. The end goal is to use a single OCIV1 image that can be run with/without acceleration for easier rolling updates, rollback, image security scanning etc.


[wip] Implement strncmp

I'll be honest and say I mostly have no idea what I'm doing here, so I'd appreciate it if I could get some extra review and guidance for this. This is my first time working with any serious compiler stuff, and my first real exposure to LLVM. I know you're probably busy for the 0.9.1 release, so feel free to put this on the backburner until you have more time.
For some reason:
BEGIN { if (strncmp("hhvm", str($1), 4)) { printf("hello!"); } }

results in
bpftrace: /usr/include/llvm/IR/Instructions.h:1117: void llvm::ICmpInst::AssertOK(): Assertion `getOperand(0)->getType() == getOperand(1)->getType() && "Both operands to ICmp instruction are not of the same type!"' failed.

but, if(1) or if(strncmp("hhvm", str($1), -1) != 0) work fine. I think I'm missing something in the semantic analyzer but I'm not sure. It might need some changes to the if logic, but I'm not sure what/where those would be.
Fixes #459

Overall code lgtm. Will do some more testing later.

For some reason:
BEGIN { if (strncmp("hhvm", str($1), 4)) { printf("hello!"); } }

results in
bpftrace: /usr/include/llvm/IR/Instructions.h:1117: void llvm::ICmpInst::AssertOK(): Assertion `getOperand(0)->getType() == getOperand(1)->getType() && "Both operands to ICmp instruction are not of the same type!"' failed.

but, if(1) or if(strncmp("hhvm", str($1), -1) != 0) work fine.

LLVM is pretty strict with Int types, it won't let you use most binary operators between two integers of different size. CreateStrncmp will save the result of the comparison as an Int8 (irbuilderbpf.cpp#L376, irbuilderbpf.cpp#L430), and if will try to compare it to a Int64 (which is always zero, codegen_llvm.cpp#L1127). So why == works? Turns out we uppercast our Binop results to Int64 to try to avoid this issue (codegen_llvm.cpp#L796). Changing the type used to store the result strcmp to Int64 fixes the issue:
diff --git a/src/ast/irbuilderbpf.cpp b/src/ast/irbuilderbpf.cpp
index c8aa040..9778aed 100644
--- a/src/ast/irbuilderbpf.cpp
+++ b/src/ast/irbuilderbpf.cpp
@@ -378,7 +378,7 @@ Value *IRBuilderBPF::CreateStrcmp(Value* val, std::string str, bool inverse) {
 Value *IRBuilderBPF::CreateStrncmp(Value* val, std::string str, uint64_t n, bool inverse) {
   Function *parent = GetInsertBlock()->getParent();
   BasicBlock *str_ne = BasicBlock::Create(module_.getContext(), "strcmp.false", parent);
-  AllocaInst *store = CreateAllocaBPF(getInt8Ty(), "strcmp.result");
+  AllocaInst *store = CreateAllocaBPF(getInt64Ty(), "strcmp.result");

   CreateStore(getInt1(inverse), store);

@@ -436,7 +436,7 @@ Value *IRBuilderBPF::CreateStrncmp(Value* val1, Value* val2, uint64_t n, bool in
   */
   Function *parent = GetInsertBlock()->getParent();
   BasicBlock *str_ne = BasicBlock::Create(module_.getContext(), "strcmp.false", parent);
-  AllocaInst *store = CreateAllocaBPF(getInt8Ty(), "strcmp.result");
+  AllocaInst *store = CreateAllocaBPF(getInt64Ty(), "strcmp.result");
   BasicBlock *done = BasicBlock::Create(module_.getContext(), "strcmp.done", parent);

   CreateStore(getInt1(inverse), store);

Thanks for the pointers, that was very helpful! While trying to fix this test:
bpftrace -e 'BEGIN { if (!strncmp(str($1), str($2), 7)) { printf("not equal");} exit();}' "python3-proc" "hhvm"

I discovered that ! dosen't work well inside if statements, for example, this fails:
bpftrace -e 'BEGIN { $x = 5; if (!$x) { printf("not equal");} exit();}'

I'll ignore that for now because that seems like a seperate issue though.

Sure, that seems fine to me, it's a little less convenient but it is better to follow the standard strncmp return values. I don't think I can emulate the greater than/less than zero behavior without  a little more work (which I'd rather leave for a separate diff), but zero/non-zero should be easy.


Issue with Enum  string as values

Environment data

Language Server version: v2021.11.2
OS and version: macOS 10.15.7
Python version:  3.8.2

Expected behaviour
from enum import Enum
class OptimizerDirection(Enum):
    MINIMIZE: str = 'minimize'
    MAXIMIZE: str = 'maximize'

Logs
Expression of type "Literal['loss']" cannot be assigned to declared type "Literal[OptimizerMonitor.LOSS]"
"Literal['loss']" cannot be assigned to type "Literal[OptimizerMonitor.LOSS]"
PylancereportGeneralTypeIssues

You should not include str annotations for enumerated values. These fields become enum literals. Simply delete the : str annotation, and the error will go away.

You should not include str annotations for enumerated values. These fields become enum literals. Simply delete the : str annotation, and the error will go away.

Ok! Thanks for you help!


[Python] Add bindings for Device and MemoryManager classes and related methods

Through the C++ foundations, PyArrow already has support for representing non-CPU data (e.g. you can have a pyarrow.Table object backed by buffers in GPU memory), but this is not yet exposed very well in the Python bindings. Specifically, over the last years, several features were added on the C++ side (the Device and MemoryManager classes, more recently a set of Copy functions that take a target memory manager) that don't yet have equivalent bindings in PyArrow.

[x] Add bindings for DeviceAllocationType, Device and MemoryManager and expose as attributes on the Buffer class: https://github.com/apache/arrow/pull/41685
[ ] Test Device and MemoryManager with CUDA (and integrate with existing pyarrow.cuda APIs?): https://github.com/apache/arrow/pull/42221
[ ] #42222

Going to close this to keep track of it in the 17.0 milestone, given the main PR has been merged. And the last item has its own sub-issue that is still open.


NSInternalInconsistencyException Reason: Invalid update: invalid number of rows in section

I'm building a professionnal App, wich turn around an Agenda...
I can't use agenda plugins because I need to control weeks synchronisation and other special things...
By the way, I discover your awesome plugin, and I decided to create my own control with it :

One Carousel for my "Weeks" (navigate or add weeks dynamicaly)
Another Carousel for the "Days" of the current week (navigate on the current week days)

I've done a demo, and it works (quite) perfectly on Android.
But when I want to test it on iOS, I got errors when I want to "Add a week" (before or after the current) :
Foundation.MonoTouchException: Objective-C exception thrown.  Name: NSInternalInconsistencyException Reason: Invalid update: invalid number of rows in section 0.  The number of rows contained in an existing section after the update (2) must be equal to the number of rows contained in that section before the update (2), plus or minus the number of rows inserted or deleted from that section (1 inserted, 0 deleted) and plus or minus the number of rows moved into or out of that section (0 moved in, 0 moved out).
I've tried to update with or without "BeginInvokeOnMainThread" the Carousel Datasource (witch is an ObservableCollection of my ViewModels), or Lock my Collection, or...
But I can't make it work on iOS...
Did I miss something ?
Does I have to update in another way ?
Is there a bug with ObservableCollection as Carousel DataSource ? (I doubt,  but maybe...)
Thanks for your lightening (I'm turning crazy...)
(PS: anybody can feel free to use my sample... and to make remarks if you see better way to evolve my code ^^)
TestCalPCL.zip
Demo

Sorry, i have been sick. Will take a look at this.

And i dont get this error in iOS. It actually works but yes PositionSelected is called twice. I have to analize your code first.

Thanks for your response.
With my sample, I reproduce each time... (I'm compiling agains iOS 10.3 -> I need to run app on iPhone4 x86, I cannot compile against iOS11)
But it seems that my original problem came with an iOS ObservableCollection "Clear()" Datasource Binding (not your CarouselView plugin).
For people who get the same error, there is a solution (DayViewModels is my carousel Datasource) :
bug on iOS (works on Android):
DayViewModels.Clear();
Resolution :
while (DayViewModels.Count > 0) {      DayViewModels.RemoveAt(0); }
But it generates others doubts about carouselview :

When the previous code is doing a "DayViewModels.RemoveAt(0);" -> the Position Binded is fired (PositionChanged)

after that "Clear Hack", I need to add new DayViewModels (7, one for each day), and when I do a "DayViewModels.Add(new dayviewmodel)" -> The Position binded is fired (PositionChanged) too...

Edit : The previous "hack" is not a good solution...
When I did the "While{VM.RemoveAt(0)}" and add after that new VM, a very strange comportment is happening... the new VM are messed-up and not in correct order...
I don't know if it is a special case with my code... but with the "VM.Clear()" method, it works perfectly on Android (but crash on iOS)
So I'm blocked and I can't make my sample work...

I'm having this issue as well when performing programmatic index changes on a static number of views. Just wanted to add my stack trace.
Objective-C exception thrown.  Name: NSInternalInconsistencyException Reason: Don't know about flushed view <Xamarin_Forms_Platform_iOS_Platform_DefaultRenderer: 0x11fd6c770; frame = (0 0; 398 531); autoresize = LM+W+RM+TM+H+BM; layer = <CALayer: 0x1c2633020>> Native stack trace: 	0   CoreFoundation                      0x0000000183eb637c <redacted> + 148 	1   libobjc.A.dylib                     0x00000001830fc528 objc_exception_throw + 56 	2   CoreFoundation                      0x0000000183eb6238 <redacted> + 0 	3   Foundation                          0x00000001848737f4 <redacted> + 112 	4   UIKit                               0x000000018dd41328 <redacted> + 580 	5   UIKit                               0x000000018dd413e4 <redacted> + 160 	6   UIKit                               0x000000018de36d10 <redacted> + 132 	7   UIKit                               0x000000018de36e9c <redacted> + 368 	8   UIKit                               0x000000018de3a5c8 <redacted> + 1060 	9   UIKit                               0x000000018de36268 <redacted> + 100 	10  UIKit                               0x000000018d54c8f0 <redacted> + 464 	11  UIKit                               0x000000018d71b244 <redacted> + 300 	12  QuartzCore                          0x0000000187e03710 <redacted> + 672 	13  IOKit                               0x0000000184121190 IODispatchCalloutFromCFMessage + 392 	14  CoreFoundation                      0x0000000183e44290 <redacted> + 188 	15  CoreFoundation                      0x0000000183e5f000 <redacted> + 56 	16  CoreFoundation                      0x0000000183e5e704 <redacted> + 440 	17  CoreFoundation                      0x0000000183e5c1d8 <redacted> + 2196 	18  CoreFoundation                      0x0000000183d7be58 CFRunLoopRunSpecific + 436 	19  GraphicsServices                    0x0000000185c48f84 GSEventRunModal + 100 	20  UIKit                               0x000000018d41c8c8 UIApplicationMain + 236 	21  Flash.iOS                           0x00000001033151a0 wrapper_managed_to_native_UIKit_UIApplication_UIApplicationMain_int_string___intptr_intptr + 368 	22  Flash.iOS                           0x0000000102bf9bac UIKit_UIApplication_Main_string___intptr_intptr + 44 	23  Flash.iOS                           0x0000000102bf9a6c UIKit_UIApplication_Main_string___string_string + 172 	24  Flash.iOS                           0x000000010212b6ec Flash_iOS_Application_Main_string__ + 140 	25  Flash.iOS                           0x000000010266abd4 wrapper_runtime_invoke_object_runtime_invoke_dynamic_intptr_intptr_intptr_intptr + 244 	26  Flash.iOS                           0x00000001065568e0 mono_jit_runtime_invoke + 824


eps for GroupNorm

Great work!
The paramter 'eps' in group norm will be initialized to 1e-5 by default.
However, the group norm in TensorFlow has a little diference, which is initialized with 1e-6.
Maybe it doesn't  have any influence on training results, but can you just modify this(for all GroupNorm in code) for aligning?
Because I want to convert the trained model from torch or tf to megengine,  the less the error is, the better it is.

Thanks for your watching! The DDPM model was based on some Pytorch code implementation at first and I'm glad to hear that you are willing to convert original pre-trained model to MegEngine. Here are some information might be helpful:

I only checked that the forward process is consistent with the Pytorch version I referenced, but not sure if all details of the original Tensorflow version are implemented.
Other converted ckpts: https://github.com/pesser/pytorch_diffusion

In my opinion, converting scripts are also important for users understand how converted pre-trained models come from. So I sugguest you upload them in this repo, which could encourage more users join us.
Btw, I'm not sure yet how to develop this library in the future, I hope it will help more people understand the implementation of diffusion models. (OpenAI's improved/guided codebase is great, but lack of readability.)

During developing this repo, I write some notes in Chinese for myself to understand more about diffusion models. Here is a post: https://meg.chai.ac.cn/ddpm-megengine/ Welcome to read it and give me some advice.

I'm willing to upload my convert codes, but it doesn't work well after converting.
The error between megengine and pytorch implementation are high with the same input.
Because of the padding of convolution in Downsample are different, which in pytorch implementation it uses  asymmetric padding.
Atfter I modified the megengine implmetation, the result:
class DownSample(M.Module):
    """"A downsampling layer with an optional convolution.

    Args:
        in_ch: channels in the inputs and outputs.
        use_conv: if ``True``, apply convolution to do downsampling; otherwise use pooling.
    """""

    def __init__(self, in_ch, with_conv=True):
        super().__init__()
        self.with_conv = with_conv
        if with_conv:
            self.main = M.Conv2d(in_ch, in_ch, 3, stride=2)
        else:
            self.main = M.AvgPool2d(2, stride=2)

    def _initialize(self):
        for module in self.modules():
            if isinstance(module, M.Conv2d):
                init.xavier_uniform_(module.weight)
                init.zeros_(module.bias)

    def forward(self, x, temb):  # add unused temb param here just for convince
        if self.with_conv:
            x = F.nn.pad(x, [*[(0, 0)
                         for i in range(x.ndim - 2)], (0, 1), (0, 1)])
        return self.main(x)

Btw, I'm also a beginner in ddpm, your blog helps me a lot!

Got it. I'm not available at the moment and  I will check the padding mode and #6 after day off.

The initial eps value has been updated and I will close this issue now to keep tracking the same thing in one issue.
Feel free to reopen it if you have any questions or suggestion.


Ray Tracing Extended

Description
Render a simple scene showing the basics of ray tracing, including Ambient Occlusion, top and bottom level acceleration structure rebuilding for animation, multiple objects in a scene, and refraction. The sample loads the sponza gltf model, loads the base fire particle texture and creates geometry for one particle and one quad showing heat above fire then creates a bottom acceleration structure for each geometry type, then makes instances of those static geometries and multiple instances of the fire particle to achieve a particle effect.  It then demonstrates TLAS (Top Level Acceleration Structure) animation via updating the fire particle(s) and BLAS (Bottom Level Acceleration Structure) animation via updating the quad geometry.
Tested Environments

Windows 10 / Ubuntu Linux
GeForce 3060 / GeForce 2080 / GeForce 1070
Vulkan version: 1.2.176

Sample Assets changes
depends upon flame texture found in https://github.com/KhronosGroup/Vulkan-Samples-Assets/pull/12
General Checklist:
Please ensure the following points are checked:

[x] My code follows the coding style
[x] I have reviewed file licenses
[x] I have updated the header Copyright to reflect the current year (CI build will fail if Copyright is out of date)
[x] I have commented any added functions (in line with Doxygen)
[x] I have commented any code that could be hard to understand
[x] My changes do not add any new compiler warnings
[x] My changes do not add any new validation layer errors or warnings
[x] I have used existing framework/helper functions where possible
[ ] My changes build on Windows, Linux, macOS and Android. Otherwise I have documented any exceptions
[x] My changes do not add any regressions
[x] I have tested every sample to ensure everything runs correctly
[x] This PR describes the scope and expected impact of the changes I am making

Sample Checklist
If your PR contains a new or modified sample, these further checks must be carried out in addition to the General Checklist:

[x] I have tested the sample on at least one compliant Vulkan implementation
[ ] If the sample is vendor-specific, I have tagged it appropriately
[x] I have stated on what implementation the sample has been tested so that others can test on different implementations and platforms
[x] Any dependent assets have been merged and published in downstream modules
[x] For new samples, I have added a paragraph with a summary to the appropriate chapter in the samples readme

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: gpx1000:x: wflohryYou have signed the CLA already but the status is still pending? Let us recheck it.

Great looking sample. Runs fine on my RTX 2060.
One thing I noticed, is that the heat haze effect doesn't look correct when viewed from different angles. E.g. if you watch the fire from the other side, it looks like this:

It would also be great of alpha masking was applied on alpha masked materials:

Afair this would require use of an additional any hit shader.

Thanks @SaschaWillems
Good observations on refraction.  It kinda looks like the image is getting flipped on the screen shot.
For the alpha, I thought about the benefits of adding it, but worry about making the sample too complex to serve the purpose of highlighting AO, animation, and shadows.  I think we also need to update the textures the sponza model is using as I don't think it currently has the alpha mask; looking at the ktx files it seems none of them have the alpha baked in either.

Refraction now looks fine from all angles.
I'm also more than fine with not having alpha, esp. with the model needing to be updated.
So this looks good to me now :)

I tried it with a NVIDIA GeForce RTX 3060Ti and it works.

Should there be a user interface to change the drawing mode? It does not show up in my test.
Is it possible that there is a problem with the animation update? The buffers are updated with map/unmap, but I don't see how that protects against updating the buffer while drawing, unless I'm missing a wait/fence somewhere.
Should the animation be created with VK_BUILD_ACCELERATION_STRUCTURE_ALLOW_UPDATE_BIT_KHR to update it?

The BLAS update could then set both srcAccelerationStructure and dstAccelerationStructure to the same acceleration structure when it is recreated.

Wouldn't it be a good idea to have the animation executed with a compute shader?

I tried it with a NVIDIA GeForce RTX 3060Ti and it works.

Should there be a user interface to change the drawing mode? It does not show up in my test.

Is it possible that there is a problem with the animation update? The buffers are updated with map/unmap, but I don't see how that protects against updating the buffer while drawing, unless I'm missing a wait/fence somewhere.

Should the animation be created with VK_BUILD_ACCELERATION_STRUCTURE_ALLOW_UPDATE_BIT_KHR to update it?

The BLAS update could then set both srcAccelerationStructure and dstAccelerationStructure to the same acceleration structure when it is recreated.

Wouldn't it be a good idea to have the animation executed with a compute shader?

Thanks for reviewing @mklefrancois.

Until the framework supports GUI use in Ray Tracing pipelines #334 addresses that shortcoming.  We've removed the GUI all together.
fixed
fixed
fixed
While I agree it's a good optimization to do all animation in a compute shader, I worry that it would add complexity to the sample in an area that doesn't highlight the general case of how to do animation with ray tracing pipelines.  Thus, while it's an excellent idea to do animation in a compute shader, it should probably be beyond the scope.

@aurolou, can we get your help to review this RT sample as well as PR #317?  These are ray tracing samples and we'd really like to make sure they support AMD before releasing them.  Thanks!


Document planned release cadence for arrow/parquet/etc

Which issue does this PR close?
Related to https://github.com/apache/arrow-rs/issues/5368
Rationale for this change
We spent a while discussing release cadence in https://github.com/apache/arrow-rs/issues/5368 so let's document that clearly on in the readmes
What changes are included in this PR?
Document the planed release cadence.
Add links in parquet / arrow-rs readme
Are there any user-facing changes?
Documentation, no functional changes

Do we have a plan to object_store? I'm guessing a breaking change to object_store could also affect parquet?

Do we have a plan to object_store? I'm guessing a breaking change to object_store could also affect parquet?

Nothing formal, but a natural implication of this would be that if we made a breaking release to object_store we would be unable to incorporate this into arrow until the next arrow breaking release. In practice this means object_store will likely follow a similar cadence for breaking releases, if not slower.

I plan to add a note about object_store release schedules, likely as a follow on PR

I am working on a follow up PR for object_store cadence


widgets in directories that contain subfolders?

Is it there any way to use widgets in directories that contain subfolders with posts?(Using the terminology of https://gohugo.io/content-management/page-bundles/, I want to use a widget in a branch folder that contains other branch or leaf folders).
For example, in content/d1 I have an index.md as follows
+++
title = "a title"
date = 2017-01-01T00:00:00
summary = "" # Add a page description.
type = "widget_page"  # Page type is a Widget Page
+++

and I have copied, or symlinked, the about.md widget under d1.
Under d1 there is a directory d2 with one or more posts, e.g., content/d1/d2/post_d2.md (or content/d1/d2/index.md).
The problem I am facing is that when I go to http://baseurl/d1 I see the output generated from post_d2.md and about.md concatenated. I am unable to get ONLY the output from about.md.  (http://baseurl/d1/d2/post_d2 works fine).
Note: this is not the same as https://github.com/gcushen/hugo-academic/issues/1032. And yes, I could place d2 not under d1 and use url on post_d2.md to modify the URL. But I want to keep d2 under d1 for other reasons (URLs generated for menu entries).

Widget pages are leaf bundles in Hugo. The Hugo docs appear to state that no bundles can be nested under widget pages (leaf bundles).

Thanks! What I want to do, then, is not doable via widgets (e.g., https://discourse.gohugo.io/t/hugo-leaf-and-branch-bundles/10892/21, https://gohugo.io/content-management/page-bundles/).
May I suggest that this could be more clear in the documentation? Two issues in particular: that widget pages are leaf bundles and that, therefore, using widget pages precludes a directory from containing subdirectories with their own content.
To give a specific example of what I want to accomplish:

That http://site/ and http://site/d1 show the same "Home page" information. This is easy using widgets.

That posts leave under d1, so that http://site/d1/posts can be an entry in the navigation bar. (i.e., that menus.toml have an entry as

[[main]]
  name = "posts"
  url = "d1/posts/"


python release type not able to find correct python packages when using poetry

Is your feature request related to a problem? Please describe.
When using poetry and package name contains dash, python release type not able to find correct version files
Describe the solution you'd like
Have a better guess by replacing dash with an underscore.
For example, I have a pyproject.toml
[tool]

[tool.poetry]
name = "my-service-client"
version = "0.1.0"
description = ""

Release please warn me
file my-service-client/__init__.py did not exist

while is file path is actually my_service_client/__init__.py
Describe alternatives you've considered
Package name and extra files should be able to bump the file correctly, however guessing the package name better is always something nice to have.

In addition it's related to issue https://github.com/google-github-actions/release-please-action/issues/396.

@trallnag I worked around with extra-files.
However you can checkout pr #1418. It is nearly finished except the include from is wrong.

Yes I saw #1418. I tried a somewhat different approach in #1526. There I simply add variations with underscores instead of dashes to the Python file updater.

#1525


created contact page

Please Provide The following Details About your PR
Type x Inside the checkbox to tick it
Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes #25
Type of change
Please delete options that are not relevant.

[x ] New feature (non-breaking change which adds functionality)

How Has This Been Tested?
I have attached the screenshot.

Checklist:

[x ] My code follows the style guidelines of this project
[x ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules
[ ] I have checked my code and corrected any misspellings

@musavveer try to use classes and ids for css instead of targeting elements!

@musavveer  Don't Forget to Star the Repository!

@musavveer Don't Forget to Star the Repository!

Do include hacktoberfest accepted label .>>>

@musavveer Don't Forget to Star the Repository!

Do include hacktoberfest accepted label .>>>

it is not required all the PRs to repo are included for hacktoberfest


Closes Issue#31

[Problem] chat/views.py don't have any code comments to have better redability of the code
[Solution] Added necessary single line comments inside functions and multiline commnets in starting of each function to describe what the function is doing

Kindly test before making a PR, there was a syntax error on line 82.


Game Tutorial

Game tutorial
The purpose of this ticket is to implement a user tutorial component within the game to provide players with essential information and guidance on how to play the game effectively. The user tutorial will be designed to enhance the overall gaming experience by helping players understand game mechanics, controls, and objectives. It will appear as a pop-up window that appears at the beginning of the game or can be accessed from the game's main menu. The tutorial will cover various aspects of gameplay, including movement controls, placing structures (e.g., walls, turrets), resource management, and other important gameplay elements.
SubFeature/component (optional)
This ticket includes the following components:

Making Wiki for the user tutorial.
Designing the user tutorial interface.
Creating tutorial content and text.
Implementing triggers to display the tutorial at the appropriate times.

Examples (optional)
Movement Controls: The tutorial will explain how to move the character within the game using keyboard keys (e.g., W for up, A for left, S for down, and D for right).
Placing Structures: Players will be guided on how to place structures like walls and turrets to defend their base or navigate the game world.
Resource Management: The tutorial will introduce players to resource gathering and management mechanics, ensuring they understand how to collect and use resources effectively.
Milestones

[x] Goal 1: #325
[ ] Goal 2: #326
[ ] Goal 3: #327
[ ] Goal 4: #319
[ ] Goal 5: Documentation, Testing and bug fixes

Members
Khushi @khushijain0302 - Khushi Jain
Raghav @NotRaghavAggarwal - Raghav Aggarwal

Delayed until next sprint

Scrapped for this sprint


is the accumulation of gradient done right, where do we divide the accumulator by number of tasks?

I was reading the maml example and saw this:
https://github.com/facebookresearch/higher/blob/941ae9f310994e728064f8edd7f17c57550e7c67/examples/maml-omniglot.py#L165
which seems right locally (since it is accumulating gradients). But then it never divides by num_tasks. I think that might actually matter. Usually I believe in normal supervised learning it is not a big deal because then the step size controls that quantity via lr * B where B is the batch size. But here the situation might be more subtle because there is correlation between the tasks and thus the standard deviation of that estimate might be different (need to check the details for this). But I wanted to draw attention to it since it might be a nuance issue in the MAML example.

In practice, you are right that you might want to divide through by the meta-minibatch size (number of tasks) if you want to interpret it as a Monte Carlo estimate of the batch meta-gradient. However, for a fixed meta-minibatch size, this doesn't really matter. The normalization term just implicitly becomes part of the learning rate.

Sorry, to answer your second question from the title, if you want to do this, "right", replace
qry_loss.backward()

with
(qry_loss / num_tasks).backward()

I actually think it does matter for meta-learning but not for regular learning. Ran out of time to make the maths formal I will tomorrow.

@egrefen got it. They are indeed not the same even in the standard machine learning with no meta-learning. Using the sum has the same mean as the average if you choose the right step size but, they do not have the same standard deviation (that cannot be fixed by any stepsize since a step size cannot be equal to the size of the batch and the square root of the same time, this indeed assumes the gradients are normally distributed which might not be always true but I have some work that shows this tends to be true). In practice, the sum one might be better (due to the different std) but that is mainly an empirical question.
With respect to meta-learning, I have not completed the analysis but things are usually worse there because the tasks are correlated (in few shot learning like mini-imagenet the issue is even worse!), so the iid assumption doesn't hold when sampling a tasks (i.e. sampling the meta-batch). The intuition is that the std noise might be even higher since the covariance is going to be included. I empirically noticed this when my estimates for the meta-test were really really noisy for some reason. I recommend dividing by num_tasks to avoid extra noisy gradients. Thus your comment:

However, for a fixed meta-minibatch size, this doesn't really matter. The normalization term just implicitly becomes part of the learning rate.

requires more precision and is more nuanced. It's even subtle for normal supervised learning.


IP Clearance

All files developed at the ASF need to have an ASF header [1], 3rd party headers for the most part need to be retained [2]

https://www.apache.org/legal/src-headers.html#headers
https://www.apache.org/legal/src-headers.html#3party

License clearing wiki page (with draft process and tools): https://cwiki.apache.org/confluence/display/NUTTX/License+Clearing

Add the related email thread:
https://lists.apache.org/thread.html/r0d30d8c95e861826a3027499fc43bc3851e19f89fdaf8606eada1818%40<dev.nuttx.apache.org>
https://lists.apache.org/thread.html/r3149c844791bd0164a3016cbebc690edd9277905678cfb33526937cb%40<dev.nuttx.apache.org>
https://lists.apache.org/thread.html/r897f825f1bfcd3501c132438acc9403a70d415652119d1e528f7349f%40<dev.nuttx.apache.org>

@adamfeuer do you have enough free time to collect the statistics inforamtion? My team leader reserve a dedicated resource help you to improve the tools and generate the report. @PeterBee97.

Thanks @xiaoxiang781216 – I should have enough time to do a high-level analysis this week or next, and I could definitely use the help!
@PeterBee97 are you able to help me do this? If so, reply here or send me an email (it's on my profile), and we'll work out what to do. 🙂

@adamfeuer Hi Adam, sure I'm here to help. BTW I spent some time yesterday on a script that doesn't modify anything yet but only tries to extract information. Hope this helps :)

@PeterBee97 Great work with the script and database! I'll update my tools branch and post it here– would you be willing to do a PR to that, so we can have a single branch that we're working on? I'm hoping we can merge these tools to master so that others can help us or continue our work.
Here's a few questions:

Are you subscribed to the<EMAIL_ADDRESS>email list? If not, would you be willing to subscribe?
What's your email address? Will you either post it here, send me an email at<EMAIL_ADDRESS>So we can correspond with the NuttX email list if necessary.
What time zone are you in?
Have you seen the NuttX license clearing wiki page? The process we need to follow and improve is there, as well as a few tools.
The authors in the file are good to have, but not enough to clear the licenses– we need to look at the git log and get authors from that. There's a script on the wiki page above that can do that.
Would you be willing to make the script you wrote also emit a plain text file, ideally tab delimited CSV?

@PeterBee97 I updated my license-clearing tools branch to upstream/master, here's where I've put my tools: https://github.com/starcat-io/incubator-nuttx/tree/feature/license-clearing-tools/tools/license-clearing

@PeterBee97 Let's try running the process that we did on the sched/ module on either fs/ or mm/– only the estimation part, not the whole clearing process. They have 100-250 files each, so it's a smaller chunk. We need git authors as well a what is in the file headers. Once we have a way to get stats for that module and all files, then we can try to do it for the whole project.

@PeterBee97 Great work with the script and database! I'll update my tools branch and post it here– would you be willing to do a PR to that, so we can have a single branch that we're working on? I'm hoping we can merge these tools to master so that others can help us or continue our work.
Here's a few questions:

Are you subscribed to the<EMAIL_ADDRESS>email list? If not, would you be willing to subscribe?
What's your email address? Will you either post it here, send me an email at<EMAIL_ADDRESS>So we can correspond with the NuttX email list if necessary.
What time zone are you in? I am in Seattle WA USA, Pacific Time Zone, UTC-7.
Have you seen the NuttX license clearing wiki page? The process we need to follow and improve is there, as well as a few tools.
The authors in the file are good to have, but not enough to clear the licenses– we need to look at the git log and get authors from that. There's a script on the wiki page above that can do that.
Would you be willing to make the script you wrote also emit a plain text file, ideally tab delimited CSV?

Not yet, sure I'm willing to subscribe
<EMAIL_ADDRESS>I'm in Beijing, UTC+8 so my work time will be about 7 pm to 7 am in your timezone :(
Yes, I browsed through the docs and mailing lists before making that tool
Yeah, actually my tool is based on your script. The author0~author2 are from git log
Sure, exporting to csv file is just one command in sqlite

@PeterBee97 Let's try running the process that we did on the sched/ module on either fs/ or mm/– only the estimation part, not the whole clearing process. They have 100-250 files each, so it's a smaller chunk. We need git authors as well a what is in the file headers. Once we have a way to get stats for that module and all files, then we can try to do it for the whole project.
You can see what we did on sched/ at this wiki subpage: https://cwiki.apache.org/confluence/display/NUTTX/Analysis+March+2020

By typing sched/ in the DB Browser filter I can see that these files either have apache license already or only owe copyrights to Greg or Xiaomi & Pinecone, which should already approved the license change. I strongly recommend the DB Browser GUI as it's really straightforward :)

@PeterBee97 Cool, thanks– I didn't realize the script already used git to find the authors, sorry for missing that. We will need all the authors, not just the top 3. I'll take a closer look tomorrow.
Re: Xiaomi and Pinecone already approving the license change, do you know if they have filed an Apache Software Grant Agreement (SGA)?
Would you be willing to run your tool on fs and mm directories, and see if you can extract a report of the authors for each section and file? That way we can see if we're dealing with 10 authors, 100 authors, etc.
I think another next step is to get you an account on the NuttX Fossology instance. At some point we'll need to get the data into there. I'll email Brennan and you on the list.
Thanks again for being willing to help with this!

@PeterBee97 Cool, thanks– I didn't realize the script already used git to find the authors, sorry for missing that. We will need all the authors, not just the top 3. I'll take a closer look tomorrow.
Re: Xiaomi and Pinecone already approving the license change, do you know if they have filed an Apache Software Grant Agreement (SGA)?
Would you be willing to run your tool on fs and mm directories, and see if you can extract a report of the authors for each section and file? That way we can see if we're dealing with 10 authors, 100 authors, etc.
I think another next step is to get you an account on the NuttX Fossology instance. At some point we'll need to get the data into there. I'll email Brennan and you on the list.
Thanks again for being willing to help with this!

Top 3 was my idea, given that some 1 commit contributors can be ignored(can't they?). For license issue I don't know exactly the details, @xiaoxiang781216 knows better. I ran the tool on the whole proj already so those two directories can just be filtered. I'll try to get a report for particular files.

@PeterBee97 https://github.com/PeterBee97 Cool, thanks??? I didn't
realize the script already used git to find the authors, sorry for
missing that. We will need all the authors, not just the top 3. I'll
take a closer look tomorrow.

I mentioned this before, but it bears repeating.?? The NuttX project was
be 13 years old in February of 2010.?? For the first 6 to 6 and a half
years, the project used CVS and SVN.?? You will find no authorship or
contact information for the first half of the project's life in the
current GIT authors.?? The log will show me as the sole author for during
that time.
I did by far most the changes in those days, but not all.?? Prior to
GIT,?? contributors were noted only in commit comments.?? It should be
possible to get the names, or in most cases just user handles, from the
comments but with no contact information.
Github apparently does not even know how to parse that early activity.??
If you look at
https://github.com/apache/incubator-nuttx/graphs/contributors you would
conclude that the project has only existed since sometime in 2013.?? The
project was actually created in February of 2007.?? This is clearer in
the Bitbucket statistics[1]:
https://bitbucket.org/nuttx/nuttx/addon/bitbucket-graphs/graphs-repo-page#!graph=contributors&uuid=4430abf9-a782-49ff-bd16-bc1df696048e&type=c&group=weeks
which goes all the way back to the day the project was created.
I think that is because prior to GIT, authors were NOT referenced by
email address, but rather with some UUID.
[1]Note you have to be logged into Bitbucket to see the statistics there.

@patacongo Are the original CVS and SVN archives saved anywhere?

@patacongo Are the original CVS and SVN archives saved anywhere?

No

@patacongo Ok. I'll see if I can look through the commit message to see if I can see what's going on there.
I'm logged in to Bitbucket, but for some reason I can't view the graph link you posted. Maybe it's a permissions issue or I don't have access to the graphs addon?

@PeterBee97 https://github.com/PeterBee97 Cool, thanks??? I didn't realize the script already used git to find the authors, sorry for missing that. We will need all the authors, not just the top 3. I'll take a closer look tomorrow.
I mentioned this before, but it bears repeating. The NuttX project was 13 years old in February of 2010. For the first 6 to 6 and a half years, the project used CVS and SVN. You will find no authorship or contact information for the first half of the project's life in the current GIT authors. The log will show me as the sole author for during that time. I did by far most the changes in those days, but not all. Prior to GIT, contributors were noted only in commit comments. It should be possible to get the names, or in most cases just user handles, from the comments but with no contact information. Github apparently does not even know how to parse that early activity. If you look at https://github.com/apache/incubator-nuttx/graphs/contributors you would conclude that the project has only existed since sometime in 2013. The project was actually created in February of 2007. This is clearer in the Bitbucket statistics[1]: https://bitbucket.org/nuttx/nuttx/addon/bitbucket-graphs/graphs-repo-page#!graph=contributors&uuid=4430abf9-a782-49ff-bd16-bc1df696048e&type=c&group=weeks which goes all the way back to the day the project was created. I think that is because prior to GIT, authors were NOT referenced by email address, but rather with some UUID. [1]Note you have to be logged into Bitbucket to see the statistics there.

@PeterBee97 can we add a column in the database to indicate the source code exist before git is used? @patacongo, we need gather the statistics information first and convert the unambiguous code base automatically(of course we need review the PR carefully) and then work on the rest case by case, otherwise NuttX can never become the TOP LEVEL PROJECT.

@xiaoxiang781216 @patacongo @PeterBee97 I cloned the Bitbucket repo last night (https://bitbucket.org/nuttx/nuttx/src/master/), looked through the commit logs, and I can see what @patacongo is talking about. I didn't compare to the github log, but we should probably also do that. Then we can see if we can do anything with the information there.
It seems like we should be able to come up with a strategy for dealing with this:

If we can get names and contact info from the commit messages, then we can run the license clearing process we already have, maybe with some additional steps about that process.
At the very least, we can collect statistics about how many contributors we are talking about.
If we can't get names and contact info from the commit messages, then we need to get help to address what @xiaoxiang781216 is talking about, so NuttX can graduate from podling status. Surely other Apache projects have faced this same issue.

Let me know if you have other thoughts about this.
@PeterBee97 Will you clone the Bitbucket repo and look at the logs to see if you have some insight about it?

This is also informative:
git log | grep author
The will produce over 30 thousand lines but you clearly see that the last several thousand commits have author:
patacongo patacongo@42af7a65-404d-4744-a932-0658087f49c3
That, I think is a bogus email that was created when the SVN repository was converted to GIT.
Then there are several thousand with author:
Gregory Nutt<EMAIL_ADDRESS>That is GIT, but when I was still using GIT as though it were SVN with no authors.
The first author that is not me appears at:
commit b0507038494cd1ae9d14807db758d4e3ae98a1ef
Author: jeditekunum<EMAIL_ADDRESS>Date:   Sat Jan 24 14:31:35 2015 -0600

First step at porting to MoteinoMEGA.  LED shows assert failure at boot.  Appears to be short double blink, short off (~1sec), followed by 250ms toggle cycles.  Most of it derived from amber board.

So it appears that there is authorship information for the first 8 years.  Only for the last 5 years.

@patacongo @PeterBee97 If I search for ' by ' I find commits like this:
commit f03cb0ff3ababdcc84245d75d795ab956d110e09
Author: patacongo <patacongo@42af7a65-404d-4744-a932-0658087f49c3>
Date:   Tue Mar 16 00:53:32 2010 +0000

    Bugfixes submitted by David Hewson

    git-svn-id: svn://svn.code.sf.net/p/nuttx/code/trunk@2543 42af7a65-404d-4744-a932-0658087f49c3

There are others. They seem to indicate patches or other code from contributors, committed by Greg.

@patacongo Thanks for pointing this out again, I am sorry I didn't remember this.

Bugfixes submitted by David Hewson

David Hewson I know.  We are connected on LinkedIn.  He just started working for HPE. He did a some of the LPC31 port in the 2010 timeframe but has not been involved significantly since.

If do git log --reverse and search for ' by ' I find commits like this

"by" or "from" would both be good search keys.  I also recorded the authors in the old ChangeLog files that were recently removed from the repositories because they are not used in the current workflow.  That should be a complete list of authors expect for a few trivial things like typo fixes that weren't normally included in the ChangeLog.

@PeterBee97 Will you clone the Bitbucket repo and look at the logs to see if you have some insight about it?

I cloned the bitbucket repo today but the git log seems to be the same with that on GitHub... So I found the latest ChangeLog from NuttX 9.0.0 RC0 and tried to filter out the names with keywords from|by and the help of some NLP library and put the results in names-changelog.txt. Also processed the git log in the same way and the result is names-gitlog.txt. Still the commit messages of earlier SVN commits are incomplete and many commits are authorless. This may help cover some corner cases. Maybe we can open an issue and mention these users? But before that let's filter out the "safe" files first as @xiaoxiang781216 suggests.

@PeterBee97 That's great! Less than 450 names in each file. The next steps are probably:

remove all the non-human-names (Atmel, CONFIG_SDIO_PREFLIGHT, etc.)
remove all the name of committers (they have ICLAs) - I manually made a list of committers
remove duplicates (may need to be done manually since there are typos in the names
merge the lists

Once this is done, it will give us a scope of how many people there are. Ideally we'd have a list of commits for each name, and only to the top N contributors... not sure what N should be, but looking at the data should tell us.  Do you have an idea how to get a list of commits per name?

I cloned the bitbucket repo today but the git log seems to be the same with that on GitHub...

Yes, the Bitbucket repositories are read-only mirrors of the incubator repositories.

* I manually made a [list of committers](https://github.com/starcat-io/incubator-nuttx/blob/feature/license-clearing-tools/tools/license-clearing/committers.txt)

A large number of people do not use there names on PRs or commits, but rather some username/handle.  A few of these I know.  For example, v01d is Matias Nitshe, raidenpl is Mateusz Szafoni.  But there are many more that I don't know.

@patacongo Yes– we should find a way to update the committer list and the contributor list with handles... I'll think of some ways to do that...

@PeterBee97 That's great! Less than 450 names in each file. The next steps are probably:

remove all the non-human-names (Atmel, CONFIG_SDIO_PREFLIGHT, etc.)
remove all the name of committers (they have ICLAs) - I manually made a list of committers
remove duplicates (may need to be done manually since there are typos in the names
merge the lists

Once this is done, it will give us a scope of how many people there are. Ideally we'd have a list of commits for each name, and only to the top N contributors... not sure what N should be, but looking at the data should tell us. Do you have an idea how to get a list of commits per name?

I used this script to get the list from git log and earlier commits by @patacongo :
git log --no-merges --author=patacongo --pretty=format:"%h %s" > gp.txt
cat ng2.txt | xargs -n 1 -I pp grep "pp" gp.txt > commits-patacongo.txt
./name-commits.sh ng2.txt name-commits.txt commits-patacongo.txt

Result:(I did some manual deduplication but didn't exclude enlisted committers yet)
https://github.com/PeterBee97/authors-tool/blob/master/name-commits.txt
The names with no commits may be issue reporters' names, or names of committers who only contributed to the apps repo (I only ran the above commands in nuttx repo). Also some names are mentioned in ChangeLog, but sadly there's no commit
authored by or mentioning them.

Any updates here? I think this is only blocker issue to prevert us graduate, let's try to make progress.
Thanks.

@Apache9 No progress since the last update, I've been busy with other things. I'll merge @PeterBee97's code today. Next we should generate a list of people and the total lines of code for each person. Then we could sort in reverse order and decide how many people we need to try to contact.
@PeterBee97 Can you help with this? Can you find out how many lines of code were in each commit, tie them to a person in our list, create a list that combines all lines of code for each person, and create a CSV sorted in reverse order by total lines of code contributed?

@adamfeuer how about we convert the source code which satisfy:
1.The first commit come from git not svn or cvs
2.The copyright owner in source code already sign SGA or ICLA
3.All contributor from git log already sign SGA or ICLA

@xiaoxiang781216 That would be a good first step for the conversion process. But as discussed on the mailing list, I thought we first wanted to do a rough total estimate of the entire project?
If we want to do both in parallel, then I think your idea will be a good start. We would need:

list of all contributors who have signed SGA or ICLA - right now we only have committers who I presume have signed ICLAs. I don't know how to get the complete list, do you?
list of all files for which

only ICLA committers are in the git log
first commit is not from svn or cvs

My recollection is not 100% clear, but I am recalling that @justinmclean mentioned in very early phases of this project that there were some legacy changes that could be just grandfathered in without following the full IP clearance process.  I understood that this was necessary for other large, established Incubator projects as well.
If my understanding is correct, then I propose that we take get permission to "cut some corners" on the pre-GIT changes that have no author associated with the individual commits.  In most cases, the author of those early changes will be noted as an author or copyright holder in the BSD license header.  In fact, I think that is true of all significant early code contributions.  I would propose that we only use the GIT author changes for any automated analysis.
pre-GIT means pre-2014 so we are referring to very old changes.
Resolution of any remaining issues in the license headers will have to be a largely manual process anyway.  We will have to examine each BSD license header and resolve all authors and copyright claims anyway.  So I think with my suggestion here, the job can be made doable and there will be no loss of authorship on any significant contributions.

In most cases, the author of those early changes will be noted as an author or copyright holder in the BSD license header. In fact, I think that is true of all significant early code contributions.

I can think of one very frequent case where this is not true.  In many cases, people clone files from one location to another.  This is particularly true under arch/ and boards/.  You will discover many files that I wrote, that have me as the copyright holder and author but GIT will claim, incorrectly, that the person doing the PR/patch was the author.  This will apply to several hundred files.  There are cases where the info in the license header is more accurate than in the file header.
Third party code brought into the OS will have the same issue.  The true author of the code is in the license header, not in the GIT log.
And there are places where people make mistakes in copying files without updating the license headers.  For example, under net/  there are a few files that include some small bits of logic from Adam Dunkels.  I see that those files with headers have been cloned numerous times and most are no longer correct.  Adam Dunkels is not the author of any of the files under net/ (except perhaps some logic under net/sixlowpan and the TCP state machine and even those are very highly customized).
It is all very complex and we cannot expect to get it all 100% correct.  I think we just have to keep a high level of integrity and do our best effort to discover and document all authorship.
I think the point is that GIT authors may not agree with the authorship in the license header and those will all need some clarification.

@xiaoxiang781216 @patacongo I updated my comment above to include "file's author headers match git author or author listed in git commit message" – that handles the cases where things would match up easily.
 I think we just need to get a count of how many there are to see what it will take to track down the ones that matter.

Any updates here? I think this is only blocker issue to prevert us
graduate, let's try to make progress.

It seems to me that there are people who have interest and good ideas
but there is not significant progress being made.  The job is really two
large for a couple of people to accomplish working now and then.

Could we start with the easy cases? I feel that reducing the size of the problem also makes it less intimidating to approach.
We are already manually changing headers from BSD to apache for files whose authors are commiters with ICLAs so I think making an automated pass for this case should not be that hard: parse header for authors, see if all are commiters, replace with apache header. If that sounds right I can script that and give it a try.
What confuses me though is that we're worrying about git authors whereas I believe that if someone contributes a file without listing themselves as the authors in the header (for the BSD case), didn't the author concede rights over the code by doing so? At least that was my understanding at the time when I submitted patches to existing files and I did not include an extra line to add me as author to every affected file. In case this is not the correct assumption, I agree that a "best effort" approach (by comparing git author to authors on header) is the only remaining possibility.

Hi

What confuses me though is that we're worrying about git authors whereas I believe that if someone contributes a file without listing themselves as the authors in the header (for the BSD case), didn't the author concede rights over the code by doing so?

Without an ICLA (or an equivalent) this is not the case. Copyright automatically applies. They may not even own rights to the code they commit if their employment contract says otherwise.
Thanks,
Justin

Hi,
BTW Apache doesn’t use author tags in any new code, doing so implies ownership by a person rather than the whole project.
Thanks,
Justin

Hi
What confuses me though is that we're worrying about git authors whereas I believe that if someone contributes a file without listing themselves as the authors in the header (for the BSD case), didn't the author concede rights over the code by doing so?
Without an ICLA (or an equivalent) this is not the case. Copyright automatically applies. They may not even own rights to the code they commit if their employment contract says otherwise. Thanks, Justin

So @justinmclean is it safe we do the batch conversion if the source code meet all following critieria?
1.The source code isn't converted from SVN or CVS
2.All commiters(or his company) in git log sign ICLA or SGA
3.The copyright holder in the source code sign ICLA or SGA

Hi,

So @justinmclean https://github.com/justinmclean is it safe we do the batch conversion if the source code meet all following critieria?
1.The source code isn't converted from SVN or CVS

I’m not sure what you mean by that.

2.All commiters(or his company) in git log sign ICLA or SGA

Small contributions don’t have to have a CLA, but the person who committed that contribution takes responsibility for ensuring teh code’s IP. If possible it's best to have one.

3.The copyright holder in the source code sign ICLA or SGA

Take care with this. The copyright holder in source may or may not be the correct one.
Thanks,
Justin

Take care with this. The copyright holder in source may or may not be the correct one.

Similarly, the author in GIT may not be the author of the file.  Often the copyright holder in the source file header is the correct one, even though that person many not appear in GIT history.
Many people copy files that wrote into different locations (very often for new architectures and for new boards which are very similar to older architectures and boards).  Very often, I am the author of the file in these cases.
Bottom line:  There is no magic, automated way to correct determine the author.  It requires collecting data and then also applying human insight.
@justinmclean https://github.com/justinmclean For many cases there are multiple contributors of changes to a file.  There is an original author, the original committer (who might be a different person) and people who have made trivial changes (as trivial as a spelling fix) or who have made substantial enhancements or re-designs.  The former would not be treated as authors or copyright holders, the the latter may be.  Is there any rule of thumb for what constitutes a significant change warranting rights to the file?  Or does this also require human insight.
There are thousands of files involved here.  This is potentially multiple man years of effort.  I don't see how we can ever accomplish this.

We can only operate on the information we have. If authorship information was lost from CVS and SVN era (git author is Greg) and the header does not list anyone else than Greg, we can either "play safe" and leave the BSD header (we would respecting original authors license even if we don't know who it really was) or assume that without further information the original author cannot prove authorship either then we are safe to change to Apache. For these "unknown" cases, I don't see any other way. We just need to decide and then act.
For other cases where there is indeed information I think we can script a header change based on various scenarios of git author/header author/author aliases where all have ICLAs. This change can be made to create one commit per file change and add the reason for the safety of the change to the commit message for traceability. Then, we can review each commit in a PR and decide if manual intervention is needed (throwing out unsafe changes, for example).

We can only operate on the information we have. If authorship information was lost from CVS and SVN era (git author is Greg) and the header does not list anyone else than Greg, we can either "play safe" and leave the BSD header (we would respecting original authors license even if we don't know who it really was) or assume that without further information the original author cannot prove authorship either then we are safe to change to Apache. For these "unknown" cases, I don't see any other way. We just need to decide and then act.

In the SVN/CVS days, I did always give credit to the contributor in comments.  However, the task of reading all comments in those 15 thousand or so commits is a very onerous task.  The information is there, just not easily accessible.
AFAIK there are no un-credited changes in the repositories.

We can try to see what wording you used in general and use some regular expression to try to match the attribution.
What I'm thinking is that in any case we will always need to analyze a file by looking at its complete git history to extract git author + header author + commit msg attribution right? The "easy" cases would then be files only touched by current commiters.

Hi,

@justinmclean https://github.com/justinmclean https://github.com/justinmclean https://github.com/justinmclean For many cases there are multiple contributors of changes to a file. There is an original author, the original committer (who might be a different person) and people who have made trivial changes (as trivial as a spelling fix) or who have made substantial enhancements or re-designs.

Ideally we wold have CLAs for those who have made significant changes or who owned the IP on the original contribution, whose owner may or may not be the author.

There are thousands of files involved here. This is potentially multiple man years of effort. I don't see how we can ever accomplish this.

I would try solving for the low hanging fruit e.g files you know that only people who currently have CLA have contributed to and work from there and change the licenses to ALv2. I think this has already been suggested. Other code is under a compatible license so that’s the fallback position.
Thanks,
Justin

Let's clear the license for the files we own first. I think it is OK to have some files under compatibile licenses for a ASF project. You just need to mention them in the NOTICE file. And there is another possible solution is to rewrite these files so we can change the license. Anyway, this depends on the number of files we can not change license.
Thanks.

Let's clear the license for the files we own first. I think it is OK
to have some files under compatibile licenses for a ASF project. You
just need to mention them in the NOTICE file. And there is another
possible solution is to rewrite these files so we can change the
license. Anyway, this depends on the number of files we can not change
license.

I don't think anyone has committed to do that work.  Adam and Peter
have, I guess, but they don't apparently have the bandwidth required to
do that effectively.
I think that even the first baby steps would require a substantial,
committed, full time effort.

I think @xiaoxiang781216 has already found someone wish to help here? But anyway, we need at least a committer to review the work...

I've been writing some scripts which convert the output of git log (over a given file) into JSON format, to obtain metadata for each revision of the file. The final JSON contains (among other information): commit author, commit message and blob hash for the file.
I then started writing a python script to parse the JSON and extract (using regular expressions) authors from commit message and file header, in each commit. It is working nicely so far.
The final goal would be to determine if a given file passes the previously discussed checks for the easy cases that can be moved to Apache header. The python script could also be used to make the header change and commit the result.
I will work a bit more on this and open a draft PR (to add the script inside tools/).

I've been writing some scripts which convert the output of git log (over a given file) into JSON format, to obtain metadata for each revision of the file. The final JSON contains (among other information): commit author, commit message and blob hash for the file.

People have been using Fossology to get historical information:  https://www.fossology.org/

Yeah, life intervened and I haven't been able to get back to this. I have less time for it than I thought.
@PeterBee97 made some progress in parsing out the list of contributors from the Git log messages. I will see if I can take his list and see if I can get a list of files and also number of lines of code for each contribution... anyway that seems to be the next steps:

get a list of people who contributed
get a list of the commits they were involved with
work out how many lines of code per person are involved
sort the list largest to smallest – this will give us an idea of how big the job is
try contacting people with the n largest contributions

There are several other approaches. This is just the one that seems most straightforward to me. If anyone wants to help, we could use help with:

writing a script that could take a list of commits and output the contribution size in lines
getting a list of names and commits from the git log (Peter's scripts are this, or very close I think)

Please see #1834
I know @PeterBee97 started some of this work but to be honest it was quite difficult for me to take advantage of those, considering it was based on sqlite databases. I chose JSON format since it is quite easy to read and parse with different programming languages.

Please see #1834
I know @PeterBee97 started some of this work but to be honest it was quite difficult for me to take advantage of those, considering it was based on sqlite databases. I chose JSON format since it is quite easy to read and parse with different programming languages.

I have to be in favor of anything that makes forward progress.

@patacongo Re: anything that makes forward progress, me too.
@v01d yes, text-based json or csv/tsv formats would be great. The scripts in #1834 look cool Maybe we combine them into one python script with the sh module. I'll try them out.

@v01d yes, text-based json or csv/tsv formats would be great. The scripts in #1834 look cool. Maybe we combine them into one python script with the sh module. I'll try them out.

There's quite a bit of escaping going on in the bash script, so embedding it inside python would probably require some work. Not sure if it is worth it, but we can think about it.

Another question is, what is the impact of copyright vs authorship indication? For example:
../../arch/arm/src/samd5e5/sam_tc.h- *   Copyright 2020 Falker Automacao Agricola LTDA.
../../arch/arm/src/samd5e5/sam_tc.h: *   Author: Leomar Mateus Radke<EMAIL_ADDRESS>../../arch/arm/src/samd5e5/sam_tc.h: *   Author: Ricardo Wartchow<EMAIL_ADDRESS>The copyright sometimes (but not always) indicates a company and the authors may be employees (in this case, maybe Leomar, but maybe not Ricard). This means that we need a SGA from the company besides ICLAs from authors right? If there's no mention to a company (only author) an ICLA would be enough?
I will parse the copyright also so that we can initially filter these more complex cases.

I think we will have a hard time with many files that once existed in submodules. Git is smart enough to know about the file coming from a submodule, but it is not possible to obtain the contents of the file at that revision as it didn't exist in the parent repo.

Oops, thought I was on the PR, I'll move the comments there

@justinmclean @adamfeuer
Hi guys, we made some progress and post it here.
https://github.com/apache/incubator-nuttx/issues/1954
Basically, we collected the author/company list which have not signed the agreement.  So the next step is to contact them via email and get them sign the agreement.
My questions are the following:

Is there an email template for contacting the authors?
Where do we return the signed ICLA to? Is there somebody from Apache Foundation to collect and verify them?

ICLAs are emailed to<EMAIL_ADDRESS>see https://www.apache.org/licenses/contributor-agreements.html

@justinmclean Thanks！One more question, how would you normally contact companies to get their SGA signed? Do you contact people you know from the company to get introduced? What department is normally responsible for this?
For other authors, shall we just auto send email to contact them?

@justinmclean One more question, shall we ask authors to send ICLA directly to<EMAIL_ADDRESS>Will someone from Apache Secretary process the mails and update the list and sync with us on the author list?

I think this issue can be closed:

It is inactive.  There have been no comments since 2020
NuttX has since graduated to a TLP so all IP clearance issues must have been resolved.

If there is something I am missing please just re-open.


chore(vault): use name instead of prefix to cache vault strategies and schemas

Summary
Just a small change/fix to use right key in strategies/schema caching.

Merged, @bungle please cherrypick to EE


Korean chapter0, 1 full

Hello, I'm the assignee of Korean translation(chapter 1)
I just created _toctree.yml file and finished translation for ko/chapter1/1.mdx
Please check out for the translation and the form!
Planned to gotta finish all other part of chapter1 in a day 😄
Related to issue #42

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Hey @Doohae we have some merge conflicts in the GitHub CI files due to some recent changes from other translations. Would you mind resolving them so we can try building the docs on this PR? Thanks!

Hey @Doohae we have some merge conflicts in the GitHub CI files due to some recent changes from other translations. Would you mind resolving them so we can try building the docs on this PR? Thanks!

Hey I just resolved them and also fixed the url&tag typos, indentation, and unnatural expressions!
Really sorry for being late on the revision 😭

The docs render beautifully, so merging this!


[Spark Dataset runner] Reduce binary size of Java serialized task related for ParDo translation

When investigating the performance of TPCDS query 83, I noticed that the query plan contains fairly heavy weight tasks. These trigger a Spark warning when the tasks get broadcasted because their serialized size (using Java serialization) is above 1MB.
This is both a cleanup of the ParDo translation, but also reduces the serialized size of tasks that include many ParDos.

Use subtypes of DoFnMapPartitionsFactory to handle the single output and multi-output case, rather than relying on a closure that has to be serialized.
Use Java default collections rather than immutable Guava collections (Why does this matter for serialization???).
Use Collections.emptyList and similar where applicable.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @aromanenko-dev
R: @echauchot

Kind ping @aromanenko-dev @echauchot

Run Spark StructuredStreaming ValidatesRunner

Thanks a lot @echauchot , I've addressed all your comments

Run Spark StructuredStreaming ValidatesRunner


I32.clz_unsafe is really unsafe

Consider this example program:
actor Main
  new create(env: Env) =>
    let array = Array[String]
    array.append(env.args)
    while array.size() < 32 do
      array.push("")
    end
    (let index, let used) = try
         env.args(1)?.read_int[U32]()?
       else
         env.err.print("error: invalid index")
         return
       end
    env.out.print("info: index: " + index.string())
    env.out.print("info: used: " + used.string())
    try
      if array.size() != 32 then
        env.err.print("error: array has wrong size")
        return
      end
      env.out.print(array(magic(index).usize())?)
      env.out.print("info: magic: " + magic(index).string())
      env.out.print("info: no error thrown")
    else
      env.out.print("info: error thrown")
    end
  fun magic(x: U32): U32 =>
    let y = x.clz_unsafe()
    if y < 32 then
      y
    else
      0
    end

(This can undoubtedly be reduced further, but it will be really brittle.)
When I run it, I get this:
./unsafe 0
Segmentation fault

I believe the reason is this: LLVM treats undefined as really undefined. The bounds check y < 32 is optimized away, and on some CPUs (mine is a Core m7-6Y75), when compiling natively, clz_unsafe is implemented as lzcnt, which yields 32 for a zero input value. The subsequent code gets confused by this because according to LLVM semantics, the value 32 is impossible here. However, this is not an LLVM bug—it does what the Pony compiler instructs it to do.
My compiler:
0.24.4-fff531cf [release]
compiled with: llvm 3.9.1 -- cc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
Defaults: pic=true ssl=openssl_1.1.0

The reproducer is probably brittle, so unless you have the same environment, it will not work.
The same issue applies to all unsafe arithmetic operations, with different reproducers. (clz_unsafe was literally the first one I tried.) I think it would be reasonable to simply remove all the unsafe arithmetic methods because the performance benefits will be extremely narrow.

On a different CPU (Core i7-980X), again with a native build, it's more obvious what is going on.  The disassembly looks like this:
   0x0000555555558925 <+773>:   callq  0x55555555c100 <pony_sendv>
   0x000055555555892a <+778>:   cmp    $0x20,%r13
   0x000055555555892e <+782>:   jne    0x555555558aa0 <Main_Dispatch+1152>
   0x0000555555558934 <+788>:   bsr    0xc(%rsp),%ebp
=> 0x0000555555558939 <+793>:   xor    $0x1f,%ebp
   0x000055555555893c <+796>:   mov    0x28(%rsp),%rax
   0x0000555555558941 <+801>:   mov    (%rax,%rbp,8),%r14

This CPU does not update the destination register when bsr is executed with an input operand which is zero.  After setting a breakpoint on the indicated instruction, I get:
(gdb) print/x $ebp
$1 = 0xf6ce97a0
(gdb) print/x  *(unsigned *)($rsp+0xc)
$2 = 0x0
(gdb) print $rip
$3 = (void (*)()) 0x555555558939 <Main_Dispatch+793>

So the generated code immediately performs an invalid pointer read.

What's the status here? I'm a bit surprised by the lack of reaction, considering that this is a memory safety issue.

I was waiting for @Praetonus to chime in to see if he had ideas about fixing this.
If this can't be fixed, I would advocate for removing the affected operations from the standard library (and also removing the syntax sugar operators), since memory safety is an important guarantee for Pony. As far as I'm concerned, the only time we're allowed to drop the memory safety guarantee is when the user invokes FFI. Any standard library code that invokes FFI internally should be wrapped in such a way that we can guarantee that no user invocation of those abstractions should trigger memory unsafety problems - the rest of the standard library is set up very carefully to keep this held true.
I don't doubt that these unsafe operations have value for performance optimizations in the hot path, but you can still make those optimizations by invoking the LLVM intrinsics via FFI. My recommendation would be to make a third-party library package available that provides convenient wrappers for the memory-unsafe FFI invocations and gives obvious warnings about what you're expected to enforce in your code.
If we can keep some of the unsafe operators that don't suffer memory safety / crash issues, I'm fine with keeping those around in the standard library - I just want to cast out anything that has this issue.

An implementation as a third-party package won't be possible. The code generation of most unsafe operations requires compiler support since it uses LLVM instructions, not intrinsic functions. If we really want to remove them from the standard library, then we'd need to come up with a solution to allow specific non-builtin packages to use compiler_intrinsic.
I think a good compromise would be to remove the operator sugar and to stress out in the documentation that if you're writing unsafe in your code, you should really know what you're doing.

It's worth noting that I toyed with these concepts a bit before using real FFI - I created an unsafe package to demonstrate (ab)using FFI to arbitrarily change a ref cap:
https://github.com/jemc/pony-unsafe/blob/b940a5475536b814ecb905327d49bfd57982a1ac/lib/pony-unsafe.c#L12-L14
https://github.com/jemc/pony-unsafe/blob/b940a5475536b814ecb905327d49bfd57982a1ac/unsafe/unsafe.pony#L15-L16
The bummer was that this kind of unsafe escape hatch could never be zero-cost in terms of performance without being built into the compiler.
Doing the faux-FFI unsafe.* intrinsics is an interesting solution that opens up those possibilities without breaking the fundamental memory-safety guarantees of the vanilla language.

For some other unsafe LLVM opcodes, it may be possible to get deterministic but unspecified results by emulating what the target does at the LLVM level. For example, if the target reduces 32-bit shifts modulo 32, you can generate code to do that, and LLVM will optimize the masking operation away if this is performed implicitly by the hardware. But for CLZ, that option doesn't exist, as far as I can tell.
There are probably some unsafe operations where there is a performance impact from safe behavior, given the current state of LLVM (conversions from floating point to integers come to my mind). But for CLZ, I find that hard to believe because the precondition is easy to check, and in many cases, the compiler can determine the result anyway. On x86, performance of the instruction does not seem to be great anyway and a dynamic check (essentially a constant load plus CMOV) would mostly contribute to icache footprint.