No error handling in cd command

On an emulator, there's a symbolic link under root:
d -> /sys/kernel/debug
trying to navigate to this link gives:
/system/bin/sh: cd: /d: No such file or directory
We do not handle this as an error, but rather as a correct result - a valid directory

After updating to Windows 11, and Visual Studio 2022, this doesn't happen anymore - cd /d is successful


zephyr: Wrap sleep value with K_MSEC

Zephyr timeout API is changing and will use opaque value (k_timeout_t)
instead of raw values. K_MSEC is used to convert raw milliseconds value
to k_timeout_t.
This change is backward compatible so can be merged independently.
Signed-off-by: Krzysztof Chruscinski<EMAIL_ADDRESS>

@nordic-krch Should this be merged after the zephyr umbrella PR?

hm, don't know what is umbrella PR. In general, this is backward compatible since K_MSEC macro exists now #define K_MSEC(ms)     (ms)

it would be good to get that fairly soon because big PR relies on it (https://github.com/zephyrproject-rtos/zephyr/pull/23382) and we would like to merge it asap due rebasing cost.


Development

Changing structure relative to baseTag generation, and fixing a bug where intialtabs appended the tag identifier

v1.1


Bump to next version

Coverage remained the same at 93.796% when pulling 73576510c8520817cb9a2851905b5e96b2f2a42c on Juanlu001:bump-version into fb792476703e4575885c1554858010a4124da246 on satellogic:master.


New features & bugfixes

I haved implemented the features I suggested previously:

Added option to start with Windows: A shortcut in the start menu to the current executable is created (or deleted). Quite clean and efficient.
Added possibility to target specific audio devices: USB devices plugging and unplugging will also be detected (only if target devices include one other than the default playback device) to ensure the targeted audio devices are always "stimulated". Targeted devices are saved in the settings, and will not be lost even if the device is unplugged (it'll be reused the next time it is plugged in).
Fixed issue when waking from sleep. Power state changes will be detected and the sounds will be restarted (they would stop otherwise).
Added ILMerge task (with a nuget package) to create a standalone executable including the referenced NAudio.dll.

SPDIF-KA now uses NAudio instead of SoundPlayer. If Silence is selected, the inaudible sound will be played for 2 seconds as in my case it resolved an issue where the soundcard still needed to be initialized by a "real" sound, even if afterwards the silence sound was sufficient to keep it alive.
SoundPlayer does not allow playing simultaneous sounds and I found it not as reliable as NAudio.
I have also partially refactored the code according to C# style standards. I know that switching convention is always annoying, so feel free to revert these changes; if you have Resharper you should be able to automatically reformat it based on your standards. Ideally the repository should contain a Resharper code styling configuration so that it is straightforward to enforce the same standards.

@mayerwin Just curious: Shouldn't you have done a Sync with the upstream master branch before doing your commits? At the time of this PR, you're 25 commits behind where handruin:master is at.

Just saw that I wasn't working on the latest version, I am merging right now and will update.

For some reason I was convinced I was already synced... My bad.

There you go. I have merged the two branches. Hopefully there is no new bug (it works fine on my end).

Some more improvements...

Just FYI, I haven't forgotten, I've been really busy the past few days.  I plan to spend time tonight merging in this PR.  Thanks for contributing the changes.

Hi handruin,
Thanks for your project, any chance you might pull maywin's request into the master branche any time soon for a new release? Particularly #20 is annoying and I very much prefer your solution to the alternatives out there.

Yes, I definitely need to revisit this.  I'll get this merged and a build asap.

When I try to Build the solution from this latest commit in master (https://github.com/handruin/spdif-ka/commit/c9a656064de54f6710d03380c40ebed4eceb5200), I get:
Error		Unable to load DLL 'mscorsn.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)	SPDIFKA


multiple outputs not supported?

Bug description
If a network returns multiple outputs, the forward hook at crawler.py:181 crashes because the out parameter is a tuple and not a tensor so it doesn't support .size() method. Are module returning multiple outputs supported and I'm doing wrong or is this a bug?
Thanks.
Code snippet to reproduce the bug
out_h, out_o = model(x_crnt, x_prev, init_h, init_o)

Error traceback
  File "/local/home/...../lib64/python3.7/site-packages/torchscan/crawler.py", line 189, in _fwd_hook
    info[fw_idx]["output_shape"] = (-1, *out.shape[1:])
AttributeError: 'tuple' object has no attribute 'shape'

Environment
Collecting environment information...
TorchScan version: 0.1.2
PyTorch version: 1.8.1+cu102

OS: Amazon Linux 2

Python version: 3.7.16
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB

Nvidia driver version: 525.60.13
cuDNN version: Could not collect

Hey there @muzafferkal :wave:
My apologies about the very late reply! Could you share a minimal reproducible snippet?
I understand that your issue is with a model that has mutiple inputs to custom PyTorch modules. But additional context & snippet would help to come up with the best solution!
Cheers :v:


[HOLD for payment 2023-04-28] [$1000] Onfido tooltip doesn't scroll with textbox

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Change language preference to Spanish
Click on “Profile’ avatar
Click on “Workspace” link
Open an existing workspace
Click on ‘Issue card’ link
Click on ‘Connect manually’ button
Fill in, and submit all forms leading to ‘Get secure link’ section
Click on ‘Get link via SMS’ link
Enter a number and click on ‘Send link’ button
Scroll down the page

Expected Result:
Recomendaciones (Tips) badge should stick to the text box.
Actual Result:
Recomendaciones (Tips) badge floats on the screen
Workaround:
None needed.
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: v1.2.97-2
Reproducible in staging?: Yes
Reproducible in production?: Undetermined
If this was caught during regression testing, add the test name, ID and link from TestRail: n/a
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos:
https://user-images.githubusercontent.com/59587260/231270844-079619ad-aa88-49de-bff6-036c07550fb1.mp4
Expensify/Expensify Issue URL: n/a
Issue reported by: @Natnael-Guchima
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1681131997630749
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01693487eb58bd1974
Upwork Job ID:<PHONE_NUMBER>976273920
Last Price Increase: 2023-04-19

As discussed here this will need to be raised with Onfido. Raised this GH to reproduce and track my reach out with them.

Dug into this today, it seems to be managed by our CSS files (specifically .onfido-sdk-ui-Theme-header which is forcing an absolute position) not Onfido. Gonna have more of a play tomorrow to see if I can nail it down.

I'm heading OOO so I don't have capacity to dig into this one.
I'm able to reproduce it:
https://user-images.githubusercontent.com/59587260/232973634-c99fe6b6-a42b-4575-8c15-c7abed8a621b.mp4
I've looked into how our Onfido hand-off is set up and I'm 95% sure this UI is handled within Expensify. Onfido seems to only accept bugs on their specific platform (after the handoff) based on their issue template. I'm possibly wrong though! Would appreciate a second look from an engineer's perspective.

Triggered auto assignment to @dylanexpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

Bug0 Triage Checklist (Main S/O)

[ ] This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
[ ] This bug is not a duplicate report (check E/App issues and #expensify-bugs)

If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead

[ ] This bug is reproducible using the reproduction steps in the OP. S/O

If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
If the reproduction steps aren't clear and you determine the correct steps, please update the OP.

[ ] This issue is filled out as thoroughly and clearly as possible

Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.

[ ] I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

Job added to Upwork: https://www.upwork.com/jobs/~01693487eb58bd1974

Current assignee @dylanexpensify is eligible for the External assigner, not assigning anyone new.

Triggered auto assignment to Contributor-plus team member for initial proposal review - @mananjadhav (External)

Triggered auto assignment to @bondydaa (External), see https://stackoverflow.com/c/expensify/questions/7972 for more details.

Proposal
Please re-state the problem that we are trying to solve in this issue.
Onfido tooltip doesn't scroll with textbox.
What is the root cause of that problem?
Element is not located inside the list's parent element.
What changes do you think we should make in order to solve the problem?
Element is not located inside the list's parent element and the parent element should  include postion:relative in its style.
That's really it.
What alternative solutions did you explore? (Optional)
Not yet
Result
screen-capture (17).webm

~Just to clarify, does this only happen when in Spanish language? Those are the first 2 things of the reproduction steps but I can't imagine this only happens there considering it's most likely an onfido thing.~
NVM just tested and it doesn't happen in the english version because the box isn't scroll-able!

If I put more context into the box I'm able to get it to happen with english language

@MahmudjonToraqulov unfortunately since the onfido is a 3rd party sdk we're using I don't believe we have access to the markup to change where that tips content is rendered so we can't move it like you suggested.
I tested @hoangzinh 's css only proposal and it worked so going to assign ya

📣 @hoangzinh You have been assigned to this job by @bondydaa!
Please apply to this job in Upwork and leave a comment on the Github issue letting us know when we can expect a PR to be ready for review 🧑💻
Keep in mind: Code of Conduct | Contributing 📖

Reviewing label has been removed, please complete the "BugZero Checklist".

The solution for this issue has been :rocket: deployed to production :rocket: in version 1.3.3-2 and is now subject to a 7-day regression period :calendar:. Here is the list of pull requests that resolve this issue:

https://github.com/Expensify/App/pull/17695

If no regressions arise, payment will be issued on 2023-04-28. :confetti_ball:
After the hold period is over and BZ checklist items are completed, please complete any of the applicable payments for this issue, and check them off once done.

[ ] External issue reporter
[ ] Contributor that fixed the issue
[ ] Contributor+ that helped on the issue and/or PR

As a reminder, here are the bonuses/penalties that should be applied for any External issue:

Merged PR within 3 business days of assignment - 50% bonus
Merged PR more than 9 business days after assignment - 50% penalty

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

[ ] [@mananjadhav] The PR that introduced the bug has been identified. Link to the PR:
[ ] [@mananjadhav] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:
[ ] [@mananjadhav] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:
[ ] [@dylanexpensify] Determine if we should create a regression test for this bug.
[ ] [@mananjadhav] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
[ ] [@dylanexpensify] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:

🎯 ⚡ Woah @mananjadhav / @hoangzinh, great job pushing this forwards! ⚡
The pull request got merged within 2 working days of assignment, so this job is eligible for a 50% #urgency bonus 🎉

when @hoangzinh got assigned: 2023-04-19 18:50:48 Z
when the PR got merged: 2023-04-20 20:00:00 UTC
On to the next one 🚀

Testing!

Dear Contributors,
I have reported this issue prior, find it here https://github.com/Expensify/App/issues/15364
However, the other C+ closed that issue ( saying we are not fixing this).
Expecting fairness.

Thanks for raising it, @dylanexpensify will take care of that when issuing the payments 🙌

@dylanexpensify quick bump of this for the payout.

@mountiny @bondydaa @dylanexpensify I don't think we need a regression test here as it is on onFido and I don't think we have any offending PR as the styles are from the SDK.

Paying now!

@mananjadhav @ashimsharma10 @hoangzinh can you please apply for job for 💸

@dylanexpensify Can you please share the invite/send an offer? I am having some issues with my Upwork profile and on mobile.

@mananjadhav sent invite!
@hoangzinh sent offer! (Bonus will be applied upon payout!)

@dylanexpensify I don’t see it on Upwork or my email, can you please confirm if you’ve sent ?

Shows as invited? 🤷♂

@hoangzinh payment sent, contract ended! Thanks for helping us out!

@dylanexpensify Haven't got any offer requests. Can you please confirm?
My upwork : https://www.upwork.com/freelancers/~018a92cf13e1e88eed

Yeah I applied, sorry about that. It didn’t show up in notification, but was available in the listing.

@ashimsharma10 sent invite!
@mananjadhav no worries!!

@mananjadhav offer sent!

Everyone has been paid!

Agree with that, thanks everyone!


generate release notes

Looks like this: https://github.com/joejulian/helm-charts/releases/tag/redpanda-5.6.34-1

Tests will not run. @redpanda-data/devprod please admin merge.


VERSION.cmake bugfix

hotfix for issue #5 , spurious newline coming in to Cmake from parsing the Git tags to get PROJECT_VERSION_FULL

I found in old iPhone 5 How can use this Device  with having linked Apple ID.


Add: sum function in tasks > javascript > easy > sum.js

Pull Request Type
What kind of change does this PR introduce?
[ ] Solved an issue/task
What is the current behavior?

task to add two numbers
Issue Number: [Insert Issue Number (if applicable)]
1514
What is the new behavior?

I add simple calculate of two numbers in javascript
Other information

No

Merged

This is an automated message from Fork, Commit, Merge [BOT].

Thank you for your contribution! Your pull request has been merged. The files have been reset for the next contributor.
What's next?
If you're looking for more ways to contribute, I invite you to check out our forkcommitmerge.io website repository. The repository contain real issues that you can help to resolve or create some of your own if you see any bugs or want to add new features to the site, etc. You can also check out the Influences section in the README to find more projects similar to this one.
Also please leave a star to this project if you feel it helped you, i would really appreciate it.
I look forward to seeing your contributions!


Please provide us with a arm-linux-gnueabihf-raspbian formular

I'm not sure if it's the right place to ask but I tried build it by my own (via crosstool-ng)
but finally I got a ld-error and I had to give up.
I know there is the arm-none-eabi formula but this bear metal version just overcomplicates things - I don't want to develop a kernel - just a simple cpp program.

Closing for several reasons:

We don’t do formulas, but casks, so you may be looking for homebrew.
I can’t find the formula you’re talking about, so I have no idea if you’re in the right place or not.
Cask requests must follow the template.


How do I access a global namespace from another file

I cant seem to be able to access the global namespace I created from another file.
I created the global namespace in a package(internal package) and I'm trying to access it where I imported the package.
I created the global namespace in globalNamespace.ts
import 'prisma-json-types-generator'
declare global {
  namespace PrismaJsonTypes {
    // Details
  }
}

you should remove the global namespace, then export PrismaJsonTypes, and any of your interfaces/types declared inside.

The prisma-json-types-generator import is wrong and just imports unused code at runtime.
The readme says to you do the following:
declare global {
  namespace PrismaJson {
    // you can use classes, interfaces, types, etc.
    type MyType = boolean;
  }
}

It probably is because your typescript setup is wrongly configured, but as you did not provide any reproducible example, neither on how you are trying to use the type or even the error it shows, I'm afraid I cannot help you.
Try checking if the file is getting included in your tsconfig, isolatedModules and/or importing it inside your index.ts file.


Renaming doWithCommandeer to cfgInit to make clearer what it does and for consistency

Regarding my proposal to simplify doWithCommandeer, I found that renaming alone will clear things up.
For one, I realized that there are two different object types:

doWithCommandeer func(c *commandeer) error)
doWithCommandeer func(cfg config.Provider) error

When initializeConfig(mustHaveConfigFile, running bool, h *hugoBuilderCommon, f flagsToConfigHandler, doWithCommandeer func(c *commandeer) error) is called, the last argument is either nil or was named cfgInit.
Therefore, renaming #1 to cfgInit clears things up because it's the only thing it ever does.
I renamed #2 cfgSetAndInit because it sets the Cfg field of commandeer (to a *viper.Viper pointer) and runs cfgInit.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

You need to gofmt your source files. I recommend you use an editor where you can configure this to happen automatically on save. I use LiteIDE where this is default.

That's strange. I do have gofmt on save and I now manually ran it. There are no changes to commit.

What Go version are you running? We do the "gofmt check" on Travis with Go 1.13 only as they tend to change their format rules slightly from version to version.

What Go version are you running? We do the "gofmt check" on Travis with Go 1.13 only as they tend to change their format rules slightly from version to version.

Fixed it, thanks!

OK, the change itself looks good, but can you squash the 3 commits and make a proper commit message  (it does not have to be long, important part is the title).

Sorry.


UAny gate cuda compatibility

Problem Description
I am trying to run a QumodeCircuit  on GPU with UAny gate. Initial states are set on 'cuda', but when the input unitary is np.array, RuntimeError turns out, saying unitary on cpu device.

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_mm)

Potential Reasons
UAnyGateclass in gate.py returns a new tensor without declaring the device, in line 830 :
if not isinstance(unitary, torch.Tensor):
    unitary = torch.tensor(unitary, dtype=torch.cfloat).reshape(-1, len(self.wires))
Possible Solutions
Though reasons are clear, I don't know how to refer device = init_state.device, as they seems to be run seperately.
If restricting users to revert unitary to torch.tensor and to('cuda') if they want to run on GPU, then we should convert it to complex data type instead of asserting a warning.

Add cir.to('cuda:0') before running forward function


PageStorage: Fix pages are not deleted under some cases (#5069)

cherry-pick of https://github.com/pingcap/tiflash/pull/5069

What problem does this PR solve?
Issue Number: close https://github.com/pingcap/tiflash/issues/5054
Problem Summary: as the issue described
What is changed and how it works?

StoragePool only need to remove DTFiles ids that are not exist in the V3.data, cause we perform force transform from V2.data to V3.data when restore
Keep tombstone when merging the delta version for page entries in V2

Check List
Tests

[x] Unit test
[ ] Integration test
[x] Manual test (add detailed scripts or steps below)
[ ] No code

Manual test steps:

Deploy cluster and load some data into TiFlash with storage.format_version = 3
Change the storage.format_version = 4
Use alter table t compact tiflash replica to perform delta-merge
Check that the old DTFiles are removed

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Documentation

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Changes MySQL compatibility

Release note

None

/run-all-tests

/merge


[WIP] Add conv2d gradients

This PR enables efficient conv2d_nchw gradients on x86. Specifically, it

adds an x86 schedule for conv2d_transpose which nets a 10x boost in performance
adds the conv2d_grad_weight_nchw operator with a custom schedule for x86

Some notes:

the method of specifying TVM tags should be made 1)  composable and 2) consistent across C++ and Python

WIP until:

[ ] streamlined functions for all layouts via nchw converters
[ ] add docs and other pylint necessities

any updates on this PR?

I'll take a pass at it this week. I'll have to take a non-superficial look at https://discuss.tvmlang.org/t/datalayout-structure/80/2, though.

close for now due to inactive status, please PR new ones when ready :)

Hello. First thanks for creating TVM, it looks very promising to deploy and speed up networks on different architectures.
(disclaimer: I'm very new to TVM, still working on understanding how it works internally)
I'd like to know if there are plans to speed up transposed convolutions, as right now this makes our networks (U-Net-like) run 2 times slower than in pytorch. Alternatively, is there a way to find a workaround to this
Is this PR also progress towards that https://github.com/dmlc/tvm/issues/2519 ?


Build Velox on mac m1

Problem description
After I executed scripts/setup-macos.sh
And then  executed CPU_TARGET="arm64" make debug
here something happend.
System information
Velox System Info v0.0.1
Commit: f076c302bb708f2a7dd8d65e5f8c434a25bd72d0
CMake Version:
System:
Arch:
C++ Compiler:
C++ Compiler Version:
C Compiler:
C Compiler Version:
CMake Prefix Path:
CMake log
The system is: Darwin - 21.5.0 - x86_64
Compiling the C compiler identification source file "CMakeCCompilerId.c" succeeded.
Compiler: /Library/Developer/CommandLineTools/usr/bin/cc
Build flags:
Id flags: -c

The output was:
0

Compilation of the C compiler identification source "CMakeCCompilerId.c" produced "CMakeCCompilerId.o"

The C compiler identification is AppleClang, found in "/Users/amory/CLionProjects/velox/_build/debug/CMakeFiles/3.25.0/CompilerIdC/CMakeCCompilerId.o"

Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" succeeded.
Compiler: /Library/Developer/CommandLineTools/usr/bin/c++
Build flags:
Id flags: -c

The output was:
0

Compilation of the CXX compiler identification source "CMakeCXXCompilerId.cpp" produced "CMakeCXXCompilerId.o"

The CXX compiler identification is AppleClang, found in "/Users/amory/CLionProjects/velox/_build/debug/CMakeFiles/3.25.0/CompilerIdCXX/CMakeCXXCompilerId.o"

Detecting C compiler ABI info compiled with the following output:
Change Dir: /Users/amory/CLionProjects/velox/_build/debug/CMakeFiles/CMakeScratch/TryCompile-GBF2y2

Run Build Command(s):/usr/local/bin/ninja cmTC_49333 && [1/2] Building C object CMakeFiles/cmTC_49333.dir/CMakeCCompilerABI.c.o
Apple clang version 13.1.6 (clang-13<IP_ADDRESS>.5)
Target: x86_64-apple-darwin21.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
clang: warning: -Wl,-v: 'linker' input unused [-Wunused-command-line-argument]
 "/Library/Developer/CommandLineTools/usr/bin/clang" -cc1 -triple x86_64-apple-macosx12.0.0 -Wundef-prefix=TARGET_OS_ -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -emit-obj -mrelax-all --mrelax-relocations -disable-free -disable-llvm-verifier -discard-value-names -main-file-name CMakeCCompilerABI.c -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-strict-return -fno-rounding-math -munwind-tables -target-sdk-version=12.3 -fvisibility-inlines-hidden-static-local-var -target-cpu penryn -tune-cpu generic -debugger-tuning=lldb -target-linker-version 763 -v -resource-dir /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6 -dependency-file CMakeFiles/cmTC_49333.dir/CMakeCCompilerABI.c.o.d -skip-unused-modulemap-deps -MT CMakeFiles/cmTC_49333.dir/CMakeCCompilerABI.c.o -sys-header-deps -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/local/include -internal-isystem /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include -internal-externc-isystem /Library/Developer/CommandLineTools/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -Wno-elaborated-enum-base -Wno-reserved-identifier -Wno-gnu-folding-constant -Wno-objc-load-method -fdebug-compilation-dir=/Users/amory/CLionProjects/velox/_build/debug/CMakeFiles/CMakeScratch/TryCompile-GBF2y2 -ferror-limit 19 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fmax-type-align=16 -fcommon -clang-vendor-feature=+messageToSelfInClassMethodIdReturnType -clang-vendor-feature=+disableInferNewAvailabilityFromInit -clang-vendor-feature=+disableNonDependentMemberExprInCurrentInstantiation -fno-odr-hash-protocols -clang-vendor-feature=+enableAggressiveVLAFolding -clang-vendor-feature=+revert09abecef7bbf -clang-vendor-feature=+thisNoAlignAttr -clang-vendor-feature=+thisNoNullAttr -mllvm -disable-aligned-alloc-awareness=1 -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o CMakeFiles/cmTC_49333.dir/CMakeCCompilerABI.c.o -x c /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/CMakeCCompilerABI.c
clang -cc1 version 13.1.6 (clang-13<IP_ADDRESS>.5) default target x86_64-apple-darwin21.5.0
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/local/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include
 /Library/Developer/CommandLineTools/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/System/Library/Frameworks (framework directory)
End of search list.
[2/2] Linking C executable cmTC_49333
Apple clang version 13.1.6 (clang-13<IP_ADDRESS>.5)
Target: x86_64-apple-darwin21.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
 "/Library/Developer/CommandLineTools/usr/bin/ld" -demangle -lto_library /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib -dynamic -arch x86_64 -platform_version macos 12.0.0 12.3 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk -o cmTC_49333 -search_paths_first -headerpad_max_install_names -v CMakeFiles/cmTC_49333.dir/CMakeCCompilerABI.c.o -lSystem /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/lib/darwin/libclang_rt.osx.a
@(#)PROGRAM:ld  PROJECT:ld64-764
BUILD 11:22:50 Apr 28 2022
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
Library search paths:
	/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/lib
Framework search paths:
	/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/System/Library/Frameworks/

Parsed C implicit include dir info from above output: rv=done
  found start of include info
  found start of implicit include info
    add: [/Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include]
    add: [/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include]
    add: [/Library/Developer/CommandLineTools/usr/include]
  end of search list found
  collapse include dir [/Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include] ==> [/Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include]
  collapse include dir [/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include] ==> [/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include]
  collapse include dir [/Library/Developer/CommandLineTools/usr/include] ==> [/Library/Developer/CommandLineTools/usr/include]
  implicit include dirs: [/Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include;/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include;/Library/Developer/CommandLineTools/usr/include]

Parsed C implicit link information from above output:
  link line regex: [^( *|.*[/\])(ld|CMAKE_LINK_STARTFILE-NOTFOUND|([^/\]+-)?ld|collect2)[^/\]*( |$)]
  ignore line: [Change Dir: /Users/amory/CLionProjects/velox/_build/debug/CMakeFiles/CMakeScratch/TryCompile-GBF2y2]
  ignore line: []
  ignore line: [Run Build Command(s):/usr/local/bin/ninja cmTC_49333 && [1/2] Building C object CMakeFiles/cmTC_49333.dir/CMakeCCompilerABI.c.o]
  ignore line: [Apple clang version 13.1.6 (clang-13<IP_ADDRESS>.5)]
  ignore line: [Target: x86_64-apple-darwin21.5.0]
  ignore line: [Thread model: posix]
  ignore line: [InstalledDir: /Library/Developer/CommandLineTools/usr/bin]
  ignore line: [clang: warning: -Wl -v: 'linker' input unused [-Wunused-command-line-argument]]
  ignore line: [ "/Library/Developer/CommandLineTools/usr/bin/clang" -cc1 -triple x86_64-apple-macosx12.0.0 -Wundef-prefix=TARGET_OS_ -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -emit-obj -mrelax-all --mrelax-relocations -disable-free -disable-llvm-verifier -discard-value-names -main-file-name CMakeCCompilerABI.c -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-strict-return -fno-rounding-math -munwind-tables -target-sdk-version=12.3 -fvisibility-inlines-hidden-static-local-var -target-cpu penryn -tune-cpu generic -debugger-tuning=lldb -target-linker-version 763 -v -resource-dir /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6 -dependency-file CMakeFiles/cmTC_49333.dir/CMakeCCompilerABI.c.o.d -skip-unused-modulemap-deps -MT CMakeFiles/cmTC_49333.dir/CMakeCCompilerABI.c.o -sys-header-deps -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/local/include -internal-isystem /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include -internal-externc-isystem /Library/Developer/CommandLineTools/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -Wno-elaborated-enum-base -Wno-reserved-identifier -Wno-gnu-folding-constant -Wno-objc-load-method -fdebug-compilation-dir=/Users/amory/CLionProjects/velox/_build/debug/CMakeFiles/CMakeScratch/TryCompile-GBF2y2 -ferror-limit 19 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fmax-type-align=16 -fcommon -clang-vendor-feature=+messageToSelfInClassMethodIdReturnType -clang-vendor-feature=+disableInferNewAvailabilityFromInit -clang-vendor-feature=+disableNonDependentMemberExprInCurrentInstantiation -fno-odr-hash-protocols -clang-vendor-feature=+enableAggressiveVLAFolding -clang-vendor-feature=+revert09abecef7bbf -clang-vendor-feature=+thisNoAlignAttr -clang-vendor-feature=+thisNoNullAttr -mllvm -disable-aligned-alloc-awareness=1 -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o CMakeFiles/cmTC_49333.dir/CMakeCCompilerABI.c.o -x c /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/CMakeCCompilerABI.c]
  ignore line: [clang -cc1 version 13.1.6 (clang-13<IP_ADDRESS>.5) default target x86_64-apple-darwin21.5.0]
  ignore line: [ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/local/include"]
  ignore line: [ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/Library/Frameworks"]
  ignore line: [#include "..."  search starts here:
#include <...> search starts here:
 /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1
 /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include
 /Library/Developer/CommandLineTools/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/System/Library/Frameworks (framework directory)
End of search list.
[2/2] Linking CXX executable cmTC_21250
Apple clang version 13.1.6 (clang-13<IP_ADDRESS>.5)
Target: x86_64-apple-darwin21.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
 "/Library/Developer/CommandLineTools/usr/bin/ld" -demangle -lto_library /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib -dynamic -arch x86_64 -platform_version macos 12.0.0 12.3 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk -o cmTC_21250 -search_paths_first -headerpad_max_install_names -v CMakeFiles/cmTC_21250.dir/CMakeCXXCompilerABI.cpp.o -lc++ -lSystem /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/lib/darwin/libclang_rt.osx.a
@(#)PROGRAM:ld  PROJECT:ld64-764
BUILD 11:22:50 Apr 28 2022
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
Library search paths:
	/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/lib
Framework search paths:
	/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/System/Library/Frameworks/

Parsed CXX implicit include dir info from above output: rv=done
  found start of include info
  found start of implicit include info
    add: [/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1]
    add: [/Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include]
    add: [/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include]
    add: [/Library/Developer/CommandLineTools/usr/include]
  end of search list found
  collapse include dir [/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1] ==> [/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1]
  collapse include dir [/Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include] ==> [/Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include]
  collapse include dir [/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include] ==> [/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include]
  collapse include dir [/Library/Developer/CommandLineTools/usr/include] ==> [/Library/Developer/CommandLineTools/usr/include]
  implicit include dirs: [/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1;/Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include;/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include;/Library/Developer/CommandLineTools/usr/include]

Parsed CXX implicit link information from above output:
  link line regex: [^( *|.*[/\])(ld|CMAKE_LINK_STARTFILE-NOTFOUND|([^/\]+-)?ld|collect2)[^/\]*( |$)]
  ignore line: [Change Dir: /Users/amory/CLionProjects/velox/_build/debug/CMakeFiles/CMakeScratch/TryCompile-KErfhr]
  ignore line: []
  ignore line: [Run Build Command(s):/usr/local/bin/ninja cmTC_21250 && [1/2] Building CXX object CMakeFiles/cmTC_21250.dir/CMakeCXXCompilerABI.cpp.o]
  ignore line: [Apple clang version 13.1.6 (clang-13<IP_ADDRESS>.5)]
  ignore line: [Target: x86_64-apple-darwin21.5.0]
  ignore line: [Thread model: posix]
  ignore line: [InstalledDir: /Library/Developer/CommandLineTools/usr/bin]
  ignore line: [clang: warning: -Wl -v: 'linker' input unused [-Wunused-command-line-argument]]
  ignore line: [ "/Library/Developer/CommandLineTools/usr/bin/clang" -cc1 -triple x86_64-apple-macosx12.0.0 -Wundef-prefix=TARGET_OS_ -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -emit-obj -mrelax-all --mrelax-relocations -disable-free -disable-llvm-verifier -discard-value-names -main-file-name CMakeCXXCompilerABI.cpp -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-strict-return -fno-rounding-math -munwind-tables -target-sdk-version=12.3 -fvisibility-inlines-hidden-static-local-var -target-cpu penryn -tune-cpu generic -debugger-tuning=lldb -target-linker-version 763 -v -resource-dir /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6 -dependency-file CMakeFiles/cmTC_21250.dir/CMakeCXXCompilerABI.cpp.o.d -skip-unused-modulemap-deps -MT CMakeFiles/cmTC_21250.dir/CMakeCXXCompilerABI.cpp.o -sys-header-deps -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk -stdlib=libc++ -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1 -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/local/include -internal-isystem /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include -internal-externc-isystem /Library/Developer/CommandLineTools/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -Wno-elaborated-enum-base -Wno-reserved-identifier -Wno-gnu-folding-constant -Wno-objc-load-method -fdeprecated-macro -fdebug-compilation-dir=/Users/amory/CLionProjects/velox/_build/debug/CMakeFiles/CMakeScratch/TryCompile-KErfhr -ferror-limit 19 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fno-cxx-modules -fcxx-exceptions -fexceptions -fmax-type-align=16 -fcommon -clang-vendor-feature=+messageToSelfInClassMethodIdReturnType -clang-vendor-feature=+disableInferNewAvailabilityFromInit -clang-vendor-feature=+disableNonDependentMemberExprInCurrentInstantiation -fno-odr-hash-protocols -clang-vendor-feature=+enableAggressiveVLAFolding -clang-vendor-feature=+revert09abecef7bbf -clang-vendor-feature=+thisNoAlignAttr -clang-vendor-feature=+thisNoNullAttr -mllvm -disable-aligned-alloc-awareness=1 -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o CMakeFiles/cmTC_21250.dir/CMakeCXXCompilerABI.cpp.o -x c++ /usr/local/Cellar/cmake/3.25.0/share/cmake/Modules/CMakeCXXCompilerABI.cpp]
  ignore line: [clang -cc1 version 13.1.6 (clang-13<IP_ADDRESS>.5) default target x86_64-apple-darwin21.5.0]
  ignore line: [ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/local/include"]
  ignore line: [ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/Library/Frameworks"]
  ignore line: [#include "..."  Please install cmake and try CPU_TARGET="arm64" make debug again from within a system CLI not from within the IDE.

I have already made CPU_TARGET="arm64" , and my cmake is always installed , but also failed ...

It seems that double-conversion is not installed (or can't be found) could you check if it was built when running setup-macos?

double-conversion  is build as .a file but here is no dylib file ？ do I miss any commands here to build it ?  I make build it as the README.md told strictly ...

Ah, we seem to only build the static version pls apply this diff and try installing again (after removing)
diff --git a/scripts/setup-macos.sh b/scripts/setup-macos.sh
index e02a661c..77ec587b 100755
--- a/scripts/setup-macos.sh
+++ b/scripts/setup-macos.sh
@@ -87,7 +87,7 @@ function install_fmt {

 function install_double_conversion {
   github_checkout google/double-conversion v3.1.5
-  cmake_install -DBUILD_TESTING=OFF
+  cmake_install -DBUILD_TESTING=OFF -DBUILD_SHARED_LIBS=ON
 }

 function install_ranges_v3 {

I fix it , but here new stuff comes up ...
first .I try:

then I change the build flag, but also wrong here:

last I try to use sse , but do not work !!! What Can I do else ?

@amorynan could you please stop posting screenshots and post proper logs incl. the invocation you used to compile?
@kgpai Could you confirm that my fix works and allows building velox on m1?

log here :
cmake --build _build/debug -j 10
[60/968] Building CXX object velox/common/caching/CMakeFiles/velox_caching.dir/SsdCache.cpp.o
FAILED: velox/common/caching/CMakeFiles/velox_caching.dir/SsdCache.cpp.o
ccache /Library/Developer/CommandLineTools/usr/bin/c++ -DBOOST_ALL_NO_LIB -DBOOST_CHRONO_DYN_LINK -DBOOST_CONTEXT_DYN_LINK -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_REGEX_DYN_LINK -DBOOST_SYSTEM_DYN_LINK -DBOOST_THREAD_DYN_LINK -DFMT_LOCALE -DGFLAGS_IS_A_DLL=0 -I/usr/local/opt/icu4c/include -I/Users/amory/CLionProjects/velox/. -I/Users/amory/CLionProjects/velox/velox/external/xxhash -I/Users/amory/CLionProjects/velox/third_party/xsimd/include -isystem /usr/local/include -isystem /Users/amory/CLionProjects/velox/velox -isystem /Users/amory/CLionProjects/velox/velox/external -isystem /Users/amory/CLionProjects/velox/velox/external/duckdb -isystem /Users/amory/CLionProjects/velox/velox/external/duckdb/tpch/dbgen/include -isystem<EMAIL_ADDRESS>-isystem /Users/amory/CLionProjects/velox/third_party/googletest/googletest/include -isystem /Users/amory/CLionProjects/velox/third_party/googletest/googletest -msse4.2 -std=c++17 -fvisibility=hidden -fvisibility-inlines-hidden -D USE_VELOX_COMMON_BASE -D HAS_UNCAUGHT_EXCEPTIONS -Werror -Wall -Wextra -Wno-unused        -Wno-unused-parameter        -Wno-sign-compare        -Wno-ignored-qualifiers        -Wno-range-loop-analysis -Wno-mismatched-tags -g -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk -fPIC -std=gnu++17 -MD -MT velox/common/caching/CMakeFiles/velox_caching.dir/SsdCache.cpp.o -MF velox/common/caching/CMakeFiles/velox_caching.dir/SsdCache.cpp.o.d -o velox/common/caching/CMakeFiles/velox_caching.dir/SsdCache.cpp.o -c /Users/amory/CLionProjects/velox/velox/common/caching/SsdCache.cpp
In file included from /Users/amory/CLionProjects/velox/velox/common/caching/SsdCache.cpp:16:
In file included from /Users/amory/CLionProjects/velox/./velox/common/caching/SsdCache.h:19:
In file included from /Users/amory/CLionProjects/velox/./velox/common/caching/SsdFile.h:19:
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:3: error: pointer is missing a nullability type specifier (_Nonnull, _Nullable, or Null_unspecified) [-Werror,-Wnullability-completeness]
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:39: note: insert 'Nullable' if the pointer may be null
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
_Nullable
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:39: note: insert 'Nonnull' if the pointer should never be null
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
_Nonnull
1 error generated.
[61/968] Building CXX object velox/common/caching/CMakeFiles/velox_caching.dir/AsyncDataCache.cpp.o
FAILED: velox/common/caching/CMakeFiles/velox_caching.dir/AsyncDataCache.cpp.o
ccache /Library/Developer/CommandLineTools/usr/bin/c++ -DBOOST_ALL_NO_LIB -DBOOST_CHRONO_DYN_LINK -DBOOST_CONTEXT_DYN_LINK -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_REGEX_DYN_LINK -DBOOST_SYSTEM_DYN_LINK -DBOOST_THREAD_DYN_LINK -DFMT_LOCALE -DGFLAGS_IS_A_DLL=0 -I/usr/local/opt/icu4c/include -I/Users/amory/CLionProjects/velox/. -I/Users/amory/CLionProjects/velox/velox/external/xxhash -I/Users/amory/CLionProjects/velox/third_party/xsimd/include -isystem /usr/local/include -isystem /Users/amory/CLionProjects/velox/velox -isystem /Users/amory/CLionProjects/velox/velox/external -isystem /Users/amory/CLionProjects/velox/velox/external/duckdb -isystem /Users/amory/CLionProjects/velox/velox/external/duckdb/tpch/dbgen/include -isystem<EMAIL_ADDRESS>-isystem /Users/amory/CLionProjects/velox/third_party/googletest/googletest/include -isystem /Users/amory/CLionProjects/velox/third_party/googletest/googletest -msse4.2 -std=c++17 -fvisibility=hidden -fvisibility-inlines-hidden -D USE_VELOX_COMMON_BASE -D HAS_UNCAUGHT_EXCEPTIONS -Werror -Wall -Wextra -Wno-unused        -Wno-unused-parameter        -Wno-sign-compare        -Wno-ignored-qualifiers        -Wno-range-loop-analysis -Wno-mismatched-tags -g -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk -fPIC -std=gnu++17 -MD -MT velox/common/caching/CMakeFiles/velox_caching.dir/AsyncDataCache.cpp.o -MF velox/common/caching/CMakeFiles/velox_caching.dir/AsyncDataCache.cpp.o.d -o velox/common/caching/CMakeFiles/velox_caching.dir/AsyncDataCache.cpp.o -c /Users/amory/CLionProjects/velox/velox/common/caching/AsyncDataCache.cpp
In file included from /Users/amory/CLionProjects/velox/velox/common/caching/AsyncDataCache.cpp:17:
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:3: error: pointer is missing a nullability type specifier (_Nonnull, _Nullable, or Null_unspecified) [-Werror,-Wnullability-completeness]
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:39: note: insert 'Nullable' if the pointer may be null
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
_Nullable
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:39: note: insert 'Nonnull' if the pointer should never be null
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
_Nonnull
1 error generated.
[62/968] Building CXX object velox/common/caching/tests/CMakeFiles/velox_cache_test.dir/SsdFileTest.cpp.o
FAILED: velox/common/caching/tests/CMakeFiles/velox_cache_test.dir/SsdFileTest.cpp.o
ccache /Library/Developer/CommandLineTools/usr/bin/c++ -DBOOST_ALL_NO_LIB -DBOOST_CHRONO_DYN_LINK -DBOOST_CONTEXT_DYN_LINK -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_REGEX_DYN_LINK -DBOOST_SYSTEM_DYN_LINK -DBOOST_THREAD_DYN_LINK -DFMT_LOCALE -DGFLAGS_IS_A_DLL=0 -I/usr/local/opt/icu4c/include -I/Users/amory/CLionProjects/velox/. -I/Users/amory/CLionProjects/velox/velox/external/xxhash -I/Users/amory/CLionProjects/velox/third_party/xsimd/include -isystem /usr/local/include -isystem /Users/amory/CLionProjects/velox/velox -isystem /Users/amory/CLionProjects/velox/velox/external -isystem /Users/amory/CLionProjects/velox/velox/external/duckdb -isystem /Users/amory/CLionProjects/velox/velox/external/duckdb/tpch/dbgen/include -isystem<EMAIL_ADDRESS>-isystem /Users/amory/CLionProjects/velox/third_party/googletest/googletest/include -isystem /Users/amory/CLionProjects/velox/third_party/googletest/googletest -msse4.2 -std=c++17 -fvisibility=hidden -fvisibility-inlines-hidden -D USE_VELOX_COMMON_BASE -D HAS_UNCAUGHT_EXCEPTIONS -Werror -Wall -Wextra -Wno-unused        -Wno-unused-parameter        -Wno-sign-compare        -Wno-ignored-qualifiers        -Wno-range-loop-analysis -Wno-mismatched-tags -g -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk -fPIE -std=gnu++17 -MD -MT velox/common/caching/tests/CMakeFiles/velox_cache_test.dir/SsdFileTest.cpp.o -MF velox/common/caching/tests/CMakeFiles/velox_cache_test.dir/SsdFileTest.cpp.o.d -o velox/common/caching/tests/CMakeFiles/velox_cache_test.dir/SsdFileTest.cpp.o -c /Users/amory/CLionProjects/velox/velox/common/caching/tests/SsdFileTest.cpp
In file included from /Users/amory/CLionProjects/velox/velox/common/caching/tests/SsdFileTest.cpp:18:
In file included from /Users/amory/CLionProjects/velox/./velox/common/caching/SsdCache.h:19:
In file included from /Users/amory/CLionProjects/velox/./velox/common/caching/SsdFile.h:19:
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:3: error: pointer is missing a nullability type specifier (_Nonnull, _Nullable, or Null_unspecified) [-Werror,-Wnullability-completeness]
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:39: note: insert 'Nullable' if the pointer may be null
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
_Nullable
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:39: note: insert 'Nonnull' if the pointer should never be null
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
_Nonnull
1 error generated.
[63/968] Building CXX object velox/common/caching/CMakeFiles/velox_caching.dir/SsdFile.cpp.o
FAILED: velox/common/caching/CMakeFiles/velox_caching.dir/SsdFile.cpp.o
ccache /Library/Developer/CommandLineTools/usr/bin/c++ -DBOOST_ALL_NO_LIB -DBOOST_CHRONO_DYN_LINK -DBOOST_CONTEXT_DYN_LINK -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_REGEX_DYN_LINK -DBOOST_SYSTEM_DYN_LINK -DBOOST_THREAD_DYN_LINK -DFMT_LOCALE -DGFLAGS_IS_A_DLL=0 -I/usr/local/opt/icu4c/include -I/Users/amory/CLionProjects/velox/. -I/Users/amory/CLionProjects/velox/velox/external/xxhash -I/Users/amory/CLionProjects/velox/third_party/xsimd/include -isystem /usr/local/include -isystem /Users/amory/CLionProjects/velox/velox -isystem /Users/amory/CLionProjects/velox/velox/external -isystem /Users/amory/CLionProjects/velox/velox/external/duckdb -isystem /Users/amory/CLionProjects/velox/velox/external/duckdb/tpch/dbgen/include -isystem<EMAIL_ADDRESS>-isystem /Users/amory/CLionProjects/velox/third_party/googletest/googletest/include -isystem /Users/amory/CLionProjects/velox/third_party/googletest/googletest -msse4.2 -std=c++17 -fvisibility=hidden -fvisibility-inlines-hidden -D USE_VELOX_COMMON_BASE -D HAS_UNCAUGHT_EXCEPTIONS -Werror -Wall -Wextra -Wno-unused        -Wno-unused-parameter        -Wno-sign-compare        -Wno-ignored-qualifiers        -Wno-range-loop-analysis -Wno-mismatched-tags -g -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk -fPIC -std=gnu++17 -MD -MT velox/common/caching/CMakeFiles/velox_caching.dir/SsdFile.cpp.o -MF velox/common/caching/CMakeFiles/velox_caching.dir/SsdFile.cpp.o.d -o velox/common/caching/CMakeFiles/velox_caching.dir/SsdFile.cpp.o -c /Users/amory/CLionProjects/velox/velox/common/caching/SsdFile.cpp
In file included from /Users/amory/CLionProjects/velox/velox/common/caching/SsdFile.cpp:17:
In file included from /Users/amory/CLionProjects/velox/./velox/common/caching/SsdFile.h:19:
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:3: error: pointer is missing a nullability type specifier (_Nonnull, _Nullable, or Null_unspecified) [-Werror,-Wnullability-completeness]
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:39: note: insert 'Nullable' if the pointer may be null
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
_Nullable
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:39: note: insert 'Nonnull' if the pointer should never be null
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
_Nonnull
1 error generated.
[65/968] Building CXX object velox/common/caching/tests/CMakeFiles/velox_cache_test.dir/AsyncDataCacheTest.cpp.o
FAILED: velox/common/caching/tests/CMakeFiles/velox_cache_test.dir/AsyncDataCacheTest.cpp.o
ccache /Library/Developer/CommandLineTools/usr/bin/c++ -DBOOST_ALL_NO_LIB -DBOOST_CHRONO_DYN_LINK -DBOOST_CONTEXT_DYN_LINK -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_REGEX_DYN_LINK -DBOOST_SYSTEM_DYN_LINK -DBOOST_THREAD_DYN_LINK -DFMT_LOCALE -DGFLAGS_IS_A_DLL=0 -I/usr/local/opt/icu4c/include -I/Users/amory/CLionProjects/velox/. -I/Users/amory/CLionProjects/velox/velox/external/xxhash -I/Users/amory/CLionProjects/velox/third_party/xsimd/include -isystem /usr/local/include -isystem /Users/amory/CLionProjects/velox/velox -isystem /Users/amory/CLionProjects/velox/velox/external -isystem /Users/amory/CLionProjects/velox/velox/external/duckdb -isystem /Users/amory/CLionProjects/velox/velox/external/duckdb/tpch/dbgen/include -isystem<EMAIL_ADDRESS>-isystem /Users/amory/CLionProjects/velox/third_party/googletest/googletest/include -isystem /Users/amory/CLionProjects/velox/third_party/googletest/googletest -msse4.2 -std=c++17 -fvisibility=hidden -fvisibility-inlines-hidden -D USE_VELOX_COMMON_BASE -D HAS_UNCAUGHT_EXCEPTIONS -Werror -Wall -Wextra -Wno-unused        -Wno-unused-parameter        -Wno-sign-compare        -Wno-ignored-qualifiers        -Wno-range-loop-analysis -Wno-mismatched-tags -g -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk -fPIE -std=gnu++17 -MD -MT velox/common/caching/tests/CMakeFiles/velox_cache_test.dir/AsyncDataCacheTest.cpp.o -MF velox/common/caching/tests/CMakeFiles/velox_cache_test.dir/AsyncDataCacheTest.cpp.o.d -o velox/common/caching/tests/CMakeFiles/velox_cache_test.dir/AsyncDataCacheTest.cpp.o -c /Users/amory/CLionProjects/velox/velox/common/caching/tests/AsyncDataCacheTest.cpp
In file included from /Users/amory/CLionProjects/velox/velox/common/caching/tests/AsyncDataCacheTest.cpp:18:
In file included from /Users/amory/CLionProjects/velox/./velox/common/caching/SsdCache.h:19:
In file included from /Users/amory/CLionProjects/velox/./velox/common/caching/SsdFile.h:19:
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:3: error: pointer is missing a nullability type specifier (_Nonnull, _Nullable, or Null_unspecified) [-Werror,-Wnullability-completeness]
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:39: note: insert 'Nullable' if the pointer may be null
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
_Nullable
/Users/amory/CLionProjects/velox/./velox/common/caching/AsyncDataCache.h:305:39: note: insert 'Nonnull' if the pointer should never be null
tsan_atomic<SsdFile * FOLLY_NULLABLE> ssdFile{nullptr};
^
_Nonnull
1 error generated.
[69/968] Building CXX object velox/common/memory/tests/CMakeFiles/velox_memory_test.dir/ByteStreamTest.cpp.o
ninja: build stopped: subcommand failed.
make[1]: *** [build] Error 1
make: *** [debug] Error 2

Hm, that FOLLY_NULLABLE macro should prevent that warning. A short term solution would be to disable that globally by adding the CXX flag -Wno-nullability-completeness)

Hm, that FOLLY_NULLABLE macro should prevent that warning. A short term solution would be to disable that globally by adding the CXX flag -Wno-nullability-completeness

Thanks for the short-term turnaround, @assignUser. Is there any other place (e.g. ~/.zshrc) to add this CXX flag?
It works when I add this line to the root CMakeList.txt file:
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-nullability-completeness")

FOLLY_GNU_DISABLE_WARNING

@Yuhta  Good to know.
Can you advise how to use the macro FOLLY_GNU_DISABLE_WARNING? If it still involves code change, I'd prefer less invasive way of updating the CMakeList.txt.

We now have m1 ci which passes. Is this still an issue for anyone here?

Stale


Describe efficiently creating a Vec of zeroes

I wonder if this needs to be said. How else would you intialize a Vec of zeroes?

I've seen a developer use .resize(0, length), realize it's too slow, change it to mem::uninitialized, and end up with a security vulnerability. I wrote an entire blog post about this.

The standard library documentation did add a notice about it after my blog post - open the Vec docs and Ctrl+F "This may be more efficient than performing allocation and initialization in separate steps, especially when initializing a vector of zeros:"

The fix for the bug you reported:

https://github.com/ruuda/claxon/commit/fec24678c7086aa4b2528bd7542f31d978b81b90

It replaces unsafe code. Using std::mem::uninitialized (deprecated) and new std::mem::MaybeUninit are also unsafe.

This is a good suggestion. I'm picky about the text of the perf book so I ended up rewriting the paragraph from scratch, in https://github.com/nnethercote/perf-book/commit/3e165fbb081e14afe6d50ed2df7e7300e209b552, so there is no need to merge this PR. Thank you for the contribution!
BTW, from your blog post:

There is a tool that can detect reads from uninitialized memory, called Memory Sanitizer, but it currently doesn’t work with Rust standard library. So unless you completely avoid using Rust standard library, there is no tool that let you detect reads from uninitialized memory in Rust.

Valgrind's Memcheck tool detects unsafe uses of uninitialized memory. Also, miri might be able to as well, though I'm not sure about that.

Since the wording is being changed anyway, should I create issues instead of pull requests in the future?

Probably makes sense, yes.


Construct Chart Label

Sometimes the title prop on the axes component leaves something to be desired, so this PR adds a customizable label that you place where ever:
Before

After

Addresses #883 and #542

Bump for review? @benshope

Please add support for x and y (in pixels) instead of percentages.


FetchRewrite: Make sure query symbols are not additionaly fetched

Queries like:
select x + x from t order by x

Resulted in a sub-optimal plan: x was once retrieved in the
query-phase and a second time as _doc['x'] in the fetch-phase.
This was probably introduced by 5f0ee9dfca66f905797e672eabdc07f58c4d7501

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@8644a0a). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master    #5425   +/-   ##
=========================================
  Coverage          ?   82.34%
  Complexity        ?    11282
=========================================
  Files             ?     1359
  Lines             ?    44305
  Branches          ?     5559
=========================================
  Hits              ?    36482
  Misses            ?     5216
  Partials          ?     2607

Impacted Files
Coverage Δ
Complexity Δ

...ain/java/io/crate/planner/fetch/FetchRewriter.java
96.96% <100%> (ø)
12 <5> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8644a0a...87fd938. Read the comment docs.


Fix typo

While browsing the API documentation I've found a minor typo in the HealthReport.Status property.

Thanks!

 I'll try to contribute with whatever little is possible 😸


Variants and css prop styles do not apply

Bug report
Describe the bug
Div component variants and css prop styles do not apply.
To Reproduce
https://codesandbox.io/s/stitches-bug-su9fq?file=/src/App.tsx
Expected behavior
Variant and css prop styles are applied.
System information

Version of Stitches: 0.1.0-canary.12

Additional context
If multiple stitch components have the same style, the component's base CSS rules will be duplicated and will take precedence over the additional CSS rules.

Try this approach instead, it should give you the behaviour you're looking for here:
const Div = styled('div', {
  margin: 8,
  variants: {
    bg: {
      pink: {
        backgroundColor: 'pink',
      },
      red: {
        backgroundColor: 'red',
      },
      green: {
        backgroundColor: 'green',
      },
    },
  },
  defaultVariants: {
    bg: 'pink'
  }
})

Hmm. This most definitely shouldnt be happening.
Looks like we have an issue with the CSS injection of base styles. It gets added twice, and the secon time later in the stylesheet, so it overrides the variants.
Here's the generated CSS
.sxbjjes {
  margin: 8px;
  background-color: pink;
}
.sxbjjesy27ki--bg-red {
  background-color: red;
}
.sxbjjesfqayk--bg-green {
  background-color: green;
}
.sxbjjes2qnvx--css {
  background-color: lime;
}
.sxbjjes {
  margin: 8px;
  background-color: pink;
}
.sxbjjeszz1ir--typography-md {
  font-size: 1.25rem;
}
.sxbjjesibggw--css {
  background-color: skyblue;
}


Add possibility to write settings

i.e., set the whole set of settings for a particular service.
We're using this already in https://github.com/opengisch/qgis-pg-service-parser-plugin/

are in you in the mood of adding tests to that repo?


Icon request: Restream

URL for the brand / site / app
https://restream.io/
Logo resources for the icon
https://restre.am/media-kit
Proof of the Aegis-compatible 2FA
https://support.restream.io/en/articles/870429-two-factor-authentication

Anything else?
No response
Confirmation

[X] I have read and followed the issue guidelines & issue form.

Here you go!
https://github.com/aegis-icons/aegis-icons/blob/master/icons/1_Primary/Restream.svg

:question: How to add icon(s) to Aegis?
:star: Support this project by giving a star! Just adding like to this comment is also appreciated.


Include limits quota in section titles

Related issue
https://github.com/CartoDB/cartodb/issues/14924

Acceptance steps:
Log with 'Professional' user:

[ ]  Go to homepage: Check that maps and datasets sections have total/limit number of items in title.
[ ]  Go to maps/datasets page: Check that title have total/limit number of items in title.
[ ]  Go to maps/datasets page: Select item check that changes correctly to bulk actions section/Deselect item and check that title and total/limit number display properly

Log with 'Enterprise' (non-Professional) user:

[ ]  Go to homepage: Check that maps and datasets sections don't have total/limit number of items in title.
[ ]  Go to maps/datasets page: Check that title doesn't have total/limit number of items in title.
[ ]  Go to maps/datasets page: Select item check that changes correctly to bulk actions section/Deselect item and check that title displays properly

Acceptance 🆗


Fix Config.idFor() recursive ID lookup in anonymous fields (fixes #732)

Prefix was not being propagated in second pass so was only correct at
first level where pref == ""

Thanks for addressing this issue! Can you please add a test case as well?


Feature Request: Hide servers from your server list.

Would be nice if you could hide servers from your server list and unhide them. Makes it look cleaner and is individual to each user.

Also being able to label the servers with a tag.

This is not something I will be building into the core of this application.
The best solution would be create a second non-admin account for your personal servers for organization so that your admin account is not directly tied to them.


Update dataset.proto

fix comment of the rotation matrix calculation from the angular velocity as well as the euler angle
fix the description of intrinsic rotation according to the definition https://en.wikipedia.org/wiki/Euler_angles#Rotation_matrix

Would it be possible for you add a utility function for this?

@LucasSloan
Sure. I will create a new PR for it.


Add remaining fields to Chain entity

Closes #92

Adds remaining fields to Chain entity
Adds respective JSON schemas to the fields
Adds respect OpenApi schemas to the new fields

Coverage increased (+1.3%) to 90.976% when pulling 97cc5d3a0031a1f19055726d7211a11de12b411a on chain-model into 91a84cfacc55ec1c607528925e1d34a9d15e898f on main.


"Ohaba" is listed in the drop-down menu

Description:
"Ohaba" is listed in the drop-down menu alongside its corresponding county and commune, according to the law
Precondition:
The website is up an running.

Step 1
Type "Ohaba" in the search bar.

Expected results
The location is listed in the drop-down menu as "Ohaba, Alba (Ohaba)".


[SPARK-48495][SQL][DOCS] Describe shredding scheme for Variant

What changes were proposed in this pull request?
For the Variant data type, we plan to add support for columnar storage formats (e.g. Parquet) to write the data shredded across multiple physical columns, and read only the data required for a given query. This PR merges a document describing the approach we plan to take. We can continue to update it as the implementation progresses.
Why are the changes needed?
When implemented, can allow much better performance when reading from columnar storage. More detail is given in the document.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
It is internal documentation, no testing should be needed.
Was this patch authored or co-authored using generative AI tooling?
No.

Merged to master.

Alright, I had an offline discussion with the PR author. While it looks fine to me, the author would like to have some more feedback and discussion. Respecting that, I will revert and reopen this.

I read through the proposal and some thoughts:
It would be really useful to add to this PR a list of ways nested structs (struct-of-structs) and array-of-structs can be represented because this is not clarified. From my understanding, a nested object path value can be represented as either a fully "typed_value", or a variant within a variant (containing a required value/metadata, and optional paths), or by directly nesting into the paths (a.b) without introducing an intermediate definition level.
Looks like the current proposal for columnarization works well if the data in a file mostly has one global structure. However, for heterogenous data sources or data sources with fields that alternate between more than one type of value occasionally, it seems there are limitations such as potentially needing to store the value in the top level value field bag if there is a single type conflict for some deeply nested path.

I would like to propose an alternate way to encode the data that allows for more flexibility in representing nested structures and also allows for more efficient encoding of the data.

I propose that simplify the design we require every path part to be immediately be followed by a definition level ($typed_value_*/$untyped_value_variant) that indicates the type of the value at that path, allowing for a fully recursive definition of the variant type as union of the types observed at each path.
I also propose that we allow storing the key paths in an untyped value variant separately as a native parquet list to enable field membership checks without having to scan the metadata. In my proposal, the metadata fields are also made optional, which if not present, means that the metadata is encoded in the value.

Simplest variant example representations, according to my proposal:
optional group message { // message: variant (untyped)
    optional group $untyped_value_variant {
        optional binary value;
    }
}

optional group message { // message: string (typed)
    optional binary $typed_value_string;
}

Nested struct example (w/ subcolumnarized paths, nested type conflicts) representation, according to my proposal:
optional group a {
    optional group $typed_value_object {
        optional group b {
            optional group $typed_value_object {
                optional group c {
                    optional group $typed_value_object {
                        optional group d { // d: string | untyped (value+metadata)
                                optional binary $typed_value_string;
                                optional group $untyped_value_variant {
                                     optional binary value;
                                     optional binary metadata; // make metadata optional, if not present, it is included in the value
                                     optional group metadata_key_paths (LIST) { // also allow to optionally store the list of flattened paths in the value as parquet array to enable  dictionary encoding / bloom filters for fast lookup without having to scan the metadata.
                                        repeated group list {
                                            optional binary element;
                                        }
                                    }
                                }
                        }

                        optional group e { // e: untyped (value) | object (subcolumnarized paths e.x, e.y)
                            optional group $untyped_value_variant {
                                optional binary value;
                            }
                            optional group $typed_value_object {
                                optional group x {
                                    optional binary $typed_value_string;
                                }
                                optional group y {
                                    optional int64 $typed_value_int64;
                                }
                            }
                        }

                        optional group f { // f: int64
                            optional int64 $typed_value_int64;
                        }
                    }
                }
            }
        }
    }
}

NOTE: To reduce the nesting in cases where a field is only present as a single type, a short form could be introduced that allows concatenating the definition level into the path name, making the simplest example representation even compacter:
optional string message.$typed_value_string;

I suggest moving the implementation of Variant to a separate repo outside of the Spark project. I see the usage of "Open Variant" instead of "Spark Variant" in recent announcements. Other projects and table formats like Delta Lake and Apache Iceberg have adopted or are considering adopting this format, respectively.
Can you clarifily if this is not a Spark specific internal Variant, because it so it would be useful in terms of specification and implementation, and interoperability for this to not be embedded inside of the Apache Spark project.

Hi @shaeqahmed, thanks for your detailed response. Your suggestions add a lot of flexibility to the shredding scheme! At the same time, we are wary of adding complexity that could be a burden on implementations to support. Our expectation is that the primary candidate for shredding is data with a fairly uniform sub-structure. In particular, this assumption simplifies the shredding decision and behavior. With a more flexible shredding scheme it also becomes more difficult to decide what to shred since there are significantly more viable options with nuanced tradeoffs. Simplicity in implementation and user-observed behavior is very important to us.
Since the benefits of shredding are both data and workload dependent, could you help us understand concrete query examples for your suggested features? Do you have particular use cases where you expect to write a non-uniform shredding scheme, and get a significant performance benefit?
A few specific points:

In my proposal, the metadata fields are also made optional, which if not present, means that the metadata is encoded in the value.
Do you have specific typed/untyped combinations in mind that are common based on your experience? Adding options to the spec increases the implementation complexity (readers need to support both versions to function correctly), and we’d like to explore the impact of these choices more concretely.
Our motivation for combining the metadata and value is to reduce the size of the Parquet schema. Large schemas can be quite a performance burden due to how Parquet stores its footer, especially for selective queries.

metadata_key_paths
Do you have a concrete query in mind for this feature? My understanding is that it is redundant, and readers could safely ignore it. We’ve purposely designed shredding without redundancy to avoid unexpected increases in storage. In the RFC for Delta here, it mentions struct fields which start with _ (underscore) can be safely ignored. I think we could add that to the spec here, and perhaps reserve _metadata_key_paths as a keyword for a future addition to the spec. As long as readers ignore fields with underscore, it shouldn’t cause any backwards compatibility issues.

union of the types observed at each path
I’d be interested in understanding the expected use cases. Distinguishing different types for scalar fields does not seem to add much value compared to storing mismatched types in an untyped_value column, and adds complexity to the spec and implementation. Could you highlight an example or query pattern where having different typed-values would provide significant benefits over a single typed-value? Our assumption is that one of the types will be most common and shredding should focus on that one.

Hi @Samrose-Ahmed, our intent is for this to be an open format that other engines can adopt. We're aiming to put common code in a Java library under common/variant, so that other engines can reuse the core operations. At this point, I think it makes more sense to leave it in the Spark repo while we iterate on the implementation. We can choose to pull it out into a separate repo later.

Thanks for the response and feedback @cashmand!
Can you please include an example on the document which clarifies how the current proposal deals with nested structs and the general way deeply nested data is expected to be converted from JSON -> the shredded variant form? It is not clear how a nested key path with an object value should be encoded as there are 2-3 implicit ways this can be done IIUC, each with their own tradeoffs: either by adding it as a nested variant (assuming this is supported), or adding the struct directly as a nested key path within the existing path/paths.*, or maybe adding it as a struct typed_value (assuming this is not supported, but it is not actually stated whether or not it is allowed to have a typed value as a struct, and if typed values can be nested within typed values?). Elaborating on this would be useful for readers to understand how this proposal deals with the different cases of structs of structs with structured and unstructured parts.

The real world use case I have in mind is semi structured log analytics, particularly on data that comes from upstream sources that contain heterogenous loosely typed data. A good example of this is AWS Cloudtrail Logs (https://www.databricks.com/blog/2022/06/03/building-etl-pipelines-for-the-cybersecurity-lakehouse-with-delta-live-tables.html), which has variant fields like requestParameters and responseElements whose schema shape is largely relational but directly determined by the AWS service  (of which there are a few hundred, the cardinality of the eventProvider field) that a given log row belongs to. Fields like requestParameters and responseElements also contain arbitrary user input that is completely unstructured and such key paths' data should ideally end up stored in an untyped blob field, while all other key paths should be subcolumnarized for performance in analytical queries. The current proposal makes it difficult to encode this data in a subcolumnarized way as there is no single global schema that can be inferred by reading the first N rows from a large batch (e.g a file).
I agree that having more than one typed field for a given key path per smaller batch of rows (e.g. ~10,000) is not necessary,
but the reason for adding this flexibility to the variant representation is that the current proposal does not allow for taking a series of row batches representing different locally discovered schemas and unioning them together to form a file containing a large batch of rows (256MB-10GB) efficiently and without type conflicts. The idea is that the writer should group the rows in smaller batches and sort in a way that is designed to place similarly shaped data closer together in the file.
My proposal is inspired by some of the state of the art research done for the Umbra database in the JSON Tiles paper (https://db.in.tum.de/~durner/papers/json-tiles-sigmod21.pdf), which describes a columnar format and approach designed to exploit implicit schemas in semi structured data and popular real world implementations of Variant such as that in Apache Doris Lakehouse (https://github.com/apache/doris/issues/26225).
In our case for open tables, Parquet v1/v2 has some limitations that must be kept in mind like extra overhead associated with wide tables / too many definition levels (> tens of thousands of columns) and the inability to have a separate subschema per row group which can result in sparse null columns. However, it is still possible to take advantage of subcolumnarization on heterogenous data if the data is laid out correctly so as to maximize RLE on null arrays and using a compact representation that doesn't require an extra definition level (e.g. x.typed_value in the current proposal) for value paths that have no conflicts in a file.

Hi @shaeqahmed, sorry for the delay, and for not replying earlier about how nested structs are handled. I’ll try to update the doc with an example, but in the meantime, the plan is to support two of the cases you described:

Adding the struct directly as a nested key path within the existing paths structure is meant to be the primary approach. The example at the end of the doc shows an array-of-struct with this form, but a struct-of-struct would look the same. At any nesting level, if a given key doesn’t exist in the parquet schema, it would be stored in the top-level value binary. A request for any non-leaf field would require checking the top-level value, and merging the result with the shredded values (as described in the pseudo-code in the PR).
Adding a nested key path as a nested Variant is supported. This is indicated by just including untyped_value, with no corresponding typed_value. But in this case, it wouldn’t be possible to recursively shred the nested value.

Please let me know if the above is clear, or if I’m misunderstanding the question.
Thanks for describing your use case and the papers you’re referenced. The CloudTrail use case makes a lot of sense, and is definitely one that we should consider carefully. For the current approach, I think it would make sense to shred a field like requestParameters as a Variant binary. This would provide a lot of the benefit, since queries on requestParameters would not need to fetch the top-level binary or any other columns.
I can see that the more flexible schema you’ve proposed could provide better performance for some query patterns, though. At the same time, we’d like to aim to minimize the complexity in the spec, the Parquet footer, and implementation.
I’d like to spend a bit more time looking at the papers you’ve linked to, and considering the trade-offs between the proposals. Can you give us a better idea of what type of queries you expect to see on the read path, and how your scheme would benefit? E.g. would you expect to typically see a mix of queries that need all of requestParameters, and others that only need a field or two? What type of query is likely to benefit significantly from shredding different types (e.g. integer and string) vs. just shredding the most common type, and fetching the rest from the binary? We would like to better understand how the shredding scheme will improve read performance for your workload. Thanks!

Thanks @cashmand and yes that is clear. The type of queries one would expect to see on AWS CloudTrail requestParameters and responseElements would both be  (needle in haystack) queries that would filter on just a few nested field paths  corresponding to the AWS service, returning all columns and analytical dashboards that aggregate/summarize on a few fields such as queries for service level dashboards (e.g. AWS S3, etc.) that perform aggregations on some of the sub fields pertaining to each service within requestParameters and responseElements.
An example of the first type is the following AWS repository containing a variety of search oriented security detection rules: https://github.com/sbasu7241/AWS-Threat-Simulation-and-Detection.
Both of these types of queries (highly selective search & analytical) would benefit massively from shredding/subcolumnarization over being stored as a binary variant due to the compression, statistics, and reduction in data scanned. Note that these fields can contain massive user input strings blobs coming from user input (e.g. a stringified SQL query string as a request parameter https://docs.aws.amazon.com/athena/latest/APIReference/API_StartQueryExecution.html#athena-StartQueryExecution-request-QueryString) alongside compact low cardinality or numerical fields which are useful in a query or a dashboard (e.g. viewing distinct requestParameters.policyDocument.statements[*].action or a search like  requestParameters.ipPermissions.items[0].ipRanges.items[0].cidrIp == "<IP_ADDRESS>"), which is why shredding is important for performance on this type of semi structured field.

Hi @shaeqahmed, I updated the scheme based on the discussion above, while still trying to keep the scheme relatively simple. At a high level, I added the option to define one or more of object, array, typed_value or untyped_value at each path segment (including at the top level, rather than having the one-off value/metadata). This provides the flexibility to union multiple schemas, and avoids the problem of having to fetch the top-level value to determine if an intermediate path was only partially shredded.
We decided to allow only one typed_value at each level, rather than providing one per type. The storage overhead of storing alternative scalar values in untyped_value should be fairly low after encoding/compression, and it should still be possible to define custom stats/metadata schemes later if that turns out to be useful for filtering applications.
Please take a look, and let me know if you have more feedback.

Hi @shaeqahmed, I think the ability to read with older engines is not a goal, and I don't think it's worth adding extra complexity to the scheme to allow it. I can update the document to make that clear.
I also don't think there's much benefit to collapsing of the levels in the schema. It adds extra complexity to parse and handle that case, and ensure that the meaning of a name can't be ambiguous. I don't think it really saves much, if anything - the number of column chunks won't change, and in the current scheme, marking the intermediate groups as required instead of optional should result in the same column chunk size as if they had been collapsed.

@cashmand Ah that makes sense, since marking those intermediate columns as required means the writer does not have to write an extra definition level. Thanks for updating the doc, this looks good to me!

Merged to master.

@Zouxxyy, thanks, these are great questions, which we don't have clear answers for yet, but I'll give you my high-level thoughts.

Will shredded variant be a new type? Because I see that it is currently a nested and changing Struct type, it is a bit difficult to imagine how to describe it.

The intent is for it to not be an entirely new type. For the purposes of describing the desired read/write schema to a data source, I think we might want to do something like extend the current VariantType to specify a shredding schema, but I don't think most of Spark should need to consider it to be a distinct type.

For the write side, how is the shredding schema generated adaptively? From the description in the document, it looks dynamic, is it at the table level / file level / or even rowGroup level? And, I see that many layers of nesting are currently designed, does this have an impact on the write overhead.

The intent is to allow it to vary at the file level. (I don't think row group level is an option for Parquet, since Parquet metadata has a single schema per-file.) The exact mechanism is still up in the air. We could start as simply as having a single user-specified schema per write job, but ultimately I think we'd like to either see Spark determine a shredding schema dynamically, or provide the flexibility in the data source API to allow connectors to determine a shredding schema.

For the read side, if it is a file-level schema, how should spark integrate it when reading. For example, if we want to obtain a certain path, but if the schemas of different files are different, how should we determine the physical plan.

Also a tough question. I think we'll either need the Parquet reader to handle the per-file manipulation, or provide an API to allow data sources to inject per-file expressions to produce the data needed by the query. (This could be useful in other scenarios like type widening, which might have data source specific requirements.) We're still looking into what the best approach is here.

@cashmand Got it, thanks for the answer. The reading and writing of shredding scheme is implicit in the variant type, and the user is unaware of it,  which is user-friendly.  looking forward to the final implementation !


Angor Vs. Traditional Crowdfunding

An article comparing Angor Vs. Traditional Crowdfunding.

When writing file names, use - instead of spaces and try to use only lowercase English letters. Please refer to my posts.