[SPARK-10978] [SQL] Allow data sources to eliminate filters

This PR adds a new method unhandledFilters to BaseRelation. Data sources which implement this method properly may avoid the overhead of defensive filtering done by Spark SQL.

Merged build triggered.

Merged build started.

Test build #44768 has started for PR 9399 at commit 16f3ca3.

Test build #44768 has finished for PR 9399 at commit 16f3ca3.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44768/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #44776 has started for PR 9399 at commit fec7d25.

Test build #44776 has finished for PR 9399 at commit fec7d25.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44776/
Test PASSed.

One more consideration for this improvement, as we probably need to optimize the filters by folding the expression, as the partition keys are actually are the constant value in execution, simply adding the unhandledFilters probably does not work for partition based data source. So I am wondering if we can leave the unhandledFilters and handledFilters to data source implementation itself, we can provide the utilities or the default implementation for the common operations within the buildScan.

@chenghao-intel Can you give an example showing unhandledFilters is insufficient? Also, regarding "So I am wondering if we can leave the unhandledFilters and handledFilters to data source implementation itself, we can provide the utilities or the default implementation for the common operations within the buildScan", can you explain it?

Actually I am talking that it probably give us some troubles in getting the unhandledFilters if we planned to optimize the cases where partition keys combined in the filter, as the partition key will be constant value during the execution for EACH PARTITION, and we may not able to make it filterable in planning stage, at least the code will be more complicated, and besides, I don't see too much benefit if we exposed the unhandledFilters for DataSourceStrategy, so I am suggesting if we can leave the unhandledFilters for buildScan, as unhandledFilters can be considered as a private/protected method in BaseRelation, or we can provide some common operation helper functions to simplify the implementation for the new data source developers.
Sorry if I missed something.

@chenghao-intel Could you please give an example?

Oh, for example: let's say we have the table src (key, value) partition (p1)
For the query like "SELECT value FROM src WHERE key > p1",
And we assume the p1 candidates are 10, 100, and the key range is (0, 50).
-- unhandledFilter = Array.empty
This probably fail in key > 10 (p1 = 10), as we may not able to filter records during the scan, before we taking out all of the records, or in buildScan, we should add an extra filter operation on RDD[Row].
-- unhandledFilter = key > p1
We will loss the optimization for partition (p1 = 100), since the concrete filter is key > 100, and we should always return RDD[Row].empty, as the range of key is (0, 50).
I mean it will be confused to the new data source developers, how to define the unhandledFilter. as the partition key is not treated like the normal attributes, at least it requires more work in getting the  concrete value and multiple filter in the planning stage for different partition keys, what's the unhandledFilter supposed to retrieve?
On the other hand, I am not sure if it's really necessary to expose the unhandledFilter, as it's will be new API for data source that the developer should be aware for optimization purpose, but, we we pass down the filters via API def buildScan(requiredColumns: Array[String], filters: Array[Filter]): RDD[Row] and its variants already. Splitting the filter expressions into 2 parts, and executed in different operators (DataSourceStrategy and DataSource impelementation) seems making thing more complicated, despite we will do the splitting in the data source implementation, but probably not wise enough to expose that externally.

Sorry, I am challenge this as it's about the API, which probably difficult to change back once it's released, and we'd better think further, by adding the partition key cases.

@chenghao-intel Thanks for the comment. That's a good point and I didn't consider this situation when writing this PR. However, fortunately we don't even try to push down predicates that reference any partition columns (see here). In general, when implementing a data source, developers shouldn't worry about partitioning. A HadoopFsRelation data source is only responsible for returning data within a single partition, the query planner does the rest including partition pruning. So I think this is fine?

Merged build triggered.

Merged build started.

Test build #44811 has started for PR 9399 at commit b658aaa.

Test build #44811 has finished for PR 9399 at commit b658aaa.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44811/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #44814 has started for PR 9399 at commit 7c17dd1.

Test build #44814 has finished for PR 9399 at commit 7c17dd1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44814/
Test PASSed.

Ok, actually I was planning to optimize the expression with partition key, which will introduce the ConstantFolding, as the partition key will be a constant value in runtime.
I know, for DataSource developer, the HadoopFsRelation will only return the single partition data as the RDD, that's what my question, how to define the unhandledFilter, will the filter with partition key always be part of the unhandledFilter?

unhandledFilter will not see filters using partitioning columns.

Ok, thanks for explanation.

Overall LGTM. Once we update the FilteredScanSuite, we are good to go.

Merged build triggered.

Merged build started.

Merged build triggered.

Merged build started.

Test build #44928 has started for PR 9399 at commit 326ea24.

Test build #44927 has started for PR 9399 at commit ddac7ac.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44929/
Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Test build #44933 has started for PR 9399 at commit 92dfc55.

test this please

Merged build triggered.

Merged build started.

Test build #44934 has started for PR 9399 at commit 92dfc55.

Test build #44928 has finished for PR 9399 at commit 326ea24.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44928/
Test PASSed.

Test build #44927 has finished for PR 9399 at commit ddac7ac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44927/
Test PASSed.

I will merge it once it passes jenkins. Let's have a test to make sure those handled filters will not show up in the Filter operator.

Test build #44933 has finished for PR 9399 at commit 92dfc55.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44933/
Test PASSed.

Merged build finished. Test PASSed.

Thanks! Merging!

Let's also have some test cases that having a column that is used in handled filters as well as in unhandled/unconvertible filters.

Test build #44934 has finished for PR 9399 at commit 92dfc55.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44934/
Test PASSed.


migrations: Merge 0517 as a no-op.

Migration 0517 migration was already run as 0497 on main, but was accidentally omitted on 8.x until this point.
Merge the 0517 migration into the migration history.  It is included as a no-op in main because it has already run as 0497.

Self-review checklist

[ ] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

Merged, thanks @alexmv! I posted on the production help thread, but I don't see any cleaner options.


Stop running debug artifacts checks on MacOS

These checks take a while, and there's no specific reason to run them on MacOS in particular. We run them only on ubuntu-latest instead.
Merge / deployment checklist

[x] Confirm this change is backwards compatible with existing workflows.
[x] Confirm the readme has been updated if necessary.
[x] Confirm the changelog has been updated if necessary.

Sorry, I'd set this to auto-merge and it merged as soon as you approved 😆


Able to add schedule to person, cannot delete schedule

Need to change schedule from set to list to be able to do so

The new feature looks good
Will review again after the checkstyle tests are passed


How to detect viewport direction or path?

I just found this plugin is simpler, and easy to use.
But i wanted to detect the viewport direction, e.g:

if element leave viewport from top, add class slideUp
if element leave viewport from bottom, add class slideDown

Thank you

That would be a nice addition. Any chance you'd make a PR for that?


Api check fails.

Hey, i’m getting an API check fail error when running the game. ImGuiRenderingPlugin.dll is in the game’s data/Plugins folder, and cimgui.dll is in the BepInEx/plugins folder.
I don’t quite understand how to use PluginsPatcher.exe to fix this. I tried using it from command line, with command
PluginsPatcher.exe ImGuiRenderingPlugin.dll “full path to game data folder”
It said “Success!” but when I launched the game again, still nothing had changed and it didn’t work. The program is not very verbose so I was unable to debug what the problem could be. Any help would be appreciated.

Last passage in the README.md:

Usage: Place cimgui.dll and ImGuiRenderingPlugin.dll into the game's Data\Plugins folder (Either X86 or X86_64). Place Trainer.dll and System.Runtime.CompilerServices.Unsafe.dll into BepInEx\plugins folder (Some Mono Unity versions will also require System.Runtime.dll) These libs are included in the BepInEx Dependencies.rar file under the libs folder.

Are you sure you did that correct?
Because your text

ImGuiRenderingPlugin.dll is in the game’s data/Plugins folder, and cimgui.dll is in the BepInEx/plugins folder.

states otherwise.


ci: add release action and scripts

This PR adds a GH action which watches merges to the main branch. It creates a new version when changesets are detected and bumps the version accordingly.
Typical flow:
Feature work

Changes are made in a working branch and it is deemed that a version (patch/minor/major) bump is needed.
On the working branch run yarn changeset and completes the prompts.
Then push the generated changesets to your working branch.
Merge working branch as you would normally, after getting reviews and CI passes.

New version and releasing

When feature work is merged into main there is a release GitHub action which will run.
It runs  yarn version and incorporates the changesets into the changelog.
It bumps the version accordingly. If changesets contain minor changes then the minor version is bumped.
The release action runs yarn release, which runs the tag command because we are using a different tool (Shipit), to publish our package. It creates a vX.X.X tag, which can then be picked up by Shipit.

The tag command creates git tags for the current version of all packages. The tags created are equivalent to those created by changeset publish, but the tag command does not publish anything to npm.
This is helpful in situations where a different tool, such as pnpm publish -r, is used to publish packages instead of changeset. For situations where changeset publish is executed, running changeset tag is not needed.
The git tags in monorepos are created in the format pkg-name@version-number and are based on the current version number of the package.json for each package. Note that in single-package repositories, the git tag will include v before the version number, for example, v1.0.0. It is expected that changeset version is run before changeset tag, so the package.json versions are updated before the git tags are created.

Safety check: when running without any version bumps:

https://github.com/Shopify/core-issues/issues/40727


fix: mapping for C# object date

Description
There was a problem in the recent addition of mapping for date and date-time field to C#.
Following AsyncAPI documentation:

And the RFC documentation:

My simple change rectify this situation.
Checklist

[x] The code follows the project's coding standards and is properly linted (npm run lint).
[x] Tests have been added or updated to cover the changes.
[x] Documentation has been updated to reflect the changes. (not required)
[x] All tests pass successfully locally.(npm run test).

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 2 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.01%) to 92.319%

Totals

Change from base Build<PHONE_NUMBER>:
0.01%

Covered Lines:
5994

Relevant Lines:
6325

💛  - Coveralls

@jonaslagoni I honestly think that the previous change was breaking (making date-time à DateTime in C#) because it can't contains the time zone / offset information. All our microservices using modelina in the code generator need to pass the offset thus making the new version unusable.
Anyway, DateTimeOffset is a "bigger" object than "DateTime" as it contains more information. People who currently use Modelina with DateTime will be able to cast a DateTime to DateTimeOffset and vice-versa as you just have to ignore the offset. For people in my situation, right now there is no possible solution than to roll back or wait for the next version.
I'm not the owner of the project but if it was mine, I would fix it directly in the main branch as soon as possible. :)
I'll let you decide!
P.S: Another solution for user of DateTime only is to change their spec to date instead of date-time.

My only concern is that people currently using Modelina to generate models and are using them, should always be able to upgrade minor and patch version of the tool.
So as long as this code is valid after this PR is merged, then we can do it on the master branch 👍
DateTime dateTime = new DateTime(2008, 6, 19, 7, 0, 0);
ModelinaModel model = new ModelinaModel();
model.dateTime = dateTime;

But if this change, breaks their implementation, i.e. having to now cast the type, then it's a breaking change and we should target next instead (it's released immediately, so it should not change anything for you). You can read more about it here: https://github.com/asyncapi/modelina?tab=readme-ov-file#versioning-and-maintenance

@jonaslagoni
I did the test and your use case do not need casting as it is implicit but on the other side, it needs casting. Depending if their implementation is publishing or subscribing, it will need a change.

So I guess the better thing to do it to create the PR against the next major version.
Updating the PR !

Ignore the extra commits, gonna be fixed by https://github.com/asyncapi/modelina/pull/1837 in a sec

@jonaslagoni I can't find a way to rebase on the next branch... My fork of the main project do not contain the next branch even if I sync it completely. Any idea how I can achieve this?

Because we squash commits, the only problem is that your branch is not up to date with the latest next changes, so should be as simple as merging them into your branch if I recall correctly 🤔

/rtm


Is this abandoned? Fresh install have errors on EVERYTHING (?)

https://github.com/user-attachments/assets/454949e2-b1c3-4116-9ade-f18770c4e664
Basically everything we do on nyoom returns an error, what is this?

2024-08-18.19-15-17.mp4
Basically everything we do on nyoom returns an error, what is this?

I found that in packer repo... maby the errors you are getting is because it's no longer maintained.


Authentication fails if more than 1 smart card is connected

Hi,
I am using a Windows Virtual Smartcard that is backed by my TPM and stores the client key for my OpenVPN tunnel.
And I am using a ReinerSCT smartcard reader with an OpenPGP card to authenticate with win-gpg-agent on my SSH server.
If I use them in isolation everything works great. But if I have both of those smart card readers connected then:

the agent only finds one key, which is refused by the SSH server.
Only if I manually through the device manager disable the Virtual Smartcard, which as far as I can tell is a whole virtual reader, then the agent finds 2 keys, and one of them is accepted from the SSH server:

I would ask that the win-gpg-agent can be configured as to which reader/smart card it should use. And if that is not possible, it should query all readers/smart cards for valid keys and present all of them to the SSH server in expectation that one of them is authorized to authenticate.

I am afraid this is not related to win-gpg-agent as it does not do anything by itself - this is why it is called "Windows helpers for GnuPG tools suite" and why documentation says "Make sure GnuPG works. Create/import keys, setup smart cards, etc.". All helper does it accepts request and sends it (translated) to GnuPG gpg-agent and then translates back returned response. If gpg-agent itself has support for multiple cards ("it should query all readers/smart cards for valid keys and present all of them to the SSH server in expectation that one of them is authorized to authenticate") - you'd get this functionality. If original GnuPG software does not support it - win-gpg-agent would not support it either. By my reading GnuPG does not support this scenario: https://dev.gnupg.org/T1621. Please, make sure that what you want is achievable using GnuPG tools - if it does and something is missing, I am willing to work on adding it. Other wise I do not see how I could help you here.

Thanks for pointing me in the right direction and even linking the relevant upstream issue. Much appreciated!

@rupor-github Just to confirm I get this right on the upstream issue report: win-gpg-agent is including/bundling version gnupg 2.3.2 right now? At least it says v1.3.2 / 2021-08-25 * Changes for gnupg 2.3.2 in the change log.
The status window is broken right now and doesn't display anything:

Oh never mind. I just realized that the gpg agent is not bundled with win-gpg-agent at all and I had installed it separately. Sorry for the noise!

Upgraded to version gnupg-w32-2.3.3_20211012. Issue still exists. Made an upstream issue comment now:
https://dev.gnupg.org/T1621#151776

Mirco, obtaining a version is a simple gpg --version with following string split. If you are running daily build or format of this string changed upstream - it could be empty. I will take care of this in upcoming release, but it is not critical and does not affect what program does.

This is what I get:
gpg-agent --version
gpg-agent[3564]: C:/Users/Markus/AppData/Roaming/gnupg/gpg-agent.conf:2: obsolete option "use-standard-socket" - it has no effect
gpg-agent (GnuPG) 2.3.3
libgcrypt 1.9.4
Copyright (C) 2021 g10 Code GmbH
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

and with
gpg-agent --version 2> nul
gpg-agent (GnuPG) 2.3.3
libgcrypt 1.9.4
Copyright (C) 2021 g10 Code GmbH
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Really strange, but in any case - I modified parsing code in the latest release with is now available. Really hope this will help.


Start built-in server

hello,
please tell me what's this and how to make it enable,
I tried to put START_SSH_SERVER in app.ini in server section but then it wasn't started anymore.

Could you please send the log and your config file?

I didn't know there is a log, so thanks a lot, it was about permission on port 22, I changed it to 3001 and all done.
just one more question
I'm using gitea as a service with a different user in my server, so I want to know does it safe to use ssh and allow its port in my firewall?

Yes, because SSH is not used to gain access to your server because it is used to transfer files via Git. It is simply an alternative to HTTP(S).

thanks a lot


OpenStackDataPlane sample CR for Nova Ceph Config

Add dataplane_v1beta1_openstackdataplane_ceph.yaml based on dataplane_v1beta1_openstackdataplane.yaml but also has extraMounts and a customServiceConfig for nova. These parameters have been confirmed to configure an EDPM compute node which can store its ephemeral data in a Ceph RBD pool. This will also be an example referenced from the documentation [1].
[1] https://github.com/openstack-k8s-operators/docs/pull/36

/lgtm


Add backwards incompatible changes at the top of the release notes draft

Description

This adds any backwards-incompatible changes at the top of the draft release notes that can be generated with the draft_release_notes.py tool.
@zklaus Would it be possible to re-generate the release notes for ESMValCore v2.4 with this version of the script? I realized that backwards-incompatible changes are currently hard to find in the changelog.

Link to documentation: https://esmvaltool--2431.org.readthedocs.build/en/2431/utils.html#draft-release-notes-py

Before you get started

[ ] ☝ Create an issue to discuss what you are going to do

Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

[x] 🛠 This pull request has a descriptive title
[x] 🛠 Code is written according to the code quality guidelines
[x] 🛠 Documentation is available
[x] 🛠 Tests run successfully
[x] 🛠 The list of authors is up to date
[x] 🛠 All checks below this pull request were successful

To help with the number of pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

I thought I already answered this, but my comment appears to have gotten lost. There were 10 backwards-incompatible changes in ESMValCore v2.4. It would be good to clearly list those in the release notes.

I found the comment https://github.com/ESMValGroup/ESMValTool/discussions/2311#discussioncomment-1700452. I guess here would have been a better place to write that.

Yes, we can regenerate the release notes. To be clear, that would be the release notes for the core only, and we don't intend to make a new (bugfix) release just for that, but just have it in the online documentation and then release it with the next version, right?

Doing the release notes is always a bit cumbersome because the entries are built from the PRs only based on date, not on the actual commits in the release branch. One day I'd like to change the release notes script to take those into account.

Yes, we can regenerate the release notes.

Awesome!

To be clear, that would be the release notes for the core only, and we don't intend to make a new (bugfix) release just for that, but just have it in the online documentation and then release it with the next version, right?

Yes

Doing the release notes is always a bit cumbersome because the entries are built from the PRs only based on date, not on the actual commits in the release branch. One day I'd like to change the release notes script to take those into account.

Note that there is some software available for doing this kind of thing, maybe we should look at using that instead of rolling our own

this introduces a fail in the full development tests - multimodel uses std_dev statistic which is not supported by the current development branch, see https://github.com/ESMValGroup/ESMValTool/runs/5094158254?check_suite_focus=true - anyone know what's the hap?


onSaveInstanceState problems

Look
https://github.com/h6ah4i/android-advancedrecyclerview/blob/7e0b4ef282547385dcbe9cbc48913404abafcbe6/example/src/main/java/com/h6ah4i/android/example/advrecyclerview/demo_e/RecyclerListViewFragment.java#L41
this code final Parcelable eimSavedState = (savedInstanceState != null) ? savedInstanceState.getParcelable(SAVED_STATE_EXPANDABLE_ITEM_MANAGER) : null;         mRecyclerViewExpandableItemManager = new RecyclerViewExpandableItemManager(eimSavedState); is bad , because
http://stackoverflow.com/a/25068471
What happened , if fragment with ExpandableStorageAdapter  is used in Pager ?

This is not a library's issue. I think it is a design fault of Android's Fragment...
I recommend you to create a custom ViewGroup which manages saved state of RecyclerView and RecyclerViewExpandableItemManager.

Create a ViewGroup which extends some stock ViewGroup class. (FrameLayout is recommended)
Override dispatchSaveInstanceState() and dispatchRestoreInstanceState() to disable the automatic child view state saving.
Override onSaveInstanceState() and onRestoreInstanceState()  and save/restore the state of child RecyclerView and RecyclerViewExpandableItemManager.

Please refer the following superb article for more details;

Saving Android View state correctly

Thank you, but what you said about this decision: http://stackoverflow.com/a/25068471
I, try, and that work for me
public StorageFragment() {
        setArguments(new Bundle());
    }

and
public void onViewCreated(View view, Bundle savedInstanceState) {
        super.onViewCreated(view, savedInstanceState);
       final Parcelable eimSavedState = (getArguments() != null) ? getArguments().getParcelable(SAVED_STATE_EXPANDABLE_ITEM_MANAGER) : null;
.....

and
@Override
    public void onPause() {
        super.onPause();
        // save current state to support screen rotation, etc...
        if (mRecyclerViewExpandableItemManager != null) {
            getArguments().putParcelable(
                    SAVED_STATE_EXPANDABLE_ITEM_MANAGER,
                    mRecyclerViewExpandableItemManager.getSavedState());
        }
    }

Or that ridiculous  ? : )

I do not recommend using Fragment's arguments for saving instance state.
The StatedFragment library was using that pattern for saving Fragment's state, but now this library is deprecated. The reason of deprecation is described on the following article written by the author of the StatedFragment library.

The Real Best Practices to Save/Restore Activity's and Fragment's state. (StatedFragment is now deprecated)
Probably be the best way (?) to save/restore Android Fragment’s state so far


[DONOT MERGE] test text only

Docs Build status updates of commit cfcb141:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msbuild-api-docs/xml/_filter.xml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


工作的时候提示

vue3-virtual-scroll-list is incorrectly packaged. Please contact the package author to fix.

vue3-virtual-scroll-list is incorrectly packaged. Please contact the package author to fix.

I didn't find this issue, could you please provide a repo?

vue3-virtual-scroll-list 打包错误。请联系包作者修复。

我无法复制，你能提供一个回购吗？

不好意思，我已经解决了


Web accessibility issues

Not working when you try using and enter to operate without mouse.
Faced this issue on chrome and mozilla firefox both.

Hi @maximelafarie ,
Can you please help with this?

Hi @learneradarsh!
You would like to jump to move to tabs with Tab and select with Enter, am I right?

Correct


Add leetcode 3096 solution

I want to add leetcode solution for problem 3096

@VaishnaviMankala19 I close your issue because you need open only 4 issue per day or active not large


tenant id should be provided in the URL

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:
/kind bug
/kind feature

What happened: Currently tenant id is being provided to the workflows in request body
What you expected to happen: It should be available in the URL
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:

Orchestration(release/branch) version:
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

/assign @kumarashit

Changes are merged.


Remote debugging - docker rust container

Which OS: Arch linux (docker)
Which LLDB version (lldb -version): lldb version 4.0.1
Which VSCode version (Help/About): Version 1.14.2
What is the problem and how did you get there:
I'm trying to setup remote debugging with VSCode, lldb-server and rust executable running in docker container.
I set docker container in privileged mode, so now I can successfully start lldb in container.
(lldb) target create target/debug/core_service
Current executable set to 'target/debug/core_service' (x86_64).
(lldb) process launch
Process 182 launched: '/srv/core_service/target/debug/core_service' (x86_64)

Now I have problem with lldb-server, I use lldb-server platform --server --listen <IP_ADDRESS>:1234 in container, I can connect to it with lldb, but there's problem with attaching/creating process.
Below, launch.json and debug console output:
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Remote launch",
            "type": "lldb",
            "request": "custom",
            "initCommands": [
                "platform select remote-linux",
                "platform connect connect://localhost:1234",
                "target create target/debug/core_service",
                "process launch"
            ],
            "sourceLanguages": [
                "rust"
            ]
        }
    ]
}

 Platform: remote-linux
 Connected: no
  Platform: remote-linux
    Triple: x86_64-*-linux-gnu
OS Version: 4.12.4 (4.12.4-1-ARCH)
    Kernel: #1 SMP PREEMPT Fri Jul 28 18:54:18 UTC 2017
  Hostname: 099bd76c07c9
 Connected: yes
WorkingDir: /srv/core_service
Current executable set to 'target/debug/core_service' (x86_64).
error: process launch failed: Connection refused

What did you expect to have happened instead (if not obvious):
Attach/create process succesfully in lldb-server in given configuration.

I am afraid I cannot provide much support for LLDB itself.
However, I would suggest to try these commands in the command line LLDB first:
platform select remote-linux
platform connect connect://localhost:1234
Does platform connect succeed?   Does lldb-server on the other end say "Connection established."?
Also, please note that if you use lldb-server in platform mode, there should be no need to create target and launch the process manually.   You should be able to use  { "request": "launch", "program": "${workspaceRoot}/target/debug/core_service" ... } in the launch config.

Hi, thanks for help anyway.
Yes, i tried with "request": "launch", "attach" and I tried with platform mode, but it didn't worked either.
I documented my efforts here, but in the end I couldn't find a solution. I gave up, but maybe it will help someone in the future, because debugging Rust services in docker with LLDB from VSCode would be nice :)

Maybe it's something about your container's networking settings?   IIRC, lldb-server forks a copy of itself after accepting connection in --server mode.  Not sure how that copy establishes connection to lldb client... Maybe you can try running it without --server?


[ScrollView] How does scroll respnder "steal" the touch lock of a touchable

If I am pressing a touchableOpacity in a list view and then scroll vertically, the touchable event will be terminated.
So I am thinking doing something similar. Basically a panResponder to horizontally swipe to close a side panel and that side panel is a list view of touchableOpacities. But couldn't get the desired result like scrollResponder which "steals" the touch lock. So lots of time the panel is closed but the onPress is called as well.
Any suggestion?
this.panResponder = PanResponder.create({
  onMoveShouldSetPanResponder: this.handleMoveShouldSetPanResponder,
  onPanResponderMove: this.handlePanResponderMove,
  onPanResponderRelease: this.handlePanResponderEnd,
  onPanResponderTerminate: this.handlePanResponderEnd
});

handleMoveShouldSetPanResponder(evt, {vx, vy, numberActiveTouches}) {
  // steal from here?
  return numberActiveTouches == 1 && Math.abs(vx) > 0.2 && Math.abs(vy) < 0.2;
}

For more implementation type questions/suggestions we recommend asking on StackOverflow as you have a better chance of getting helped. Your question is  a little hard to follow, but you should disable scrolling on the scroll view with the prop scrollEnabled
@brentvatne

Yeah this is something that I would like to see fixed too - it would be great if we were able to block the native ScrollView from becoming responder. Dup of #1046 although worded slightly differently :) See my comment at the bottom for more info.


pre submit keeps failing: investigate why it takes minutes for pod to be ready some times

see #533 and many build failures after > 2mins of retry

cc @douglas-reid , @sebastienvas

Is it the pod or just the ingress. Could we be running out of external ip address ?

I don't know but it's really crippling - pretty much 100% failure lately on my PR:
https://github.com/istio/istio/pull/519

I had to recreate the cluster as some firewall rules were missing. Could you try it again?

is this problem solved?

@Brian-Xincheng-Zhang it's a bit less frequent but unfortunately no there is still a problem:
W0808 17:22:47.757] E0808 17:22:47.756137   10751 framework.go:199] Failed to complete Init. Error unable to find ingress

https://k8s-gubernator.appspot.com/build/istio-prow/pull/istio_istio/546/istio-presubmit/253/
ps: those seem more like errors or fatals than warning btw

OK so I am sure we are running out of external ips. I asked for more but this is going to take some time. @mandarjog proposes to use pod ip in the meantime. The e2e framework already support this we just need to add a flag. @kyessenov I am not sure if the pilot e2e supports using the pod ip instead of the ingress. Anyways should we use pod ip instead of ingress ?

For now I don't think there's a reason we need ingress over using pod ip, and I'm all for it if it means we get faster/more reliable test execution. Long term I think we will will want to use ingress resources since Istio plans to do work on ingress (e.g. policy via Mixer on ingress traffic), but we may still be able to get away with Pod IP of the ingress proxy pod in that case. Either way we're not there yet so I don't see a reason to use ingress/assign public IPs today.

@ldemailly to provide background on why we use a global ip.

Use service cluster IP if you want to address Istio services. Istio doesn't support direct access to pods (yet, by design, ...).

We also should be using ingress resources for externally exposed services. We don't have to expose ingress proxy externally.

The way e2e tests are structured, we do not need to expose ingress as LoadBalancer type, because all requests to ingress are made from within the cluster, via kubectl exec  ....
However, I would like to keep the istio.yaml as LoadBalancer type, and use NodePort only when running e2e.
The fix is to add a type field to the ingress template, and set it depending on the context.

We have to wait for @yutongz changes to be in in order to use the NodePort. @Brian-Xincheng-Zhang Can you make sure rbac rules are being updated correclty as this is blocking Yutong PR.

now it's
W0810 04:15:44.548] I0810 04:15:44.547373   10410 demo_test.go:131] Error talking to productpage: Get http://<IP_ADDRESS>:31748/productpage: dial tcp <IP_ADDRESS>:31748: getsockopt: connection refused
W0810 04:15:44.548] E0810 04:15:44.547427   10410 demo_test.go:144] Couldn't get to the bookinfo product page, trying again in 5 second
W0810 04:15:49.688] I0810 04:15:49.686928   10410 demo_test.go:133] Get from page: 404
W0810 04:15:49.688] E0810 04:15:49.686969   10410 demo_test.go:144] Couldn't get to the bookinfo product page, trying again in 10 second
W0810 04:15:59.759] I0810 04:15:59.757391   10410 demo_test.go:133] Get from page: 404
W0810 04:15:59.759] E0810 04:15:59.757431   10410 demo_test.go:144] Couldn't get to the bookinfo product page, trying again in 15 second
W0810 04:16:14.829] I0810 04:16:14.827767   10410 demo_test.go:133] Get from page: 404
W0810 04:16:14.829] E0810 04:16:14.827806   10410 demo_test.go:144] Couldn't get to the bookinfo product page, trying again in 20 second
W0810 04:16:34.899] I0810 04:16:34.897930   10410 demo_test.go:133] Get from page: 404
W0810 04:16:34.899] E0810 04:16:34.897967   10410 demo_test.go:144] Couldn't get to the bookinfo product page, trying again in 25 second
W0810 04:16:59.969] I0810 04:16:59.967835   10410 demo_test.go:133] Get from page: 404
W0810 04:16:59.969] E0810 04:16:59.967862   10410 demo_test.go:144] Couldn't get to the bookinfo product page, trying again in 30 second
W0810 04:17:30.040] I0810 04:17:30.038359   10410 demo_test.go:133] Get from page: 404
W0810 04:17:30.040] E0810 04:17:30.038427   10410 demo_test.go:144] Couldn't get to the bookinfo product page, trying again in 35 second
W0810 04:18:05.109] I0810 04:18:05.107870   10410 demo_test.go:133] Get from page: 404
W0810 04:18:05.109] E0810 04:18:05.107948   10410 demo_test.go:144] Couldn't get to the bookinfo product page, trying again in 40 second
W0810 04:18:45.179] I0810 04:18:45.178009   10410 demo_test.go:133] Get from page: 404
W0810 04:18:45.179] E0810 04:18:45.178031   10410 demo_test.go:144] Couldn't get to the bookinfo product page, trying again in 45 second
W0810 04:19:30.249] I0810 04:19:30.248460   10410 demo_test.go:133] Get from page: 404
W0810 04:19:30.250] E0810 04:19:30.248487   10410 demo_test.go:144] Couldn't get to the bookinfo product page, trying again in 50 second
W0810 04:20:20.321] I0810 04:20:20.320483   10410 demo_test.go:133] Get from page: 404
W0810 04:20:20.322] E0810 04:20:20.320520   10410 framework.go:199] Failed to complete Init. Error unable to set default route

Exactly, we need to solve this first https://github.com/istio/istio/issues/550

This has been resolved by Kuat and Doug.  The only errors that I see were mine, and I was acutally changing the framework.
auth enabled never passes, and rbac + no_auth is flaky. It seems to mostly be the mixer test that is failing.
I did find some istioctl exec format issues, and I am thinking this could be because we might be uploading the file (running the presubmit while doing the smoke test) when we are downloading it. I ll add some condition to check if an image exist before pushing (or even better building) it.

check pilot's ?

I counted less than 5% (3 actual flakes out of 69 runs) flakiness on the run combining mixer-e2esmoktest pilot-e2esmoketest and e2e-suite-no_rbac-no_auth, which I think is pretty good for an e2e test.
Coming back on the exec format, it cannot be an issue when we download a binary that we uploading, it has to be related to the way we download and save the file.

@ldemailly the thing that you opened is for rbac which is not presubmit, maybe it was at some point but by mistake. I would like to close this bug and open specific issues to help resolution. I think we should open different issues on why the rbac and auth test are failing. What do you think ?

exec format error is that the file is actually missing, I guess we are not checking the http code :). Will file a bug for this.

It is an RBAC issue only if the logs shows some error type cannot create/list/... .
Otherwise, the simple fact it is rbac test failing does not indicate an rbac issue.

The only difference is that we apply the rbac yaml the rest of the test is
unchanged. I think it s safe to assume it is rbac related.
Anyway, we should capture those in different issue as I don't see the point
of leaving an endless issue opened.
On Aug 27, 2017 7:51 AM, "Andra Cismaru"<EMAIL_ADDRESS>wrote:

It is an RBAC issue only if the logs shows some error type cannot
create/list/... .
Otherwise, the simple fact it is rbac test failing does not indicate an
rbac issue.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/537#issuecomment-325203201, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AQ80Z-NTLD08G2xZ3thj1QuaefW5qbVsks5scYJ4gaJpZM4Or9y6
.

https://github.com/istio/pilot/pull/1102 for instance doesn't seem to go through

One more time, the issue is with pilot-presubmit not the e2e. So I am closing this. Please open issues for new problems. I will open a new issue for the RBAC one.

not sure the value of closing and opening new issues but sure...


Branch to a NULL target crashes mcsema

Currently at line 142 of /home/pnp/mcsema/mcsema/cfgToLLVM/x86Instrs_Branches.cpp, mcsema asserts:
TASSERT(tgt != NULL, "Branch to a NULL target");
It needn't crash though if a jmp 0 is contained in the binary being lifted.  So, maybe:
if (!tgt) return EndBlock;

Can you provide an example binary that triggers this?

Closing for now, please re-open if there is still an issue.


Add flag to register entities created

This flag will allow users to take control of the entities created and delete them after execution. Some king of clearing database after test.

@jorgebodega Do you have any plans to get this merged in soon? This would be super helpful to tear down seeded data during the execution of several tests.


Docs search only seems to index on main

Describe the bug
Search returns no results on versioned pages
To Reproduce
Steps to reproduce the behavior:
Navigate to the docs, select the main version. Search for something, eg terraform you will see 108 results.

Switch to some other version, eg 0.18.0. Search again, you will see 0 results.

Expected behavior
Search should return results from all versions.

cc @darrylweaver

https://github.com/weaveworks/weave-gitops/issues/3478


[BugBash][CLI]descriptions of hub commands are not clear enough

Describe the bug
[BugBash][CLI]descriptions of hub commands are not clear enough
To Reproduce

view the help of comand az webpubsub hub create -h. better to have:

--allow-anonymous: include the default value in description. And the input false/true is not consistent with the actual output allow/deny. Better to explain their relationship.
--event-handler: better to clearly explain that user-event-pattern is a string. If included mutliple times the last will be used while others ignored. But for system-event, it's a array, all inputs will be combined together as an array. Besides, it doesn't take   a string seperated by comma or other chars.

Exceptions (if any)

Further technical details

SDK and its version you use

https://github.com/Azure/azure-cli-extensions/pull/4049


Let division by zero be error

parameter DIVISION_BY_ZERO=1/0; // will happily compile and "work"

That is not a bug — according to Verilog semantics, this will just give an undefined result (ie. 32'hxxxxxxxx).  As far as I can see, yosys is doing the right thing here.


cant create new filters

For whatever reason adding more filters will make cypress fail. After adding new filter and commenting out a old one the cypress test didn't fail.

sure you can


Migrate away from ImprovMX for mailing lists

We currently use ImprovMX to handle mail sent to @nixos.org (see relevant dns entries).

We only use ImprovMX for mail forwarding (teams like infra@, marketing@, etc). Today, nobody sends mail from @nixos.org, and nobody has any inboxes.
You need a web account with ImprovMX to see and to update these mail forwards. The Nix community can't see/audit any of this.
There are various limits (number of forwards, perhaps the number of emails an address can forward to?). See https://improvmx.com/pricing/. I don't know if we're currently paying for ImprovMX. I think I heard that we've run into some of these limits.

The plan
A few weeks ago, @Mic92 asked me to look into self hosting this instead. He recommended Simple NixOS Mailserver (SNM). I've played with it a bit, and it does seem like a good fit here.

[ ] Install SNM on umbriel.

The configuration docs here are great: https://nixos-mailserver.readthedocs.io/en/latest/setup-guide.html.
Leave mailserver.loginAccounts empty, and disable pop/imap.
Port the existing mailing lists from ImprovMX to mailserver.forwards

@Mic92 has posted a dump [REMOVED] (accurate as of 2024-09-30).

[ ] Verify this server can successfully send mail (target: 10/10 on https://www.mail-tester.com/). Either by temporarily adding a login account, or speaking directly to postfix via the cli.
[ ] TBD (see below): configure a metrics/alerting solution. Intentionally break emailing somehow and confirm that the infra team gets notified about stale mail stuck in queues.
[ ] Wait until the Nix Steering Committee Election is done: https://nixos.org/blog/announcements/2024/sc-election-2024/.
[ ] Rollout the change:

[ ] Check that mailserver.forwards is up to date
[ ] Switchover the MX records from ImprovMX to umbriel.nixos.org.
[ ] After the MX record change has propagated everywhere (check with https://www.whatsmydns.net/), verify that email forwards still work. If not, switch the MX records back.
[ ] Cleanup: shut down our ImprovMX account, or do whatever we can to reduce confusion about this

Open questions/TBD:

Monitoring

Ideally, the infra team would get alerted if emails have been sitting in a postfix queue for a long time. Are there any best practices for this? We use Prometheus, perhaps https://github.com/kumina/postfix_exporter is a good pick? It's packaged in nixpkgs =)
Anything else?

Backups

Arguably not necessary. This service is pretty much stateless (except for the mail stuck in queues, which perhaps we can live with?)

Alternatives considered

I don't know if there's been any serious discussion about paying someone (ImprovMX or something else) to handle this for us. Since declarative management and audit-ability are important to us, it would either have to be a provider that  has a Terraform provider, or we could build one ourselves.
@Mic92, can you shed any light on this?

I just want to make awareness that you probably need to write a mail to t-online and outlook (none 356) to whitelist your IP otherwise mails cannot be delivered.

I just want to make awareness that you probably need to write a mail to t-online and outlook (none 356) to whitelist your IP otherwise mails cannot be delivered.

I hear you on this. I've never run a mailserver before, and honestly have no idea what our deliverability is going to be like. I believe the current set of emails is quite tiny, and may not even include any t-online or outlook. My personal opinion on this is that we should make sure we've solved the monitoring story: if we get notified for email stuck in queues, then we can tackle these allowlists as necessary, or we can give up and pay someone to handle this for us.

After the leak of the email mappings I would be interested in discussing the privacy aspect of the email mappings.

Sorry about that. I asked one person about this, but should have talked to more people before posting.
Ideas:

We could encrypt the email addresses. This would be hard to code review.
We could seek consent from all the relevant people. I don't know how hard this would be. I don't have the list anymore, but it didn't seem like an insurmountable number.
Do this behind some self-hosted (or paid) webapp with a login. That's basically what we do today with ImprovMX.

I just want to make awareness that you probably need to write a mail to t-online and outlook (none 356) to whitelist your IP otherwise mails cannot be delivered.

For T-Online at least this is just one email after setting up reverse DNS and everything up correctly.

We could encrypt the email addresses. This would be hard to code review.
We could seek consent from all the relevant people. I don't know how hard this would be. I don't have the list anymore, but it didn't seem like an insurmountable number.
Do this behind some self-hosted (or paid) webapp with a login. That's basically what we do today with ImprovMX.

@zimbatm started to ask existing users of email addresses about that.

I hear you on this. I've never run a mailserver before, and honestly have no idea what our deliverability is going to be like. I believe the current set of emails is quite tiny, and may not even include any t-online or outlook. My personal opinion on this is that we should make sure we've solved the monitoring story: if we get notified for email stuck in queues, then we can tackle these allowlists as necessary, or we can give up and pay someone to handle this for us.

Some DMARC and reading the mail logs in case there are delivery problems. I didn't had any big issues with emails for the NixOS wiki and that looks more like bulk messages compared to what I expect to be sent from nixos.org.

@jfly Is it possible to move the email addresses into sops-encoded secrets, or is this part only configurable with plain Nix code?

For T-Online at least this is just one email after setting up reverse DNS and everything up correctly.

And you need to have a proper imprint on the TLD of the rDNS entry and contact means via I think telephone and e-mail that is not going over the mail server.
I have recently done it and it took me a few back and forths but it is doable.

@jfly Is it possible to move the email addresses into sops-encoded secrets, or is this part only configurable with plain Nix code?

It currently requires plain Nix code:

The relevant SNM code sets nixpkgs's services.postfix.virtual.
nixpkgs massages that into services.postfix.config. This code is setting Postfix's virtual_alias_maps setting.

Adding support for encrypted emails seems like it might actually not be too hard:

We could adjust the nixpkgs service to allow for multiple virtual_alias_maps (currently it supports exactly 0 or 1), and then we could add a new entry to that array to point at a virtual alias map generated with a sops-nix template

I think the nixpkgs change I have in mind will look weird. We might need a more generic solution that has a satisfying answer to this question: "why does virtual_alias_maps get this special escape hatch but not other maps like alias_maps?"

Adding a new entry is a little tricky because you actually need to run postmap to "compile" these mappings, but I think the existing services.postfix.mapFiles option is flexible enough to do this for us without changes.

tl;dr:

It's possible, but requires changes to nixpkgs, and perhaps SNM, depending on how we want to expose this.
I'm willing to do this work, but would prefer to wait until we know if it's necessary first.

If it makes sense to do this work, I could use a brainstorm partner on the nixpkgs change.


[SUPPORT] Slow commit times with Spark Structured Streaming from Kinesis to MOR Hudi table

Describe the problem you faced
We could use some guidance on a performance issue we are seeing with a Spark Structured Streaming job reading from Kinesis and writing to a MOR Hudi table in EMR 7.2 (Hudi 0.14.1-amzn-1). We have noticed very long-running commits in Hudi and the iterator age gets behind, up to a day or more, on the Kinesis stream as data flows in.
Steps to reproduce the behavior:
Some details about the load, application, and the Hudi table:

20 shard kinesis stream with around 100-500MB per minute, with periodic spikes up to 2GB, typically with increased load during the work week
Each microbatch appears to consume around 8GB incoming data
MOR, Simple Index table. Currently 100.5 GB with 886 partitions dimension table.  No meaningful, time-based attribute to cluster or partition by.
Record key is a UUID. All writes use the upsert operation. In a given batch, it is very likely that close to all partitions will have an update or insert.
EMR Cluster version 7.2 with 9 m5a.2xlarge workers (plus 1 m5a.2xlarge driver). We scaled this up to 20 and saw some write improvement, but still the same behaviors as I will describe below.

What we have noticed is that deltacommits typically take around 15 minutes, however every few commits we notice an excessively long deltacommit which takes around 1.5-3 hours to complete. While this commit is completing, there are no GetRecords calls to the stream to continue processing. We compared the commit metrics between one of these faster commits and a slow commit and there isn't a significant difference in the amount of data being written, maybe about 30k records. What I observed was some log file writes as low as 16 records and 100KB in size took ~4.5 minutes (totalUpsertTime). This seems an excessive amount of time for a log file (not parquet) with so few records.
Things we tried:

Turn down the frequency of compaction to every 6 hours - no improvement.
Reduce the number of records read per shard from Kinesis (in case it was memory pressure of incoming data) - no improvement
Turn off async archive service - This fixed the periodic slow commit (see below notes), however commits still appear very slow generally (15-20 minutes)
Upgrade to EMR v.7.5.0 (Hudi 0.15) - no improvement

The jobs that took the most time in these slow batches:

Loading latest base files for all partitions (590 tasks at 9.8 minutes)
Getting small files from partitions (590 tasks at 9.7 minutes)
Building workload profile (104 tasks at 13 minutes)
Doing partition and writing data (~4648 tasks at 1.5+ hours, still running but only halfway through). We believe this correlates to the number of files written, so at 72 available cores we can understand this will be slow, however we'd like to improve write speeds here.

What is interesting is the jobs Loading latest base files, Getting small files jobs typically run in less than a minute, yet during this period of the "slow commit," these jobs take many minutes. During the "fast commits," it appears the Building Workload Profile job is where most of the commit time is spent. I do observe shuffle spill to disk, so we could use guidance on improving that as well.
We have conducted some stress testing in a comparable test environment and were able to recreate the issue even starting on a blank table. We did the same stress testing earlier in the year, using EMR 6.15 (Hudi 0.14.0-amzn), and did not perceive this behavior.
What I found in the logs during the time of the slow commits are these logs. During the faster commits, these logs don't show up. I noticed around the same time the Delete Archived instants jobs had run, which prompted me to try to turn off the async archive service.
24/12/03 02:50:15 WARN PriorityBasedFileSystemView: Got error running preferred function. Likely due to another concurrent writer in progress. Trying secondary
24/12/03 02:50:17 WARN PriorityBasedFileSystemView: Routing request to secondary file-system view
Turning off async archive service alleviated the periodic extremely slow commit, however we still believe commit times are lackluster given the load and cluster size.
What other configurations should we look at to tune the write speeds? Do we need to reconsider our table design?
Below are the Hudi configurations in our production environment. Attached are some Spark UI screenshots as well.
hudi-defaults on the cluster:
{
"hoodie.archive.async": "true",
"hoodie.archive.automatic": "true",
"hoodie.compact.inline.max.delta.seconds": "21600",
"hoodie.compact.inline.trigger.strategy": "TIME_ELAPSED",
"hoodie.datasource.write.schema.allow.auto.evolution.column.drop": "true",
"hoodie.embed.timeline.server.async": "true",
"hoodie.schema.cache.enable": "true"
}
Spark writer configuration:
"hoodie.cleaner.policy.failed.writes": "LAZY",
"hoodie.write.concurrency.mode": "optimistic_concurrency_control",
"hoodie.write.lock.dynamodb.region": "us-east-1",
"hoodie.write.lock.dynamodb.table": "oitroot-op1-datasync-hudi-locks",
"hoodie.write.lock.provider": "org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider"
"path": hudi_output_path,
"hoodie.table.name": clean_table_name,
"hoodie.datasource.write.storage.type": "MERGE_ON_READ",
"hoodie.datasource.write.operation": "upsert",
"hoodie.datasource.write.recordkey.field": record_key_fields,
"hoodie.datasource.write.precombine.field": precombine_field,
"hoodie.datasource.write.partitionpath.field": partition_fields,
"hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
"hoodie.datasource.write.hive_style_partitioning": "true",
"hoodie.datasource.hive_sync.enable": "true",
"hoodie.datasource.hive_sync.database": clean_db_name,
"hoodie.datasource.hive_sync.table": clean_table_name,
"hoodie.datasource.hive_sync.partition_fields": partition_fields,
"hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor",
"hoodie.datasource.hive_sync.use_jdbc": "false",
"hoodie.datasource.hive_sync.mode": "hms",
"hoodie.datasource.hive_sync.omit_metadata_fields": "true",
"hoodie.datasource.meta_sync.condition.sync": "true"
Thank you for your assistance!
Expected behavior
Consistent and faster commit times, ideally under 10 minutes
Environment Description

Hudi version : 0.14.1-amzn-1

Spark version : 3.5.1

Hive version : 3.1.3

Hadoop version : 3.3.6

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : yes

Slow Commit

Faster commit

Building Workload Profile job / slow stage with shuffle spill

@sumosha The tasks with disk spill is normally a lot slower. Do you see spills in the fast run also? Also noticed very high GC for them. Did you tuned your GC configs. https://hudi.apache.org/docs/tuning-guide/#gc-tuning

@ad1happy2go It does appear there is spill even in the faster commit (which explains why that job seems to stay consistent at around 15 minutes).

I haven't been able to recreate the disk spill in my stress testing, so I assume it is the size difference in the underlying table and files being written (production is around 100GB now, I started with a fresh table in testing and haven't built up a good size yet). I was planning to play around with this setting mentioned in your guide: hoodie.memory.merge.fraction. Does this seem the right track? I'm wondering if just a larger instance size is warranted as this grows (maybe fewer instances to get a comparable core count).
We are currently on the default collector in EMR (Parallel) in production. I have updated to the G1 (this is jdk 17) in stress testing, though I didn't see much change in the overall commit times. We'll move forward with the G1 since it's recommended anyway.

@sumosha Is there any other difference you could point out in the fast run or slow run. Is it possible to share the two hudi commit files for them, we want to understand the number of file groups updates and time taken in updating each filegroup. You can also directly send to me on apache hudi slack (Aditya Goenka) if you are not comfortable to share it here.

@ad1happy2go Hi Aditya! I PM'ed you in Slack with the commits as requested. As far as I can tell, the only difference I see is those warnings in the logs during the slow commit. Thanks!


Object.constructor should have a constructor type

in lib.es6.d.ts:
interface Object {
constructor: Function;
Shouldn't this be:
constructor: { new (...args: any[]): any };
so that it is well-typed to say:
new x.constructor

https://github.com/Microsoft/TypeScript/issues/3841?

more specifically dup of #4356


CI: (Re)enable style checks

Description
WIP.
Checklist

[ ] Investigated and inspected CI test results

Testing Performed
CI is sufficient.

FYI, I will add these to the allowed list:
/home/gavin/go/src/github.com/stackrox/stackrox/tools/detect-large-files.sh "/home/gavin/go/src/github.com/stackrox/stackrox/tools/allowed-large-files"
Found large files in the working tree. Please remove them!
If you must add them, you need to explicitly add them to the allow list file /home/gavin/go/src/github.com/stackrox/stackrox/tools/allowed-large-files
Files were:
  central/pruning/pruning_test.go
  deploy/charts/monitoring/dashboards/dataplane.json
make: *** [Makefile:228: no-large-files] Error 1


SPIKE - Michelson Prototype

Description
Purpose of the spike is to validate the plugin framework design can accommodate Michelson smart contracts in the future.
Note: consider using Michel codec from Taquito
Acceptance Criteria

[x] Explore the ability for Taqueria to validate a smart contract written in Michelson
[x] Explore the ability for Taqueria to originate smart contract written in Michelson
[x] Create Michelson compile user story

Closing: All requirements met


Cocoa Pods - [!] Unable to find a specification for RGPageViewController

target 'Project' do
  use_frameworks!
  pod 'RxSwift'
  pod 'RxCocoa'
  pod 'RxAlamofire'
  pod 'NVActivityIndicatorView'
  pod 'RGPageViewController'
end

When I run pod install, it gives me the error :

Cocoa Pods - [!] Unable to find a specification for RGPageViewController

Solved by replacing pod 'RGPageViewController' with pod 'RGPageViewController', :git => 'https://github.com/eRGoon/RGPageViewController.git'. However, you should take a look at why the first one did not work :)

that's weird. works fine for me
$ pod install
Analyzing dependencies
Downloading dependencies
Installing RGPageViewController (1.0.0)
Generating Pods project
Integrating client project

[!] Please close any current Xcode sessions and use `xxxxxx.xcworkspace` for this project from now on.
Sending stats
Pod installation complete! There is 1 dependency from the Podfile and 1 total pod installed.

Podfile
platform :ios, '9.0'

target 'xxxxxx' do
  use_frameworks!

  pod 'RGPageViewController'

  target 'xxxxxxTests' do
    inherit! :search_paths
  end

end


missing file

I installed 3.6.6 and I was able to select the "Kind" property for the TStyledBitBtn and the resource loaded. When I compiled the app the Errror E1026 File not found: 'CommandLinkPNG.RES'

Delphi version?

Fixed in last version.


Fixed typos in docs and moved CONTRIBUTING guide to one level up for ease of access.

Just starting out with contributing to this awesome project. So, correct grammar is the first thing to sort out ;)

Awesome.


people auto complete: Send user ID for users having same full name.

Previously, our @-mention autocomplete had a bug, wherein it
failed to correctly mention the correct user if more than one user
had the same full name.
Now, if more than one users have the same full name, we now send
the user ID as well to ensure that the correct user is
@-mentioned.

@chrisbobbe, @gnprice, @ray-kraesig this PR is ready for a review.

@chrisbobbe I've pushed some changes.
This version uses binary search and is faster than the linear one.
On CZO, the linear scan takes ~18ms or more.
This version takes  1ms or lesser (Date.now() has a resolution of 1ms)

Hmm, interesting! I actually didn't have binary search in mind when I commented at https://github.com/zulip/zulip-mobile/pull/4094#discussion_r421843264.
Instead, I was thinking that we wouldn't be working with arrays here anymore, and instead something like an Immutable.Map (now much more possible with https://github.com/zulip/zulip-mobile/pull/4047#issuecomment-624881408 merged).
At #3339, @gnprice said:

Instead, we should make these lookups through efficient, indexed data structures. A JS object works great. A Map works equally great, and can be a bit cleaner in the code.

However! The natural thing to do for #3339, to get users using an "efficient, indexed data structure" is, I think to do an Immutable.Map where keys are user IDs — not full names, which is what we care about here. So unless we have a separate index for full names, we're kind of worse off than we are with an array; we'd be awkwardly (and still linearly) traversing all entries in the user ID-keyed Map.
One solution, as you've shown here, is to use binary search on an array. (One step that I think is missing currently is to ensure users is sorted by full_name before passing it to binarySearch.) Better than linear, true, but a lookup in an Immutable.Map, as I understand it, is near-constant.
@gnprice, as someone who used to do research on algorithms, will know what to do here. But the one thing that hasn't come up in my discussions with him is the need to maintain indexes on things besides a canonical, unique ID.

Instead, I was thinking that we wouldn't be working with arrays here anymore, and instead something like an Immutable.Map (now much more possible with #4047 (comment) merged).

I did consider maps. But JavaScript does not support MultiMaps, so if we attempt to hash with full_name as keys, the last value will be overwritten. So we can't use a Map in this use case.
If we hash using user_id - well, in this case, we will have to iterate over the map, which is again O(n).

@rk-for-zulip thanks for the review. I've made some changes to fix the issues you pointed out. Can you take a look again?

Seeing this add an implementaion of binary search, my first reaction is that binary search is notoriously tricky to implement completely correctly. (Perhaps I'll go make a thread in #learning about that story -- it might be fun to read about.)
So I'm inclined to look for other solutions for that reason.
From an algorithmic perspective, one thing I notice here is that this is one of several places that we do a linear pass through this same array. If you look at getAutocompleteSuggestion, which is called in the render method of this component:
  const allAutocompleteOptions = getUsersAndWildcards(users);
  const startWith = filterUserStartWith(allAutocompleteOptions, filter, ownEmail);
  const initials = filterUserByInitials(allAutocompleteOptions, filter, ownEmail);
  const contains = filterUserThatContains(allAutocompleteOptions, filter, ownEmail);
  const matchesEmail = filterUserMatchesEmail(users, filter, ownEmail);

Each of those five lines represents a linear pass through all this data. The first one, in fact, actually allocates a copy:
export const getUsersAndWildcards = (users: User[]) => [
  { ...NULL_USER, full_name: 'all', email: '(Notify everyone)' },
  { ...NULL_USER, full_name: 'everyone', email: '(Notify everyone)' },
  ...users,
];

which tends to be especially expensive.
So all of those together are going to do much more work than we'd do by a linear search here. In algorithmic terms, that means the most we could gain by doing something fancier is a constant factor -- O(n) + O(log n) is still O(n), just like O(n) + O(n) is.
And actually the situation is sharper than that -- we're doing all that work every time the user types another letter, to search again with the new query. This code, on the other hand, only runs once at the end when the user chooses an item from the list.
So, let's just not do anything fancy here for now. :slightly_smiling_face:
I do find, empirically on czo, that the people autocomplete can be pretty slow to respond with results. That would be good to improve, and it might involve using some more-clever algorithms and/or data structures in getAutocompleteSuggestion and its helpers. Happy to talk more about that on #mobile-team, if you're interested.

@gnprice Thanks for the review, I've reverted it back to the simple version. Can you take a look again?
CC @chrisbobbe

Great, thanks! This looks good to me, though it just occurred to me that it would be good to refer to an authoritative source on this syntax, in the code comment you've added. I believe the get_mention_syntax function in static/js/people.js is what we want here.

@chrisbobbe I've updated the commit message.

Merged, thanks, @agrawal-d! I also made a small tweak: adding the reference to the web app in a code comment, in addition to the commit message.


Add functional test to ensure we are getting all conda environments

This is supposed to do the following:
conda create -n "env1" python=x.x
conda create -p "<dir path>/env2" python=x.x
conda create -p "./env3" python=x.x

The environment detection code should find the base and the three above environments.
If we add this test before we remove the env file service, then we can verify if we are getting all of the environments just via conda env list


Name add

Related issue

What changes does this PR do?
Screenshots (If any)
Questions

closing due to inactivity


Button and Dashboard is no longer working

Need some confirmation... but I have tried to generate a new clone from the repo and the dashboard does not appear to be working for any of the buttons.
Is anyone else having this issue with a fresh copy?

@makkoli : is there any reason some of the buttons on the dashboard would stop working?

I'm trying to test if the issue is because of the depreciation of material-ui  and reference @material-ui/core instead in CreateLeagueSelector.jsx
ie.
import React from 'react';
import PropTypes from 'prop-types';
import {
	Card,
	CardActions,
	CardTitle
} from '@material-ui/core/Card';
import CreateLeagueButton from './CreateLeagueButton.jsx';

import { sports } from '../sports.js';
import { cssCreateLeague } from '../styles';

// Creates a list of different buttons for selecting
// a sport type for the new league
// Once a sport type is selected, the proper form will be rendered
const CreateLeagueSelector = ({onSelect, selectedSport})=> (
	<Card style={cssCreateLeague.card.style}>
		<CardTitle
			subtitle='Choose a sport'
			subtitleColor={cssCreateLeague.card.title.subtitleColor}
			title='Create a New League'
			titleColor={cssCreateLeague.card.title.titleColor}
		/>
		<CardActions>
			{sports.map((sport, i) => {
				return (
					<CreateLeagueButton
						active={selectedSport === sport.name}
						icon={sport.icon}
						key={i}
						label={sport.name}
						onClick={onSelect}
					/>
				);
			})
		}
		</CardActions>
	</Card>
);

CreateLeagueSelector.propTypes = {
	onSelect: PropTypes.func,
	selectedSport: PropTypes.string
};

export default CreateLeagueSelector;

However the Eslint in my Atom is picking up the following:
Error	ESLint	Unable to resolve path to module '@material-ui/core/Card'. (import/no-unresolved)	7:8
Even though I added and updated the package.json @material-ui is still not being added to the webpack.

After studying the history of TravisCI builds I git checkout  74a6fb9 which was the last successful build.
The dashboard appears to functional at this point.
Some of the recent commits will have have to be checked as to which causing the issue.

@ahstein3521 : what were the features you were implementing in  b1982a2?

Commit for master: 8916011 appears to compile correctly as well.

I figured it out. The onTouchTap event has been deprecated from material-ui and needed to be replaced with onClick in the CreateLeagueButton.jsx file
it should work now, I submitted a pull request [https://github.com/freeCodeCamp/league-for-good/pull/109]

@lieberscott Can you check the other view/panels?
ie the Games list icon is not working, but the Teams roaster list icon is for mine.

@paulywill Ahh, yes, same here. It seems there are onTouchTap functions throughout the code. I'm making my way through to replace them all with onClick. Games icon you referenced is here ...
client > src > components > dashboard > seasons > seasons_list > seasonLink.jsx
changing onTouchTap to onClick should work there. Let me know. Meanwhile I'll make a new PR when I think I've replaced all the onTouchTap's

I just updated the PR, I think I got them all working

@lieberscott, nice!
There may be a few more.
The delete button under games doesn't appear to work.

Nor the back button when editing a game.

This is where some more TDD and tests would catch these ones.


Gzip document

Is your feature request related to a problem? Please describe.
When I run watchtower it is suggesting that the main document be gzipped or compressed as seen in this screenshot.

Describe the solution you'd like
Ability to gzip or compress the document root
Describe alternatives you've considered
After a bit of research, I wasn't able to determine if this is even possible with lambad@edge.

Here is a forum post https://forums.aws.amazon.com/thread.jspa?messageID=893421
Currently unanswered and points to this guide that seems to only suggest its possible with static pages https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-examples.html#lambda-examples-generated-response-examples

Looking at the example, this is essentially zip'ing the resulting content sent back to the client, and ensuring that the correct headers are set for the client to interpret the payload as being zipped. This does not seem to be too difficult to implement, perhaps put it in the lambda handler. What do you think @danielcondemarin

This should be easy to implement via CloudFront.
Looks like we just need to pass compress: true in the CloudFront defaults:
https://github.com/danielcondemarin/serverless-next.js/blob/master/packages/serverless-nextjs-component/serverless.js#L347
defaults: {
    ...
    compress: true // this should enable gzipped responses
 }

Also probably need to bump to the latest cloudfront version @6.0.0.
Relevant docs: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html

Interesting, I thought the document was really just "passed through" cloudfront (meaning that there is no caching). I didn't know that one can zip this as well.
BTW: Can you confirm that the document should not get cached.

@barrysteyn Currently the Server Side rendered pages and api routes are cached for 5 seconds. https://github.com/danielcondemarin/serverless-next.js/blob/master/packages/serverless-nextjs-component/serverless.js#L348
I was hoping that by enabling compression in CloudFront the document would be automatically compressed whenever the client browser supported it but this might not be the case as per  @oste's comment https://forums.aws.amazon.com/thread.jspa?messageID=893421
As a starting point I think we should enable compression in Cloudfront regardless. This should at the very least gzip /next/* and /static/*. If it doesn't work for ssr page documents or even api route responses we can do it in the lambda@edge handler  like in this example https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-examples.html#lambda-examples-generated-response-examples

Yes exactly. The lowest hanging fruits are definitely enabling cloudfront zip compression. BTW, the link the you pasted above showing that client browser may not be supported is not correct (could you perhaps paste it again, I would like to read that comment).
However, let's first enable cloudfront compression (that should be very quick for you). I will tackle (if you want) the gzip on the handler (once I read that comment though :) )

Excited that this seems possible. Let me know if I can chip in

In cloudfront CacheBehaviors Compress is true but still, the main request to the page is not gzipped.

Could would be nice to have a flag which adds a gzip function to the default response behavior...something like this.
As a workaround, consider using Cloudflare as a proxy.  It does brotli, http3 and other things.  Just remember to enable full ssl encryption to avoid infinite redirects.

what if we add another header Content-Encoding in
https://github.com/danielcondemarin/serverless-next.js/blob/af17d7101164fae6eb24cb3ec2d51ebb890a221f/packages/serverless-nextjs-component/next-aws-cloudfront.js#L3
I think that will fix this issue @Enalmada @danielcondemarin

@danielcondemarin Hey, I have created PR for gzip response. https://github.com/danielcondemarin/serverless-next.js/pull/272
It's working perfectly for me.

^^ was just about to weigh in on how gzipping works across lambdas and cloudfront, but danielcondemarin's implementation is correct. Hope this gets merged asap!

If I still want to Gzip the response what is the recommended way to do so?
Some context:
The content is not changing regularly and there are no dynamic changes on the server side. So my plan is to extend the cache TTL settings and use invalidations when needed.

@danielcondemarin any info regarding gzip?

If I still want to Gzip the response what is the recommended way to do so?
Some context:
The content is not changing regularly and there are no dynamic changes on the server side. So my plan is to extend the cache TTL settings and use invalidations when needed.

In that case there is value in gzipping. I’ll try and get the proposed PR merged this week and add some documentation around it how it should be used.

Having said all that I can see a LOT of benefit in combining GZIP (or other compression algorithms) with powerful caching policies in the component. I am particularly interested in an approach similar to ZEIT's serverless pre-rendering. I'll keep digging ... watch this space

I really love the idea of SPR (serverless pre-rendering)   :)    This would solve so many issues. It would almost eliminate all invalidations and boost the speed of dynamic pages fetching data from APIs by a lot. Thanks for pointing this feature out, I had not seen it before!  I am almost inclined to switch to the Zeit platform  :)

Hi folks! After reading more on gzip I am now convinced it is fine to merge https://github.com/danielcondemarin/serverless-next.js/pull/272 as it is.
This is now available in version @1.9.4. Thanks to @kartikag01 for the great work 🙏


Add overloads for CopyTo for Span.

Extracted from #1535 while we discuss the proper way of supporting Memory.

Ran a CI build, the changes look fine to me.  @Sergio0694 any overlapping concerns from the other PR?

FYI, CI build did fail: https://dev.azure.com/microsoft/Dart/_build/results?buildId=93364272&view=results


Update Jackson to 2.9.10

Security fixes available in version 2.9.10 for:

CVE-2019-16335
CVE-2019-14540

https://github.com/FasterXML/jackson-databind/issues/2410
https://github.com/FasterXML/jackson-databind/issues/2449

Closed by #154


failed install hermes-engine

pods install failed install hermes engine on RN 0.64.1
Installing hermes-engine (0.7.2)
[!] Error installing hermes-engine
[!] /usr/bin/curl -f -L -o /var/folders/1_/yx65j5bj7xxg4wg1wqlr5smm0000gn/T/d20210510-11664-hephga/file.tgz https://github.com/facebook/hermes/releases/download/v0.7.2/hermes-runtime-darwin-v0.7.2.tar.gz --create-dirs --netrc-optional --retry 2 -A 'CocoaPods/1.10.1 cocoapods-downloader/1.4.0'

facing the same issue after enabling hermes in ios

same issue here

Same here

ninguna novedad para solucionarlo?

the installation get stuck on Installing hermes-engine (0.7.2)

Same here on iOS. RN 0.65.0

Stuck for me too on Installing hermes-engine (0.8.1)
RN version: 0.65.1

You need to clean and re-install your pods:
1 - sudo gem install cocoapods-clean
2 - pod deintegrate
3 - pod clean ( this is no longer available )
4 - pod install
Enjoy!

update: I had to wait for some mins , but def. more than usual package install, but at the end it got installed

Please note that the installation of hermes-engine takes some time as the Pod needs to download ~1.2Gb of binaries.

Why am i getting this when trying to install hermes-engine?
Installing hermes-engine (0.69.0)

[!] Error installing hermes-engine
[!] /usr/bin/tar xf /var/folders/3k/jl2zckj12qbgrjp5jpmhr84m0000gp/T/d20220623-79686-2rfd1n/file.tgz -C /var/folders/3k/jl2zckj12qbgrjp5jpmhr84m0000gp/T/d20220623-79686-2rfd1n

./destroot/Library/Frameworks/universal/hermes.xcframework/ios-arm64_x86_64-maccatalyst/hermes.framework/Versions/0/hermes: Write failed
./destroot/Library/Frameworks/universal/hermes.xcframework/ios-arm64_x86_64-simulator/hermes.framework/hermes: Write failed
./destroot/Library/Frameworks/universal/hermes.xcframework/ios-arm64_x86_64-simulator/hermes.framework/Info.plist: Write to restore size failed
./destroot/include/jsi/: Can't create 'destroot/include/jsi'
./destroot/include/hermes/TraceInterpreter.h: Write failed
./destroot/include/hermes/Public/RuntimeConfig.h: Can't create 'destroot/include/hermes/Public/RuntimeConfig.h'
./destroot/include/hermes/Public/GCTripwireContext.h: Write failed
./destroot/include/jsi/instrumentation.h: Write failed
./destroot/bin/hermesc: Write failed
./destroot/bin/hbcdump: Write failed
./destroot/bin/hdb: Write failed
./destroot/bin/hermes: Write failed
./destroot/bin/hvm: Can't create 'destroot/bin/hvm'
tar: Error exit delayed from previous errors.

any ideas? i tryied to do a pod clean then pod install but same result.

Closing as this was resolved essentially. Please open a new issue if the issue comes back.

In "node_modules/react-native/sdks/hermes-engine/hermes-engine.podspec", there's an environment variable "HERMES_ENGINE_TARBALL_PATH"
if ENV.has_key?('HERMES_ENGINE_TARBALL_PATH')   Pod::UI.puts '[Hermes] Using pre-built Hermes binaries from local path.' if Object.const_defined?("Pod::UI")   source[:http] = "file://#{ENV['HERMES_ENGINE_TARBALL_PATH']}"
So I downloaded the tar with chrome used the command below:
HERMES_ENGINE_TARBALL_PATH="/Users/henry/Downloads/hermes-runtime-darwin-v0.70.0.tar.gz" pod install --verbose
It works!

@ofavor that is a bit hacky, do you know why this is happening?

but what if i want to update all dependencies in ios folder. so i go with pod install, then get an hermes error (screen) -> use @ofavor solution to update it, but in the case all other dependencies, which were supposed to be installed after hermes update (with pod install command), wont be, cuz we interrupted the flow, no? i hope u get what i mean. so how to update the rest of them after using @ofavor command

Why has this issue gained traction today after 3 months, maybe an intermittent problem with the download location?

indeed. got the same question, any ideas?

it seems that i didnt have enough space on mac. u can go to about this mac -> storage -> and there u will see how "available space" is going down. when it got to 0, i got an error. so i deleted everything that i could and it downloaded

Why has this issue gained traction today after 3 months, maybe an intermittent problem with the download location?

RN v0.70.0 comes bundled with hermes-engine default, so RN community may start seeing this more: https://reactnative.dev/blog/2022/09/05/version-070

pods install failed install hermes engine on RN 0.64.1
Installing hermes-engine (0.7.2)
[!] Error installing hermes-engine [!] /usr/bin/curl -f -L -o /var/folders/1_/yx65j5bj7xxg4wg1wqlr5smm0000gn/T/d20210510-11664-hephga/file.tgz https://github.com/facebook/hermes/releases/download/v0.7.2/hermes-runtime-darwin-v0.7.2.tar.gz --create-dirs --netrc-optional --retry 2 -A 'CocoaPods/1.10.1 cocoapods-downloader/1.4.0'

Why am i getting this when trying to install hermes-engine?
Installing hermes-engine (0.69.0)

[!] Error installing hermes-engine
[!] /usr/bin/tar xf /var/folders/3k/jl2zckj12qbgrjp5jpmhr84m0000gp/T/d20220623-79686-2rfd1n/file.tgz -C /var/folders/3k/jl2zckj12qbgrjp5jpmhr84m0000gp/T/d20220623-79686-2rfd1n

./destroot/Library/Frameworks/universal/hermes.xcframework/ios-arm64_x86_64-maccatalyst/hermes.framework/Versions/0/hermes: Write failed
./destroot/Library/Frameworks/universal/hermes.xcframework/ios-arm64_x86_64-simulator/hermes.framework/hermes: Write failed
./destroot/Library/Frameworks/universal/hermes.xcframework/ios-arm64_x86_64-simulator/hermes.framework/Info.plist: Write to restore size failed
./destroot/include/jsi/: Can't create 'destroot/include/jsi'
./destroot/include/hermes/TraceInterpreter.h: Write failed
./destroot/include/hermes/Public/RuntimeConfig.h: Can't create 'destroot/include/hermes/Public/RuntimeConfig.h'
./destroot/include/hermes/Public/GCTripwireContext.h: Write failed
./destroot/include/jsi/instrumentation.h: Write failed
./destroot/bin/hermesc: Write failed
./destroot/bin/hbcdump: Write failed
./destroot/bin/hdb: Write failed
./destroot/bin/hermes: Write failed
./destroot/bin/hvm: Can't create 'destroot/bin/hvm'
tar: Error exit delayed from previous errors.

any ideas? i tryied to do a pod clean then pod install but same result.

but i have this issue:
[!] Error installing hermes-engine
[!] /usr/bin/curl -f -L -o /var/folders/py/ywshkpqj4pvb285158rlrbb40000gn/T/d20221011-92879-f2mx73/file.tgz https://github.com/facebook/react-native/releases/download/v0.70.2/hermes-runtime-darwin-v0.70.2.tar.gz --create-dirs --netrc-optional --retry 2 -A 'CocoaPods/1.11.3 cocoapods-downloader/1.6.3'
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
43  496M   43  215M    0     0  1899k      0  0:04:27  0:01:56  0:02:31 1913k
curl: (92) HTTP/2 stream 1 was not closed cleanly before end of the underlying stream

@conor909

@ofavor that is a bit hacky, do you know why this is happening?

Just like @cortinico said, hermes-engine is a little bigger, curl cannot download it due to a network error(timeout, peer reset...). I had to download it by chrome or other utilities.

Thank God, I solved it with this method, but need to be reminded that the file path cannot have spaces

Thank God, I solved it with this method, but need to be reminded that the file path cannot have spaces

ofavor commented on 13 Sep
In "node_modules/react-native/sdks/hermes-engine/hermes-engine.podspec", there's an environment variable "HERMES_ENGINE_TARBALL_PATH"
if ENV.has_key?('HERMES_ENGINE_TARBALL_PATH') Pod::UI.puts '[Hermes] Using pre-built Hermes binaries from local path.' if Object.const_defined?("Pod::UI") source[:http] = "file://#{ENV['HERMES_ENGINE_TARBALL_PATH']}"
So I downloaded the tar with chrome used the command below:
HERMES_ENGINE_TARBALL_PATH="/Users/henry/Downloads/hermes-runtime-darwin-v0.70.0.tar.gz" pod install --verbose
It works!

Thank God, I solved it with this method, but need to be reminded that the file path cannot have spaces

tried many times, still not work

tried many times, still not work

I went to this issue a couple of times this week, here are my insights
From what I experienced, there are a couple of probable reasons why you are facing this, as @ofavor @proto1994 said:

Either your network is slow and getting a timeout error, then stops the installation ==> https://github.com/facebook/react-native/issues/31505#issuecomment-1245424395 this solution worked for me one time
Either you don't have enough space on your disk, then you will get a  Write failed just because the space is not enough (even though your network is perfect :) ). If this is your case, you need to clean up your disk that will work, the solution worked for me
RN is now supporting the Hermes engine on ios to increase its performance, since the RN 0.64.0 but the package itself is too weight, to avoid this issue, u need to make sure u have enough space on your disk + a good internet network
https://reactnative.dev/docs/hermes#ios

Hello All i am using RN Version  "0.70.6"
While Installing Pod the terminal is getting stuck on  Installing hermes-engine (0.70.6) , it nearly downloaded/ consume 4gb data but didn't proceed to the next step.

what happend at the end ?

Http download
$ /usr/bin/curl -f -L -o /var/folders/q3/kjnn366n7ndcg1m4130jgctc0000gn/T/d20221223-97777-r61yhm/file.tgz
https://github.com/facebook/react-native/releases/download/v0.70.6/hermes-runtime-darwin-v0.70.6.tar.gz --create-dirs --netrc-optional --retry 2 -A 'CocoaPods/1.11.3
cocoapods-downloader/1.5.1'
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2  496M    2 11.8M    0     0  15006      0  9:37:46  0:13:50  9:23:56 10407
SO SLOW...

"source": {
"http": "file:///Users/xxx/packages/hermes-runtime-darwin-v0.70.6.tar.gz"  // replace this line
},
/ios/Pods/Local Podspecs/hermes-engine.podspec.json

Nothing just go to IOS>podfile>{hermes_enabled => false}>Pod install >rebuild the App and it works.
The issue has been resolved

It worked for me!!! THANKS!!!!

As people mentioned here, it's only about size of the Hermes engine, according to your network speed waiting for some time will do the thing. I almost waited around 10 minutes with my GSM network.

Installing hermes-engine (0.71.1)
Http download
$ /usr/bin/curl -f -L -o
/var/folders/jm/qyc0mdpj0mz8hzt4pd3pzqxh0000gn/T/d20230120-1973-rt3ctg/file.tgz
https://repo1.maven.org/maven2/com/facebook/react/react-native-artifacts/0.71.1/react-native-artifacts-0.71.1-hermes-ios-debug.tar.gz
--create-dirs --netrc-optional --retry 2 -A 'CocoaPods/1.11.3 cocoapods-downloader/1.4.0'
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
0  458M    0 3154k    0     0  21457      0  6:13:37  0:02:30  6:11:07 12578

It is so funny that mine taking around 6 hours to finish, this issue happened with "react-native": "0.71.1" fresh install.
even though my connection speed  almost 200 Mbps. so strange :(

Hello, I came across this issue too and I noticed @ofavor 's solution to download the hermes compressed file; I just need a download link. I've been googling for a few minutes and still can't seem to find it laying around anywhere.

I changed the network and it's solved.
A wifi -> B wifi

Building on CI this also happens occasionally.

This is what I was looking for.
Thanks a lot.
I manually downloaded the Hermes as well from the github release URL and provided the path in ENV and it worked.

In "node_modules/react-native/sdks/hermes-engine/hermes-engine.podspec", there's an environment variable "HERMES_ENGINE_TARBALL_PATH" if ENV.has_key?('HERMES_ENGINE_TARBALL_PATH') Pod::UI.puts '[Hermes] Using pre-built Hermes binaries from local path.' if Object.const_defined?("Pod::UI") source[:http] = "file://#{ENV['HERMES_ENGINE_TARBALL_PATH']}" So I downloaded the tar with chrome used the command below: HERMES_ENGINE_TARBALL_PATH="/Users/henry/Downloads/hermes-runtime-darwin-v0.70.0.tar.gz" pod install --verbose It works!

cd ios && pod install --verbose . Check network speed, try change dns ...., change network

when use commond:
/usr/bin/curl -f -L -o /var/folders/1_/yx65j5bj7xxg4wg1wqlr5smm0000gn/T/d20210510-11664-hephga/file.tgz https://github.com/facebook/hermes/releases/download/v0.7.2/hermes-runtime-darwin-v0.7.2.tar.gz --create-dirs --netrc-optional --retry 2 -A 'CocoaPods/1.10.1 cocoapods-downloader/1.4.0'
Error creating directory /var/folders/1_/yx65j5bj7xxg4wg1wqlr5smm0000gn.
curl: (23) Failed writing received data to disk/application
get error above, any one can help me?

I'm also getting this error after updating from 0.70.6 -> 0.71.6

In "node_modules/react-native/sdks/hermes-engine/hermes-engine.podspec", there's an environment variable "HERMES_ENGINE_TARBALL_PATH"
if ENV.has_key?('HERMES_ENGINE_TARBALL_PATH') Pod::UI.puts '[Hermes] Using pre-built Hermes binaries from local path.' if Object.const_defined?("Pod::UI") source[:http] = "file://#{ENV['HERMES_ENGINE_TARBALL_PATH']}"
So I downloaded the tar with chrome used the command below:
HERMES_ENGINE_TARBALL_PATH="/Users/henry/Downloads/hermes-runtime-darwin-v0.70.0.tar.gz" pod install --verbose
It works!

This is a good way, very nice!

This worked like magic for me
 enable Hermes for iOS by editing your ios/Podfile file

this works me arch -arm64 npx react-native init ProjectName

Please check whether your PC has enough space or not !

Hi @react-native-bot
What kind of libraries you guys are developing.  this is 2024 still we have issues with Hermes. even a sample app pod install does't work and fires with below error
Please guide this. It was working perfectly fine 2 days back and suddenly started throwing this kind of nonsense issues,
➜  SampleRNLatestVersion git:(main) ✗ cd ios
➜  ios git:(main) ✗ pod install --repo-update
Framework build type is static library
[Codegen] Generating ./build/generated/ios/React-Codegen.podspec.json
[Codegen] generating an empty RCTThirdPartyFabricComponentsProvider
Updating local specs repositories
Analyzing dependencies
Fetching podspec for DoubleConversion from ../node_modules/react-native/third-party-podspecs/DoubleConversion.podspec
[Codegen] Found FBReactNativeSpec
Fetching podspec for RCT-Folly from ../node_modules/react-native/third-party-podspecs/RCT-Folly.podspec
[Codegen] Found rncore
Fetching podspec for boost from ../node_modules/react-native/third-party-podspecs/boost.podspec
Fetching podspec for glog from ../node_modules/react-native/third-party-podspecs/glog.podspec
Fetching podspec for hermes-engine from ../node_modules/react-native/sdks/hermes-engine/hermes-engine.podspec
[Hermes] Using the latest commit from main.
Const Defined!
[!] Failed to load 'hermes-engine' podspec:
[!] Invalid hermes-engine.podspec file: [!] Unable to locate the executable cmake.
from /Users/arunsa/Documents/HNMyClub/RNUpGrade73/SampleRNLatestVersion/node_modules/react-native/sdks/hermes-engine/hermes-engine.podspec:116
-------------------------------------------
puts "Const Defined!"

   CMAKE_BINARY = Pod::Executable::which!('cmake')

# NOTE: Script phases are sorted alphabetically inside Xcode project
-------------------------------------------
[!] Do not use "pod install" from inside Rosetta2 (x86_64 emulation on arm64).
[!]  - Emulated x86_64 is slower than native arm64
[!]  - May result in mixed architectures in rubygems (eg: ffi_c.bundle files may be x86_64 with an arm64 interpreter)
[!] Run "env /usr/bin/arch -arm64 /bin/bash --login" then try again.
➜  ios git:(main)
My System data -
`➜  SampleRNLatestVersion git:(main) ✗ npx react-native info
info Fetching system and libraries information...
System:
OS: macOS 13.6.6
CPU: (10) x64 Apple M1 Max
Memory: 1.62 GB / 32.00 GB
Shell:
version: "5.9"
path: /bin/zsh
Binaries:
Node:
version: 20.12.0
path: ~/.nvm/versions/node/v20.12.0/bin/node
Yarn:
version: 1.22.19
path: ~/node_modules/.bin/yarn
npm:
version: 10.5.0
path: ~/.nvm/versions/node/v20.12.0/bin/npm
Watchman:
version: 2<IP_ADDRESS>
path: /usr/local/bin/watchman
Managers:
CocoaPods:
version: 1.15.2
path: /Users/arunsa/.rbenv/shims/pod
SDKs:
iOS SDK:
Platforms:
- DriverKit 23.2
- iOS 17.2
- macOS 14.2
- tvOS 17.2
- visionOS 1.0
- watchOS 10.2
Android SDK: Not Found
IDEs:
Android Studio: 2023.1 AI-231.9392.1.2311.11330709
Xcode:
version: 15.2/15C500b
path: /usr/bin/xcodebuild
Languages:
Java:
version: 18.0.2
path: /usr/bin/javac
Ruby:
version: 3.3.0
path: /Users/arunsa/.rbenv/shims/ruby
npmPackages:
"@react-native-community/cli": Not Found
react:
installed: 18.2.0
wanted: 18.2.0
react-native:
installed: 0.73.6
wanted: 0.73.6
react-native-macos: Not Found
npmGlobalPackages:
"react-native": Not Found
Android:
hermesEnabled: true
newArchEnabled: false
iOS:
hermesEnabled: Not found
newArchEnabled: false
➜  SampleRNLatestVersion git:(main) ✗ `

@arunsa-hrbl you can always submit a PR with the fix for the issues you find on open-source software. They're open-source for this reason.


Add a filled out safety training assessment

Resolved by recent commits. Tanyas still needs to be moved to the safety training assessment folder.


Support to change message's icon emoji

Support to change message's icon by iconemoji parameter.
Usage:
echo "wow" | slackcat -i ":+1:" -c my-channel -u yasuhiroki --stream

In Slack,

merged; thanks @yasuhiroki !


Does css-loader check for naming collisions

Does css-loader using localIdentName with (as an example) localIdentName=_[hash:base64:5] check for naming collisions?

That I'd like to know, too. For instance in production mode I'd like to use as few characters in the hash as possible to reduce size of the generated stylesheet. So far I guess I'm ok with base64:3 which would allow for around 262k classnames. I guess for the first few thousand classnames I am quite save but eventually the chance for collision should increase if there are no counter measures.

@hurrtz don't expect an answer though ;) i am looking for one since May.