Multiplication with a Plain Constant doesn't work

plain constant multiplication doesn't work for the algorithms that are supported for this operation, i.e. Paillier, Damgard-Jurik, etc.
Run the sample code in the README,
from lightphe import LightPHE
cs = LightPHE(algorithm_name = "Paillier")
m1 = 17
c1 = cs.encrypt(m1)
k = 1.05

assert cs.decrypt(k * c1) == k * m1

I get
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError

I think this is due to the type of k. If k is integer, e.g. 5, the above code works. I tested in 3 environment, the result is the same:

py3.10 64bit mac
py3.12 64bit mac
py3.10 32bit windows

The issue is that multiplication of 17 and 1.05 is not integer independent from k.
Try it for m=10000 and k=1.05

I see. Is this (the multiplication must be integer) the limitation of the algorithms or your implementation?

Yes. These algorithms are working on integers. Even float numbers can be represented as integers as well.

These algorithms are working on integers
Sorry I don't quite understand your answer. Are you saying the (mathematical) algorithms themselves can't deal with float numbers?  Or is it just your implementation (your code) doesn't work with float?

Algorithms' limitations

Got it, thanks for the clarification.


Integrate cancellation

A dependency on https://github.com/tc39/proposal-cancellation to auto-cancel tasks on early break would be incredibly useful, and can't be added in a non-breaking way after the fact. Consider the README example:
await.all users.map(async x => fetchProfile(x.id))

If await.all were updated to cancel its operand on rejection and fetchProfile does stuff like fill a cache of profile pictures, cancelling on failure will result in a lot less cache filled and possibly some page breakage, while not cancelling would be considerably more tolerant to that.

All child promises could just share the same cancellation context. Note that this stands independent of #29 - it's somewhat necessary for both if you want one or the other.

It can be added to await.all the same way it could be added to Promise.all - this proposal's intention is to mirror the static combinators.

It can be added to await.all the same way it could be added to Promise.all - this proposal's intention is to mirror the static combinators.

You can't add it to Promise.all and have it transparently cancel promises, not unless you can somehow propagate cancellation through said promise. And from what I've observed, it seems that form of cancellation has been rejected by the committee.
Cancellation would have to be bound to when the tasks are created.

Right - if it can't be added to Promise.all, then it can't and shouldn't be added to await.all. This proposal is only sugar, and won't be anything more.


configuration

I didnt know how else to ask this, but how do you configure all the hitbox settings? I couldn't find any info on where the settings are

@takoyak20 The settings are managed through modmenu. Install it, click the "mods" button on the pause menu, fid the mod in there, and click the settings button.


Broken link --  Page not found -- Using Clarity Values

Under the section " Using Clarity Values" the word "Clarity Value" is hyperlinked, but the link sends a page not found message.

Hi @losatnick !
The mentioned link seems to be working properly. Can we close this issue?

@csgui, the link still looks broken to me! I'll take this one.


No cargo version configured

Hi,
I get
error: rustup could not choose a version of cargo to run, because one wasn't specified explicitly, and no default is configured.
help: run 'rustup default stable' to download the latest stable release of Rust and set it as your default toolchain.

when using container with GHA:
jobs:
  build_container:
    runs-on: ubuntu-latest
    container: blackdex/rust-musl:aarch64-musl-stable-1.82.0
    steps:
      - uses: actions/checkout@v4
      - name: Build
        run: cargo build --release

Strange, as you can see i kinda do the same during the test builds
https://github.com/BlackDex/rust-musl/actions/runs/11477497940/job/31939721088#step:14:9
Though that uses the docker run and not my image as GHA image.
I actually have never tried that, but in theory it should work.
Maybe something in your checkout is overwriting something?
Ill have to test this my self to see what happens.
Is the repo/branch you use for this public?

https://github.com/Ludea/lucle/actions/runs/11540151227/job/32120633661

Strange, when i try to use act, it seems to work just fine.
``bash
act push -j build_container

I suspect it has something to do with some default env's Github sets. Ill check a bit further.

You can probably fix this for now by setting:
env:
  HOME: /root

env:
  HOME: /root

Fix my issue, thanks

Ok, cool I'm working on a fix for this so that the next build's will be supported this way.

@Ludea, you should now be able to remove that env again. I fixed the images.

Now I get /__w/_temp/b84a7ab6-8893-49f7-8d31-b598ae0128ba.sh: 1: cargo: not found Error: Process completed with exit code 127

I try with ghcr.io/blackdex/rust-musl:aarch64-musl-nightly-2024-10-25, and it works as expected (like your test)
With ghcr.io/blackdex/rust-musl:aarch64-musl still get issue

Yea i noticed. GitHub Actions is doing something strange.

I have it working now, and i tested it using a fork of your project.
The images are currently being built and once done it should be able to run without any extra env needed to be set.
Fixed via 8831be56e896d22aea117ec2cb8d6051a7fef46b


composer and better exception handling

added an directly executable create and write example
added namenoderpcaddress as WebHDFS url

Sorry it took me a bit to get around to this, I had some access issues.
 If you remove that I'm happy to merge this into the main branch.

I changed the naming of the composer package, please check again.


chore(deps): bump go-paymail to 0.21.3

Pull Request Checklist

[ ] 📖 I created my PR using provided  : CODE_STANDARDS
[ ] 📖 I have read the short Code of Conduct: CODE_OF_CONDUCT
[ ] 🏠 I tested my changes locally.
[ ] ✅ I have provided tests for my changes.
[ ] 📝 I have used conventional commits.
[ ] 📗 I have updated any related documentation.
[ ] 💾 PR was issued based on the Github or Jira issue.

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 45.48%. Comparing base (149f409) to head (b6a3b90).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #783   +/-   ##
=======================================
  Coverage   45.48%   45.48%
=======================================
  Files         352      352
  Lines       17546    17546
=======================================
  Hits         7980     7980
  Misses       8974     8974
  Partials      592      592

Flag
Coverage Δ

unittests
45.48% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 149f409...b6a3b90. Read the comment docs.


Mod cannot be downloaded via third party launchers

Just as the title says.
To fix this, go to the CurseForge page, and go to license, and select the appropriate option.

Should be fixed!


Naming matter transmitter/receiver causing client crash

Hi McJty,
I'm currently playing through Project Ozone 2 - Kappa and have the following crash when naming matter transmitters/receivers.  Is this a pack issue or something new/known?
http://pastebin.com/QxDmvG68
Thanks!
MC_Tirm

This has been reported before but I have never been able to find this and 1.7.10 is no longer maintained so...

After some hours of messing around, I discovered it only happens when I have a tinkers tool in my hand!
Sorry for seeing the previously reported issue, I did do a search, but nothing really came close, so must have been worded differently!

Close as 1.7.10 is no longer supported. If this continues to be a bug on 1.12 or newer, please open a new issue.


Analytics and Alerts don't work locally

When running this application locally, the dashboard page works.However, the analytics or the alerts don't appear when clicking the associated button in the navigation bar at the top. However, when I push it to the cloud everything works fine.

That's strange, I've never seen that issue before. I'm not doing anything fancy with app-route or anything, just swapping out the conditional areas of the page based on the selection in app-nav. What are you using for serving it locally? gulp serve or something else? Can you try inspecting the <px-sample-app> tag and verifying that the selected property is getting changed when you click another option?

Sure I will check the tag and let you know.  I am using gulp serve.  on starting the instance the address is http://localhost:8080 so am not sure if that could be the issue.

One thing to add is that I am using the Predix development machine to run this app.

OK, I have it working now. :)
When running firefox as sudo this works fine and the navigation works locally.  There must be a permissions issue with the predix dev VM.  Would be nice if anyone else is experiencing any issues like this.

Glad you were able to get up and running. If you wouldn't mind summarizing the issue on forum.predix.io, someone from the DevBox team may be able to help you via that channel.


Make Snapshot Logic Write Metadata after Segments Backport

Summary of the changes / Why this improves CrateDB
Backport of elastic/elasticsearch#45689
Write metadata during snapshot finalization after segment files
to prevent outdated metadata in case of dynamic mapping updates.
Checklist

[x] User relevant changes are recorded in CHANGES.txt
[x] Touched code is covered by tests
[x] Documentation has been updated if necessary
[x] CLA is signed
[x] This does not contain breaking changes, or if it does:

It is released within a major release
It is recorded in CHANGES.txt
It was marked as deprecated in an earlier release if possible
You've thought about the consequences and other components are adapted
(E.g. AdminUI)

Should this have a release notes entry (fix?) and be backported to 4.0?

Yes please.

Should this have a release notes entry (fix?) and be backported to 4.0?

Yes please.

ok added, maybe have a final look before i merge about the change message.

👍

yay

retest this please
Running: Test class org.elasticsearch.index.engine.InternalEngineTests
FAILURE: Test testRebuildLocalCheckpointTracker(org.elasticsearch.index.engine.InternalEngineTests)
[2019-11-15T03:04:45,320][INFO ][o.e.i.e.InternalEngineTests] [testRebuildLocalCheckpointTracker] before test
[2019-11-15T03:04:45,436][INFO ][o.e.i.e.InternalEngineTests] [testRebuildLocalCheckpointTracker] after test
REPRODUCE WITH: ./gradlew null -Dtests.seed=18A00E0DB6C5742C -Dtests.class=org.elasticsearch.index.engine.InternalEngineTests -Dtests.method="testRebuildLocalCheckpointTracker" -Dtests.locale=ta-LK -Dtests.timezone=NZ
java.lang.AssertionError: seq_no=54 max_seq_no=62 checkpoint=-1
Expected:
 but: was <false>


Erratic tabbing between timepickers

I have three time fields in my form, which I want the user to be easily able to tab and back-tab between the three fields.  However I found that with just three timepickers in my form that the focus jumped from the first to the second and back to the first, for several cycles, then it jumps to third, and then goes back to jumping between the first two for a while, before going back to the third field.
Any ideas on how to get the field navigation to work properly?
Here are the definitions for the three timepickers, perhaps I just missed something:
<body>
...
<input type="text" id="timepicker-1" class="timepicker" tabindex="1" />
<input type="text" id="timepicker-2" class="timepicker" tabindex="2" />
<input type="text" id="timepicker-3" class="timepicker" tabindex="3" />
...
<script>
$( '#timepicker-1' ).wickedpicker( { now : "13:00", title : 'Timepicker-1', show : showPicker } );
$( '#timepicker-2' ).wickedpicker( { now : "14:00", title : 'Timepicker-2', show : showPicker } );
$( '#timepicker-3' ).wickedpicker( { now : "15:00", title : 'Timepicker-3', show : showPicker } );
function showPicker( element ) {
$( '.wickedpicker__title' ).contents().first().replaceWith( ( ( this.options !== undefined )
? this.options.title
: this.title ) );
}
</script>
...
</body>

Any ideas on how to get the field navigation to work properly?
Thanks

I wasn't really able to recreate the issue with tabbing between three pickers, but I know that the plugin does add tab indexes to each of the time spans (ie wickedpicker__controls__control--hours, ..minutes, seconds, etc..) so that when you focus on an input that has a wickedpicker the focus is shifted to the hours element.  I may need to rethink the keyboard accessibility part.

Thank you for your attention on this issue,I used the standard external javascripts and css file, shown in the attached file and without the timepickers actually opening, if I press the tab key the focus jumps between the three instances on the page, but not in the expected order.  Instead the focus goes as follows:From					Totimepicker input box 1	timepicker input box 2timepicker input box 2	timepicker input box 1back and forth between timepicker input box 1 and timepicker input box 2 a few times, and then finallytimepicker input box 2	timepicker input box 3timepicker input box 3	timepicker input box 1,timepicker input box 1	timepicker input box 2back and forth between timepicker input box 1 and timepicker input box 2 a few times, and then eventuallytimepicker input box 2	timepicker input box 3This pattern repeats itself randomly for as long as you press the tab key.I tried to put the code that I've attached into jsFiddle and CodePens, but couldn't really do that because the external css and js code is local on my computer.I know that none of this is new information, but perhaps seeing that the html file that I'm getting these results from isn't doing anything special or unusual will help.  I can send you a video of the focus jumping between the fields if you like, but only if you want it.Thanks again,Howard Brown-----Original Message-----
From: Eric Gagnon
Sent: Aug 26, 2016 4:11 PM
To: ericjgagnon/wickedpicker
Cc: howardb3115 , Author
Subject: Re: [ericjgagnon/wickedpicker] Erratic tabbing between timepickers (#29)
I wasn't really able to recreate the issue with tabbing between three pickers, but I know that the plugin does add tab indexes to each of the time spans (ie wickedpicker__controls__control--hours, ..minutes, seconds, etc..) so that when you focus on an input that has a wickedpicker the focus is shifted to the hours element.  I may need to rethink the keyboard accessibility part.
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or mute the thread.
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/ericjgagnon/wickedpicker","title":"ericjgagnon/wickedpicker","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/ericjgagnon/wickedpicker"}},"updates":{"snippets":[{"icon":"PERSON","message":"@ericjgagnon in #29: I wasn't really able to recreate the issue with tabbing between three pickers, but I know that the plugin does add tab indexes to each of the time spans (ie wickedpicker__controls__control--hours, ..minutes, seconds, etc..) so that when you focus on an input that has a wickedpicker the focus is shifted to the hours element.  I may need to rethink the keyboard accessibility part.  "}],"action":{"name":"View Issue","url":"https://github.com/ericjgagnon/wickedpicker/issues/29#issuecomment-242873520"}}}Work Phone:<PHONE_NUMBER>Home Phone:<PHONE_NUMBER>Cell Phone:<PHONE_NUMBER>Hungry Genius URL:	www.hungrygenius.comWork Email<EMAIL_ADDRESS>(coming soon!)Home Email<EMAIL_ADDRESS>Phone:<PHONE_NUMBER>Home Phone:<PHONE_NUMBER>Cell Phone:<PHONE_NUMBER>Hungry Genius URL:	www.hungrygenius.comWork Email<EMAIL_ADDRESS>(coming soon!)Home Email<EMAIL_ADDRESS>


Nix/NixOS compatibility; remove installation-time dependency on opam

I am trying to get this to build and work on Nix+NixOS, a "declarative" package manager+operating system with a lot of appeal to the functional programming community.
The existing installation/build process relies on opam to generate and write the Jupyter kernelspec files (kernel.json, kernel.sh).  This approach is incompatible with Nix, which manages its own OCaml versions/dependencies and does not use opam.  Consequently, when Nix tries to build it like other Dune packages by calling dune build -p jupyter, Dune first tries to generate the kernelspec files by calling config/kernelspec.exe, which tries to call opam, and throws a "command not found" error.
The NixOS-specific fix would be to not generate these kernelspec files at all or at least remove the kernelspec generation script from the default installation step, since NixOS provides its own way of configuring Jupyter to use additional kernels via its jupyter module.  But I recognize doing so may make the installation process less convenient for others.
Personally, I think it is best if the installation process is package-manager-neutral, i.e. it doesn't assume what OCaml package manager people are using.  Considering that the kernelspec is a Jupyter configuration and is not in any way an actual dependency of the OCaml Jupyter kernel program itself, it makes sense that one should be able to just build the OCaml Jupyter kernel without needing to generate the kernelspec first (NixOS users especially don't need the kernelspec to be generated at all).  Following this reasoning, my suggestion is this:

Separate the kernel.{json,sh} generation from the Dune installation process.  Instead, make kernelspec.ml a separate public executable (named something along the lines of ocaml-jupyter-genspec) that may rely on opam if it needs to and generates the kernelspec files as a convenience tool only for those who want it.
Add instructions in the README, telling users to either manually create the kernelspec files following some example or, for opam users, generate it using ocaml-jupyter-genspec.

I believe this is a reasonable compromise: the convenience of auto-generating the kernelspec files remains, but non-opam users won't need to install opam just to build the standalone kernel (opam is only needed for the kernelspec generation), and for opam users there are only 1-2 extra commands to run during installation.
What do you think about this?

Thanks. I don't know how popular Nix is. However removing dependency to opam seems good idea. I will remove kernelspec.ml, and add ocaml-jupyter-opam-genspec for manual generation of kernel.json instead.


Cursor should have some kind of lifetime

A Cursor cannot outlive its TranslationUnit, which cannot outlive its Index.

After the bindgen rewrite lands, the translation unit, index and cursor can have the lifetime of the context.
Extra points would be migrating to clang-sys, which already provides this.

clang-sys is just raw bindings to libclang (and a build script to find and link to libclang). clang-rs is my (somewhat) idiomatic wrapper, and it makes use of lifetimes for this particular purpose.

Whoops, sorry, of course I meant clang-rs instead of clang-sys, thanks for the catch!

I looked into migrating this crate to clang-rs, and I'm not sure it's worth the effort. The entire ir module would need to be rewritten using the clang-rs API and using lifetimes for almost every type.
I put a few hours into it before giving up.

I think it shouldn't be that hard if we always parameterize on the lifetime of the context. You're right it's a tedious task though :/


Revert "improvementEnable avx512 support in bitshuffle for performance boost"

Reverts apache/doris#15972 due to issue #24145

run buildall


Change to config read from file

Remove unnecessary or redundant command options.
Modify some features that were only applicable to log types to be available to all types.
Update README document.

Close: #455

Codecov Report
Attention: Patch coverage is 0% with 818 lines in your changes are missing coverage. Please review.

Project coverage is 21.04%. Comparing base (859595d) to head (ac051c6).

Files
Patch %
Lines

src/producer.rs
0.00%
678 Missing :warning:

src/controller.rs
0.00%
91 Missing :warning:

src/main.rs
0.00%
20 Missing :warning:

src/config.rs
0.00%
14 Missing :warning:

src/report.rs
0.00%
8 Missing :warning:

src/syslog.rs
0.00%
7 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #465      +/-   ##
==========================================
- Coverage   21.65%   21.04%   -0.62%
==========================================
  Files          47       47
  Lines        8615     8809     +194
==========================================
- Hits         1866     1854      -12
- Misses       6749     6955     +206

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[incubator/honeydipper] add repo archive notice

See #24030

Should we provide a redirect link or anything here?
cc: @Charles546

/test pull-charts-e2e

Manually merging as this change is required. The failed test is not due to a change in this PR.


Add preloaders for log in and register

Hey @mohityadav0903, I would like to work on this issue for GSoC'22.

Hey @mohityadav0903, I have added the mentioned task i.e. Preloader for Login and Signup page. Here is an example -

Hello @mohityadav0903, I noticed that in DropDownItems.jsx, the import has been changed to something else. The original package was "react-loader-spinner". That's creating the issue.

hey it's solved now i am reviewing your PR

hey @Shubham29012001 i have merged your PR. I  am closing this issue u can work on other issues if u want thanks


add state abbreviations

hi - i noticed that the API you're using handles alpha2 and alpha3 codes when searching countries, but the API does not handle USPS codes for United States data.
For example /countries/KR will return data for Korea.  However, /v2/states/NY, will not return data for New York.
I've added proof of concept for handling United States codes.  It makes it much easier to compare states:
kovid check va ny ca tx
I could also add same logic for Provinces, so you could also do kovid province ON instead of kovid province Ontario

Impressive work Jamie! This is a very much welcome PR! Thanks for putting time into it

If you would like to take care of the provinces, please go ahead! @Jberczel


feat: support for n-way comparisons and ratings

Validation tests now done in UI rather than SDK

:tada: This PR is included in version 0.335.0 :tada:
The release is available on:

npm package (@next dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


update title and add "more information portion" in readme of drivers/nodejs

Synchronize the title of drivers/nodejs readme file with the titles of other drivers readme files.
And add "more information portion" in its readme.

@abduu11aahh Unfortunately, someone else fixed this #642. So, it is no longer necessary.


Embed Onboarding Form in App

This way ppl will be able to get onboarded to updating PDF's easier.
iframe>src="https://docs.google.com/forms/d/1gYvzB1NNDYi19OBxPDhe74MMnYemNTwQZZgSZDWs4NU/viewform?embedded=true" width="760" height="500" frameborder="0" marginheight="0" marginwidth="0">Loading...</iframe

Which onboarding form do you mean?

https://docs.google.com/forms/d/1gYvzB1NNDYi19OBxPDhe74MMnYemNTwQZZgSZDWs4NU/viewform?usp=already_responded
This gathers information, when you complete it, it directs you to another form that's basically a checklist of all the places involved in PDFing, Google Drive, etc etc. Both of these forms are in the onboarding folder in the DH drive. The embed code got messed up in my comment, but I took away the opening/closing brackets so you can see it.
Happy to discuss whether this is relevant, or if instructions should geared towards Stacy's staff are more relevant as we roll it out and they start updating stuff.

A good idea, but currently not a priority. We may not be editing any more pdfs or editing them in an entirely different process.


[VL] Simplify the conversion from Substrait type to Velox type

What changes were proposed in this pull request?
Previously, there are two kinds of conversion from Substrait type to Velox type.

From Substrait type to SubstraitType, which is composed of a type name and the nullability. The type name in SubstraitType is then used to convert to Velox Type.
From the short name of Substrait type to Velox type.

For 1, string is used as the intermediate format to allow reusability of different backends. But as for now, this function is used only for Velox backend, and the intermediate string brings extra complexity for code maintainability.
For 2, it firstly converts the short name to aligned name format with SubstraitType, and reuses the first conversion. The process of string-string conversion is unnecessary and increases the complexity.
With this PR, above two conversions will become more straight forward as below.

From Substrait type to Velox Type directly.
From the short name of Substrait type to Velox type directly.

How was this patch tested?
under test

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query
log/native_3471_time.csv
log/native_master_10_25_2023_05650cc34_time.csv
difference
percentage

q1
33.92
35.20
1.275
103.76%

q2
24.90
24.96
0.058
100.23%

q3
40.07
40.49
0.418
101.04%

q4
37.36
37.34
-0.024
99.94%

q5
71.45
71.45
-0.002
100.00%

q6
7.30
7.39
0.094
101.28%

q7
86.86
86.09
-0.771
99.11%

q8
88.10
85.50
-2.597
97.05%

q9
120.04
121.65
1.608
101.34%

q10
54.67
53.87
-0.806
98.53%

q11
19.90
20.21
0.312
101.57%

q12
26.43
26.96
0.535
102.02%

q13
48.13
49.50
1.375
102.86%

q14
20.26
18.52
-1.742
91.41%

q15
34.84
33.91
-0.930
97.33%

q16
16.59
16.40
-0.189
98.86%

q17
100.94
100.59
-0.352
99.65%

q18
147.89
148.05
0.153
100.10%

q19
16.94
17.18
0.243
101.43%

q20
31.09
29.74
-1.356
95.64%

q21
225.43
222.15
-3.281
98.54%

q22
13.38
13.28
-0.097
99.27%

total
1266.51
1260.44
-6.076
99.52%


Stable support for MSC4115-style membership properties

MSC4115 has now completed FCP, so we can move it to the main test suite and use stable properties.
Requires https://github.com/element-hq/synapse/pull/17282 to land before it will pass CI.

Requires https://github.com/element-hq/synapse/pull/17282 to land before it will pass CI.

Oh, no it doesn't, because complement's CI checks out the matching branch of Synapse.


Beginner's Confusion

I'm attempting to use this package but I'm at a complete loss as to how it's installed. I have a project using npm and ZURB foundation. I'm not sure if I need to create a gulp task, a helper (handlebars) file or attempt to directly call the package. I've completed the npm install but just not sure what to do next in order to implement. Any insight would be greatly appreciated. Thank you

Hi! I just realized I forgot to add a pretty key bit of the documentation (namely, how to use it once it's installed). Here are the steps:

install via npm (npm install --save quiz-text)
use server-side by requiring the parser (var parse = require('quiz-text').parse)
pass a string of markdown into the parser to get the generated JSON output (var output = parse(' some markdown ')

I'm going to update the readme to include this (kinda necessary 😅 ) info, but that should work. Right now quiz-text is only officially supported in node, but I think it should work in the browser (compiled via browserify, webpack, rollup, etc). I'm going to run some tests to verify it works in the browser over the next week or so.
Until then, here's a pretty decent tutorial on getting npm modules to work via browserify, and here's a webpack one (if you replace lodash with quiz-text)
Please let me or @alliesahn know if you have any questions!

Excellent and thanks for the response & I will give it a shot!

Thanks! I'm gonna close this issue, but feel free to reopen if you have additional questions.


feat: add event searching

Closes #8

Looks good so far, but I think it might be better to use .contains(e) than .startsWith(e). Lmk what you think

fixed


Allow passing of custom context to tx_*_notify

I’m working with threadx and I wanted to use the event chaining for one of my applications. I noticed that it is not possible to pass down a private context for the callback (unlike other functions that allow arbitrary user context such as the thread_create, timer_create). Since this is not possible, it forces the callee to have a static context for modules that use the event chaining feature.

struct DRIVER_X
{
    TX_THREAD WorkingThread;
    TX_SEMAPHORE ThreadWakeup;

    TX_EVENT_FLAGS_GROUP InterruptStatus;
    TX_QUEUE UserRequests;
};

DRIVER_X instanceA;
DRIVER_X instanceB;

InitDriverX(&instanceA);
InitDriverX(&instanceB);

....

void InitDriverX(DRIVER_X* driver)
{
    tx_thread_create(driver->WorkingThread, "WorkingThread", driver, ....);
    tx_semaphore_create(driver->ThreadWakeup, "Wakeup", 0);
    tx_event_flags_create(driver->InterruptStatus, "Status");
    tx_queue_create(driver->UserRequests, "Reqs", ...);

    /* Chain flag and queue into the semaphore */
    tx_queue_send_notify(..., QueueNotify);
    tx_event_flag_set_notify(..., FlagNotify);
}

static void QueueNotify(TX_QUEUE* pQueue)
{
    // How to differentiate between a call for instanceA vs instanceB?
    tx_semaphore_put(??????);
}

static void FlagNotify(TX_EVENT_FLAGS_GROUP* pQueue)
{
    // How to differentiate between a call for instanceA vs instanceB?
    tx_semaphore_put(??????);
}

vs
struct DRIVER_X
{
    TX_THREAD WorkingThread;
    TX_SEMAPHORE ThreadWakeup;

    TX_EVENT_FLAGS_GROUP InterruptStatus;
    TX_QUEUE UserRequests;
};

DRIVER_X instanceA;
DRIVER_X instanceB;

InitDriverX(&instanceA);
InitDriverX(&instanceB);

....

void InitDriverX(DRIVER_X* driver)
{
    tx_thread_create(driver->WorkingThread, "WorkingThread", driver, ....);
    tx_semaphore_create(driver->ThreadWakeup, "Wakeup", 0);
    tx_event_flags_create(driver->InterruptStatus, "Status");
    tx_queue_create(driver->UserRequests, "Reqs", ...);

    /* Chain flag and queue into the semaphore */
    tx_queue_send_notify(..., QueueNotify, driver);
    tx_event_flag_set_notify(..., FlagNotify, driver);
}

static void QueueNotify(TX_QUEUE* pQueue, void* pContext)
{
    DRIVER_X* pDriver = pContext;
    tx_semaphore_put(pDriver->ThreadWakeup);
}

static void FlagNotify(TX_EVENT_FLAGS_GROUP* pQueue, void* pContext)
{
    DRIVER_X* pDriver = pContext;
    tx_semaphore_put(pDriver->ThreadWakeup);
}

Yeah, you're right, there's no good way to pass custom context.
One possibility is to use the extension members of the structs to store custom context, e.g. you can define the TX_QUEUE_EXTENSION member of the queue struct: https://github.com/azure-rtos/threadx/blob/master/common/inc/tx_api.h#L939

Closing. Anyone feel free to re-open if there are further questions related to this.


✨  Add opus to audio types

I also sorted the list and added this line:
"audio/opus": ".opus",
https://en.wikipedia.org/wiki/Opus_(audio_format)

Thank you, @rdwz! I'll get this published tomorrow. 🚀


Cannot $provide alternate templateUrl for typeaheadMatchDirective

Hi, I am using $provide to change the templateUrl for the typeaheadPopupDirective like this:
$provide.decorator('typeaheadPopupDirective', function ($delegate) {
    $delegate[0].templateUrl = 'bower_components/angular-ui-bootstrap/template/typeahead/typeahead-popup.html';
    return $delegate;
});

It does not work for typeaheadMatchDirective because the code is built differently:
  link:function (scope, element, attrs) {
    var tplUrl = $parse(attrs.templateUrl)(scope.$parent) || 'template/typeahead/typeahead-match.html';
    $http.get(tplUrl, {cache: $templateCache}).success(function(tplContent){
       element.replaceWith($compile(tplContent.trim())(scope));
    });
  }

Is it possible to allow a place to global override, i.e. $provide in my app config?

Yuck, this sounds like it was built wrong - might be worth doing a breaking change here. What are your thoughts @Foxandxss?

Will take a peek tomorrow, but it seems that that code is really old, there are better ways to do that concrete thing and we could "normalize" it to what we are doing now (templateUrl function) but that will remove features (no dynamic template).

It was never truly dynamic to begin with - one workaround would be to change it to template-url="{{templateUrl}} or whatever. It should accomplish the same thing.

I am having an issue with this as well. I dont think the above suggestion works, at least it did not work for me. My suggestion is to modify the line in the link function of the typeaheadMatch directive to be
var tplUrl = attrs.templateUrl || 'template/typeahead/typeahead-match.html';
My reasoning for this is that since template-url sounds like it should be a URL, it should always be a string, so there is no need to do anything fancy such as evaluating a function that is returned from another function based on what templateUrl is. This is a simple approach and it has worked for me. Any one have any objecting thoughts?

I'm a bit torn here.  There's not really a reason templateUrl has to be a string that I can think of.  That said, EVERY implementation of if that I can think of is a string.  So maybe there's the answer?
A third solution would be to allow the user to override the parsing behavior w/ an override-parse (or similar) attribute, but I'm not loving that idea.

I agree it does not have to be a string, but ultimately it always resolves to a string. I cannot think of an example where it would not, just as you said.  Im all for flexibility in code, but if no one will ever use the added flexibility, does it really need to be there?

I second OP.  I'd like to DRY up my templates by changing the typeahead-match template URL globally via $provide.

I have spent a bit of time thinking about this, and I am not quite sure if there is anything we can do here without a significant refactor maybe, along with breaking changes. May need to think about this some more, but because of the intermediaries involved, I am not sure there is anything we can do.

Noting investigation, attempted to strip out the manual compile logic today and replace it with just replace: true and a pluggable custom directive with logic straight from the attribute, ran into a problem with the directive attribute not containing the proper value because the child directive is compiled before interpolation occurs.
We need to investigate how to fix this cascading value not being preserved at the ideal step.

I'm better at AngularJS now so I figured out the solution:
.run(['$templateCache', function (
    $templateCache
) {
    $templateCache.put('template/typeahead/typeahead-match.html', typeaheadMatchHtml);
}]);


[Fix-13913][Master] syncMasterNodes does not make current slot correctly after zookeeper reconnect

Purpose of the pull request

This pr closes https://github.com/apache/dolphinscheduler/issues/13913

Brief change log

Set initial master heartbeat.

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

(or)
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

cc @zhongjiajie @ruanwenjun

Codecov Report

:exclamation: No coverage uploaded for pull request base (3.0.6-prepare@565bc97). Click here to learn what that means.
The diff coverage is n/a.

:exclamation: Current head 72d1d26 differs from pull request most recent head a8b5813. Consider uploading reports for the commit a8b5813 to get more accurate results

@@               Coverage Diff                @@
##             3.0.6-prepare   #14014   +/-   ##
================================================
  Coverage                 ?   40.30%
  Complexity               ?     4640
================================================
  Files                    ?      889
  Lines                    ?    35253
  Branches                 ?     3901
================================================
  Hits                     ?    14207
  Misses                   ?    19654
  Partials                 ?     1392

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Thanks @Radeity for the help


[stable/elasticsearch-exporter] Add envs to deployment template

What this PR does / why we need it:
This PR allows environment variables to specified as part of the deployment template. With upcoming stable releases of the elasticsearch exporter we'll be able to specify configuration parameters through environment variables. This PR will allow that functionality.
Which issue this PR fixes
None
Special notes for your reviewer:
Last Pr was closed due to inactivity so re-opening again.
Checklist

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md

sorry for delay.
/assign
/ok-to-test

/lgtm
thanks!

Cheers @desaintmartin!


Open previous/next weekly note does not work

I feel like these two commands do not work for me.  I am on the current weekly note, hit the "open next weekly note" and none is created for me.
I thought I would use this plugin during my "sunday" weekly reviews, where I would create new weekly note. According my locale, I have Monday set as week's starting day.
Maybe I'm using the plugin incorrectly...

I believe this is the expected behavior, and previous/next note is not intended to create one if it does not exist? Certainly previous/next daily note does not create a new note - they skip through the existing notes.
Very glad it works the current way, or I would have to wade through a sea of freshly created blank daily/weekly notes; I do not make a note for every day.

I'm having the same issue. In this issue it is said to be a normal behavior.
It is stated in the readme that it Skips over weeks with no weekly note file.
However I remember seeing somewhere that it was developed to create a note if it doesn't exist.
Edit : Nope, it stated the same.
TLDR : Not possible with the current version of the plugin and not planned to be so as far as I know.

I understand there may be a use case where days / weeks / months are skipped, but it would be nice to have it available for configuration, so that a user could create the next before the day / week starts. This is useful for those who have a weekly habit of planning the next week at the end of the previous week.

I can only agree with you. I don't know if there's a feature request yet, but I can't check right now, I don't even know if this message will be sent because of the low reception I have around here.

I would love this feature as well; my shitty workaround is using the Calendar plugin and clicking the next week to create the note if it isn't made yet. Even just adding a command from the calendar plugin would help my use case, as I'm just trying to have "Previous Week" and "Next Week" buttons in each of my Weekly Notes.

I would love this feature as well; my shitty workaround is using the Calendar plugin and clicking the next week to create the note if it isn't made yet. Even just adding a command from the calendar plugin would help my use case, as I'm just trying to have "Previous Week" and "Next Week" buttons in each of my Weekly Notes.

If you want, I created an obsidian workflow template, and there's buttons for periodic notes navigations using some templater script. It's just at the beginning of the periodic notes and you can just take that code if you feel like it ✌

The issue I have is that after having year of weekly notes, the app is confused about which year it is. For example, if the current week is 2023-W50 and I want to open the next week note (2023-W51) it opens W51 note from 2022. Moving the 2022-W51 into a separate folder doesn't change the behavior.

The issue I have is that after having year of weekly notes, the app is confused about which year it is. For example, if the current week is 2023-W50 and I want to open the next week note (2023-W51) it opens W51 note from 2022, not from 2023. Moving the 2022-W51 into a separate folder doesn't change the behavior. Even if I manually create 2023-W51, when I try to open it as "next week note" it opens 2022-51.

Hey, I suggest that you take a look at this repo for a templater alternative with periodic notes navigations.
Here's my setup for a daily note :

<%*
const currentMoment = moment(tp.file.title, "YYYY-MM-DD");
const hash = '# ';
const slash = ' / ';
const pipe = ' | ';
const leftAngle = '❮ ';
const rightAngle = ' ❯';
tR += leftAngle;
tR += '[[' + currentMoment.format('YYYY') + ']]' + slash;
tR += '[[' + currentMoment.format('YYYY-[Q]Q|[Q]Q') + ']]' + slash;
tR += '[[' + currentMoment.format('YYYY-MM|MMMM') + ']]' + slash;
tR += '[[' + currentMoment.format('GGGG-[W]WW|[Week] WW') + ']]';
tR += rightAngle;
tR += '\n';
tR += '\n';
tR += leftAngle;
currentMoment.add(-1,'days');
tR += '[[' + currentMoment.format('YYYY-MM-DD|dddd Do') + ']]' + pipe;
currentMoment.add(1,'days');
tR += currentMoment.format('dddd Do') + pipe;
currentMoment.add(1,'days');
tR += '[[' + currentMoment.format('YYYY-MM-DD|dddd Do') + ']]';
currentMoment.add(-1,'days');
tR += rightAngle;
%>

And for a weekly note :

<%*
const currentMoment = moment(tp.file.title, "GGGG-[W]WW");
const hash = '# ';
const slash = ' / ';
const pipe = ' | ';
const leftAngle = '❮ ';
const rightAngle = ' ❯';
tR += leftAngle;
tR += '[[' + currentMoment.format('YYYY') + ']]' + slash;
tR += '[[' + currentMoment.format('YYYY-[Q]Q|[Q]Q') + ']]' + slash;
tR += '[[' + currentMoment.format('YYYY-MM|MMMM') + ']]' + slash;
tR += '[[' + currentMoment.format('GGGG-[W]WW') + '|' + currentMoment.format('[Week] WW') + ']]';
tR += rightAngle;
tR += '\n';
tR += '\n';
tR += leftAngle;
currentMoment.add(-1,'weeks');
tR += '[[' + currentMoment.format('GGGG-[W]WW|[Week] WW') + ']]' + pipe;
currentMoment.add(1,'weeks');
tR += currentMoment.format('[Week] WW') + pipe;
currentMoment.add(1,'weeks');
tR += '[[' + currentMoment.format('GGGG-[W]WW|[Week] WW') + ']]';
currentMoment.add(-1,'weeks');
tR += rightAngle;
tR += '\n';
tR += '\n';
tR += leftAngle;
tR += '[[' + currentMoment.format('YYYY-MM-DD|dddd Do') + ']]' + pipe;
currentMoment.add(1,'days');
tR += '[[' + currentMoment.format('YYYY-MM-DD|dddd Do') + ']]' + pipe;
currentMoment.add(1,'days');
tR += '[[' + currentMoment.format('YYYY-MM-DD|dddd Do') + ']]' + pipe;
currentMoment.add(1,'days');
tR += '[[' + currentMoment.format('YYYY-MM-DD|dddd Do') + ']]' + pipe;
currentMoment.add(1,'days');
tR += '[[' + currentMoment.format('YYYY-MM-DD|dddd Do') + ']]' + pipe;
currentMoment.add(1,'days');
tR += '[[' + currentMoment.format('YYYY-MM-DD|dddd Do') + ']]' + pipe;
currentMoment.add(1,'days');
tR += '[[' + currentMoment.format('YYYY-MM-DD|dddd Do') + ']]';
tR += rightAngle;
%>

This works in the beta release (just created some future weekly notes), to use the beta release use BRAT


HDFS-16283. RBF: reducing the load of renewLease() RPC

Description of PR
HDFS-16283: RBF: improve renewLease() to call only a specific NameNode rather than make fan-out calls
Currently RBF will forward the renewLease() rpc to all the available name services. So the forwarding efficiency will be affected by the unhealthy downstream name services. And along with as more as NSs are monitored by RBF, this problem will become more and more serious.
In our prod cluster, there are 70+ nameservices, the phenomenon that renewLease() rpc is blocked often occurs.
This patch is be used to fix this problem and work well on our cluster, and the main ideas is:

Carrying nsId to client when creating a new file
Store this nsId in DFSOutputStream
Client renewLease() rpc carries all nsIds of all DFSOutputStream to RBF
RBF parses the nsIds and forwards the renewLease rpc to the corresponding name services

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  4s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
buf
0m  0s

buf was not available.

+0 :ok:
buf
0m  0s

buf was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 8 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 58s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
32m  6s

trunk passed

+1 :green_heart:
compile
8m  1s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
7m  8s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m 47s

trunk passed

+1 :green_heart:
mvnsite
4m 11s

trunk passed

+1 :green_heart:
javadoc
3m 29s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
4m 19s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
9m  5s

trunk passed

+1 :green_heart:
shadedclient
26m 31s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 27s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 23s

the patch passed

+1 :green_heart:
compile
7m 37s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
cc
7m 37s

the patch passed

+1 :green_heart:
javac
7m 37s

the patch passed

+1 :green_heart:
compile
7m 23s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
cc
7m 23s

the patch passed

+1 :green_heart:
javac
7m 23s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 30s

the patch passed

+1 :green_heart:
mvnsite
3m 44s

the patch passed

-1 :x:
javadoc
0m 50s
/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>.txt
hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> generated 1 new + 98 unchanged - 1 fixed = 99 total (was 99)

+1 :green_heart:
javadoc
3m 15s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
7m 52s

the patch passed

+1 :green_heart:
shadedclient
23m 40s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 29s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
345m 29s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
unit
32m 38s

hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
1m  1s

The patch does not generate ASF License warnings.

558m 56s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestDFSClientRetries

hadoop.hdfs.TestLease

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4524

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat

uname
Linux f0e1052376d4 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 802a4057e5d1a7333646f2c59a2ef29eca3ca057

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/1/testReport/

Max. process+thread count
2386 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Thanks @ayushtkn for your review, I learned a lot from it. Thank you again.
I have modified the path based on your helpful suggestions, please help me to review it again.
Because the apend() rpc is forwarded by invokeSequential, so I try to fill the namespace info in invokeSequential method like below. Please help me confirm whether this modification is reasonable or not.
if (ret instanceof LastBlockWithStatus) {
    ((LastBlockWithStatus) ret).getFileStatus().setNamespace(ns);
}

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 48s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
buf
0m  0s

buf was not available.

+0 :ok:
buf
0m  0s

buf was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 8 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 44s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
28m 28s

trunk passed

+1 :green_heart:
compile
6m 45s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
6m 26s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m 35s

trunk passed

+1 :green_heart:
mvnsite
3m 41s

trunk passed

+1 :green_heart:
javadoc
3m  3s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
3m 39s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
8m  4s

trunk passed

+1 :green_heart:
shadedclient
23m 36s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 27s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 57s

the patch passed

+1 :green_heart:
compile
6m 39s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
cc
6m 39s

the patch passed

+1 :green_heart:
javac
6m 39s

the patch passed

+1 :green_heart:
compile
6m 15s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
cc
6m 15s

the patch passed

+1 :green_heart:
javac
6m 15s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m 19s
/results-checkstyle-hadoop-hdfs-project.txt
hadoop-hdfs-project: The patch generated 2 new + 334 unchanged - 0 fixed = 336 total (was 334)

+1 :green_heart:
mvnsite
3m  8s

the patch passed

+1 :green_heart:
javadoc
2m 22s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
3m  6s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
7m 55s

the patch passed

+1 :green_heart:
shadedclient
23m 11s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 29s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
342m 58s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
unit
32m 57s

hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
1m  2s

The patch does not generate ASF License warnings.

539m 50s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.mover.TestMover

hadoop.hdfs.TestDFSClientRetries

hadoop.hdfs.TestLease

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4524

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat

uname
Linux c1dc81e47ee5 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / de5120f96c70102df6271d0e59a5669fc1ac59cf

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/2/testReport/

Max. process+thread count
2372 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Thanks @ayushtkn for your good idea, and I have updated the patch.
About caching availableNamespace, there is a point that is not easy to deal with, I'm looking for your help.
The available namespaces is dynamically changing along with RBF runs, such as name services becomes unavailable, end user manually disable some name services. If we want to cache available FederationNamespaceInfos, we need to consider cache invalidation.  For simple processing, so I removed the cache.
I'm looking for your help, thanks.

hmm, the caching may be can have a follow up post this, might be tricky but doable. you missed a couple of comments, rest things look almost good to me
@goiri / @Hexiaoqiao mind giving an additional check..

Thanks @ayushtkn for your review and ideas.
If needed, I will create a new PR to support caching availableNamespace

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 48s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
buf
0m  1s

buf was not available.

+0 :ok:
buf
0m  1s

buf was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 8 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 30s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
28m  5s

trunk passed

+1 :green_heart:
compile
6m 55s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
6m 27s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m 36s

trunk passed

+1 :green_heart:
mvnsite
3m 42s

trunk passed

+1 :green_heart:
javadoc
3m  2s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
3m 39s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
8m  5s

trunk passed

+1 :green_heart:
shadedclient
24m 17s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 30s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 17s

the patch passed

+1 :green_heart:
compile
8m  0s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
cc
8m  0s

the patch passed

+1 :green_heart:
javac
8m  0s

the patch passed

+1 :green_heart:
compile
7m 12s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
cc
7m 12s

the patch passed

+1 :green_heart:
javac
7m 12s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m 19s
/results-checkstyle-hadoop-hdfs-project.txt
hadoop-hdfs-project: The patch generated 1 new + 334 unchanged - 0 fixed = 335 total (was 334)

+1 :green_heart:
mvnsite
3m  8s

the patch passed

+1 :green_heart:
javadoc
2m 21s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
3m  3s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
7m 53s

the patch passed

+1 :green_heart:
shadedclient
23m 36s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 27s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
351m 10s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
unit
33m 38s

hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
1m  1s

The patch does not generate ASF License warnings.

552m 28s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestLease

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4524

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat

uname
Linux 0cbc9b2b2237 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 2d6e01e1da15f207d1e323fdcd0db16ef3f7eadb

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/3/testReport/

Max. process+thread count
2370 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@ZanderXu @ayushtkn, Thanks for your great works here. After a quick glance, it seems one solution to improve renewLease for RBF.
I would like to share my practice for this issue. I also meet this renewLease performance issue when upgrade to RBF architecture. When I observed that renewLease request count to NameNode grew unexpected and time cost obviously, I try to analysis if it is possible to bring file path as one parameter to renewLease. After collect create and renewLease audit, there are less than 3% renewLease requests to renew different files lease for one client (generally less than 5 files) in our data warehouse scenario (Maybe it is not true for other scenarios). Then I try to add a new interface public void renewLease(String path, String clientName) and route request based on path at Router side which is common logic.
Just propose another choice for this improvement. This is not objection for this PR, just another solution for discussion. If we have reached agreement, I would like to give deep reviews. Thanks again.

Thank @Hexiaoqiao for your solution. In the beginning, we try to carry the writing paths to RBF to fix this issue. After running for a while, I found some cases also need to be fixed:

Long run client case. There may be many files being written at the same time
Multiple destination case. RBF always forwards the renew lease rpc to all destination name service

Also, the number of renewLease requests between client and rbf will also increases, depending on the number of files being written at the same time.

Thanks for quick response.

Long run client case. There may be many files being written at the same time.

In my practice, the cost with split-path to renewLease will be under control even for long running applications, such flink applications (I have not observed that many files being written concurrently, it will be helpful if any cases could offer.)

Multiple destination case. RBF always forwards the renew lease rpc to all destination name service.

For both create and renewLease (with file path), I think they will apply the same MountTableResolver for same file. So it does not seem to one issue for renewLease. Maybe some corner case I do not catch. Please correct me if something missed.

the number of renewLease requests between client and rbf will also increases.

Yes, it is true. I am totally agree. Based on my internal production cluster, it will be less than 5% increase.
BTW, My consideration here is that it will be more smooth and understandable only one rbf namespace expose for client, rather than rbf and all namespaces behind router expose to client. Another side, renewLease is one lightweight request, less than 5% overhead is acceptable in my opinion.
Of course, the above information is totally based on my internal practice, maybe some other cases are not included. Very glad to hear more discussions and suggestions. Thanks.

For both create and renewLease (with file path), I think they will apply the same MountTableResolver for same file.

Although they all used MountTableResolver, create and append rpc can get the NS which the file belongs, but renewlease can only obtain the full NSs which the file mounted. So int this case, the renewlease rpc always forwarded to some unnecessary nameservices.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 40s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
buf
0m  1s

buf was not available.

+0 :ok:
buf
0m  1s

buf was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 8 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 55s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
25m  7s

trunk passed

+1 :green_heart:
compile
5m 57s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
5m 40s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m 19s

trunk passed

+1 :green_heart:
mvnsite
3m  8s

trunk passed

+1 :green_heart:
javadoc
2m 36s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
3m 11s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
7m 35s

trunk passed

+1 :green_heart:
shadedclient
20m 31s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 36s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 44s

the patch passed

+1 :green_heart:
compile
6m 28s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
cc
6m 28s

the patch passed

+1 :green_heart:
javac
6m 28s

the patch passed

+1 :green_heart:
compile
5m 57s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
cc
5m 57s

the patch passed

+1 :green_heart:
javac
5m 57s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 14s

the patch passed

+1 :green_heart:
mvnsite
2m 48s

the patch passed

+1 :green_heart:
javadoc
2m  1s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
2m 46s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
7m 31s

the patch passed

+1 :green_heart:
shadedclient
21m 23s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 24s

hadoop-hdfs-client in the patch passed.

+1 :green_heart:
unit
239m 35s

hadoop-hdfs in the patch passed.

+1 :green_heart:
unit
23m  2s

hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 51s

The patch does not generate ASF License warnings.

412m 37s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4524

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat

uname
Linux 8db473773c03 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 642f92050aa3877adef81dd8154cc55c8d5dff96

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/4/testReport/

Max. process+thread count
3297 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

For both create and renewLease (with file path), I think they will apply the same MountTableResolver for same file.

Although they all used MountTableResolver, create and append rpc can get the NS which the file belongs, but renewlease can only obtain the full NSs which the file mounted. So int this case, the renewlease rpc always forwarded to some unnecessary nameservices.

Exactly true. For MultipleDestinationMount, it could forward to different NS when request with file path only, especially for DestinationOrder.RANDOM and related order. cc @ayushtkn @goiri Anymore feedback here? Thanks.

@Hexiaoqiao I am ok with using path, but if I catch correct, the only save with using path will be like we won't be exposing the namespaces to the end client? but in exchange we will be saving I think a bunch of RPCs, especially in case of multi destination mount points.
May be from the performance point of view, it might be better with namespaces( the present approach). But I don't have any strong objections, if you feel we shouldn't expose the namespaces to end client.
if there is a particular use case where we shouldn't expose namespace to end client, in that case we may hide this change behind a config, and this optimisation won't work in that case, but in general ViewFs also knows about all namespaces and usually a lot of clients too have these namespaces defined in their configs. so, that is not a big secret, and this namespace info will also be their in back-end and I even don't think exposing them via this route can have any security issue?
But I am Ok, with whichever approach you folks feel better..

@ayushtkn It is not related with any security issue when I propose to use path as one parameter of renewLease. Actually in my opinion, it will be confused and poor readable with both namespaces and router name at client side, without other strong support points.

For MultipleDestinationMount, it could forward to different NS when request with file path only, especially for DestinationOrder.RANDOM and related order.

As mentioned above, for MultipleDestinationMount it will be difficult to reduce requests to NameNode at Router side. (I am limited by my internal case where no MultipleDestination with DestinationOrder.RANDOM hash configured.)
In conclusion, I agree that the current approach (expose namespaces to client and use ns to renewLease) will be more general solution, especially for MultipleDestinationMount cases. Thanks all for your detailed explanation.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 37s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
buf
0m  1s

buf was not available.

+0 :ok:
buf
0m  1s

buf was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 8 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
74m 49s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
24m 57s

trunk passed

+1 :green_heart:
compile
5m 56s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
5m 44s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m 20s

trunk passed

+1 :green_heart:
mvnsite
3m 12s

trunk passed

+1 :green_heart:
javadoc
2m 33s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
3m  8s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
7m 11s

trunk passed

+1 :green_heart:
shadedclient
20m  1s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 27s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 41s

the patch passed

+1 :green_heart:
compile
5m 54s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
cc
5m 54s

the patch passed

+1 :green_heart:
javac
5m 54s

the patch passed

+1 :green_heart:
compile
5m 33s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
cc
5m 33s

the patch passed

+1 :green_heart:
javac
5m 33s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  6s

the patch passed

+1 :green_heart:
mvnsite
2m 48s

the patch passed

+1 :green_heart:
javadoc
2m  3s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
2m 50s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
7m  7s

the patch passed

+1 :green_heart:
shadedclient
19m 54s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 24s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
234m 55s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
unit
21m 41s

hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 51s

The patch does not generate ASF License warnings.

462m 18s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestRollingUpgrade

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4524

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat

uname
Linux db558b1d5a3b 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 86ad8912390d21661196511dcca51a76847abf8e

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/5/testReport/

Max. process+thread count
3345 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 48s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
buf
0m  1s

buf was not available.

+0 :ok:
buf
0m  1s

buf was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 8 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 25s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
28m 16s

trunk passed

+1 :green_heart:
compile
6m 55s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
6m 24s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m 37s

trunk passed

+1 :green_heart:
mvnsite
3m 41s

trunk passed

+1 :green_heart:
javadoc
3m  4s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
3m 39s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
8m  5s

trunk passed

+1 :green_heart:
shadedclient
26m 16s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 31s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 25s

the patch passed

+1 :green_heart:
compile
7m 27s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
cc
7m 27s

the patch passed

+1 :green_heart:
javac
7m 27s

the patch passed

+1 :green_heart:
compile
6m 40s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
cc
6m 40s

the patch passed

+1 :green_heart:
javac
6m 40s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 22s

the patch passed

+1 :green_heart:
mvnsite
3m 14s

the patch passed

+1 :green_heart:
javadoc
2m 23s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
3m  7s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
7m 51s

the patch passed

+1 :green_heart:
shadedclient
23m 44s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 28s

hadoop-hdfs-client in the patch passed.

+1 :green_heart:
unit
340m 53s

hadoop-hdfs in the patch passed.

+1 :green_heart:
unit
33m 11s

hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
1m  2s

The patch does not generate ASF License warnings.

543m 12s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4524

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat

uname
Linux 89e41ce6bc5b 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 6739c2214b3a50de9200fcf6ee979a24777c376e

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/6/testReport/

Max. process+thread count
2375 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4524/6/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@ayushtkn @Hexiaoqiao Thanks for your discussion and review. I will continue to word hard to submit more patches to the community.


Unable to select in some cases

Version 5.1
Sometimes clicking to select doesn't select.
And when something is selected, sometimes right clicking de-select it.

Needs retesting in v5.5.

Did not experience this issue again, and cannot reproduce it by any means now. Closing as solved


chore: add .tractusx metafile

Addition to our TRG's
This PR adds an initial version of the .tractusx metadata file like described in TRG 2.05.
If there are any other repositories, that are connected to the Semantic Models, that together form a product, please point that out or add the connected repos later on
Fixes #265

@FaGru3n thanks for providing the fix. PR is merged!


WIP: chpi estimation changed

use SVD to estimate phase, correlation estimates corrected

cc @bmaess in case you aren't automatically notified about this PR, I made a minor comment. Glad to have someone else looking into this with me :)
Have you done many tests? I did some a while back and compared to MaxFilter, sometimes they matched well and other times they didn't. But I wasn't working with data where I knew ground truth so I couldn't evaluate which was correct...

Concerning the head position as estimated by maxfilter or alternatively by chpi.py. until now both methods created time courses of changes which follow almost perfectly the same track. With some datasets both results are on top of each other concerning the head position, but the rotation and tilting shows a somewhat larger difference. until now, I cannot explain this unfortunately ....

Current coverage is 85.46% (diff: 100%)

No coverage report found for master at 5355095.

Powered by Codecov. Last update 5355095...bdb60f9

With some datasets both results are on top of each other concerning the head position, but the rotation and tilting shows a somewhat larger difference. until now, I cannot explain this unfortunately ....

This is true with the version in master and with your changes here, I take it. Did the changes here make a big difference in the fits (position or rotation) sometimes?
This makes me think that I should do a phantom recording with some known tilts/rotations... it should be doable.

I did several tests - 25 subjects, 4 blocks each. Compared that to maxfilter outputs - it is very much comparable. Shall i upload some headpos files to somewhere?

Do you have any files that are preferably short (< 1 minute?) and where you knew what the subject did in terms of movement, so you know that both MaxFilter and MNE-Python are accurate? Then yes we can add them to mne-testing-data for testing purposes.

I am not fully sure about a possible improvement due to the subtraction of the mean outside of the model computation.

I am fine with mean subtraction, but if we do this then we should also remove the mean component from the model, right?
If we're going to subtract the mean outside the model, I wonder if we should also do the linear detrending outside the model, too. There is a simple scipy.signal function to do such first-order detrending so it would be easy. WDYT?

Style is a difficult issue. I am a total beginner with python.

This is a great way to learn, then, if you have the time :) Are you up for trying to fix the style issues? Travis has already rendered his judgment, which you can also get locally by doing make flake from the mne-python root directory (assuming you have flake8 installed).

This is true with the version in master and with your changes here, I take it. Did the changes here make a big difference in the fits (position or rotation) sometimes?

You mean jumps? No, jumps did not occur anymore.

This makes me think that I should do a phantom recording with some known tilts/rotations... it should be doable.

shifting in a horizontal plane would be o.k., I guess. well described rotations would be more difficult.

apply :
https://github.com/agramfort/mne-python/commit/2fdd6d9df63c5aff7899fdf6042796319f8bff35

@bmaess can you apply @agramfort's patch? You can do something like:
git remote add agramfort git://github.com/agramfort/mne-python.git
git fetch agramfort
git checkout cHPI
git cherry-pick 2fdd6d9
git push origin cHPI

@Eric89GXL  and @agramfort: I have applied the patch and pushed it to bmaess/cHPI. make flake now runs smoothly. Should I make another pull request or do you plan to run further tests?

@bmaess any change needs a test (ideally). Once you added a test to this PR, set it to MRG (not WIP) and we'll merge.
I'll let @Eric89GXL review the changes

Ideally the test would be broken on master and fixed in this PR. Perhaps
you can get a small raw FIF and pos text file (you can truncate both to
make them small) that we can add to the mne-testing-data repo to catch the
issue this fixes?

I have applied the original movement correction by maxfilter (mc) and the chpi.py routine from MNE-PYTHON to a block of real MEG data. Below you see in each figure four graphs. Each graph compares both procedures. The left upper graph shows the three quaternion parameters (rotations), the left lower graph shows the three Cartesian coordinates and the two right graphs show GOF (upper) and error and velocity (lower). I also have cropped the fiff file to the more interesting time intervals, e.g. 70-100s or 570-610s, but this changes the behavior of the chpi.py routine from the master branch. In summary, the cHPI branch results in a more stable estimation of the positions and the results are closer to the results of maxfilter.

Comparing the pos-files by maxfilter (mc) and the master-version of chpi.py (np)

Comparing the pos-files by maxfilter (mc) and the cHPI branch of chpi.py (np)

Much better!
Can you:

Crop the file to a known (currently) problematic section, say 95 - 105 sec (raw.crop(95, 105).save('test_movement_small_raw.fif')
Run maxfilter on the cropped file to get a .pos file.
Add the cropped file and resulting .pos file via PR to the mne-testing-data repository
Add a test that passes on your branch that uses these files, e.g. that the X/Y/Z differences with MF are reasonably small.
Check to ensure that test fails on master (it should easily fail)

This is what @agramfort meant to suggest with his small comment :) This may seem like a lot of work but it's really necessary to ensure future changes to this code are not harmful.

... and by "add a test" I mean add a test to mne/tests/test_chpi.py. You can get a sense of what to do from the existing tests. Speaking of which, the existing tests are passing on your PR -- I wonder if we can make some of the tests more stringent on your branch. I'll check.

@bmaess is your mne-python fork private? I can't make a PR into your branch.
In any case, can you cherry-pick https://github.com/Eric89GXL/mne-python/commit/dc1aeb6d7f2e9bc277971149adc7459389e90d1b ? It should make it easier/clearer for you to write a test. Yours should look a lot like the first few lines of test_calculate_chpi_positions up to _assert_quats.

And then also cherry-pick https://github.com/mne-tools/mne-python/commit/9d431ca8dd95d1b65566feecd2bd458dc5383901, please (adds comments, changes to more MNE-standard variable names).
FYI the head-angle-estimation accuracy did increase in the test.

@Eric89GXL - it says that I cannot make my fork private, because it is a fork from a public source. There must be another reason why it does not allow you to make a PR into my branch?
Concerning the cherry picks - I did this:
git remote add Eric89GXL git://github.com/agramfort/mne-python.git
git fetch Eric89GXL
git checkout cHPI  # was already in that branch
git cherry-pick dc1aeb6
fatal: bad revision 'dc1aeb6'
git cherry-pick 9d431ca
fatal: bad revision '9d431ca'
What is wrong?
I will create such a 10s version of my test file - the shocking observation is, however, that the results of the previous version of chpi.py did depend on the file length :( The consequence being that the 'improvement' of the newer version is less impressive ;)

You added the link using my username but Alex's URL. You can use git remote set-url to fix it.

i.e.,
$ git remote set-url Eric89GXL git://github.com/Eric89GXL/mne-python.git

the shocking observation is, however, that the results of the previous version of chpi.py did depend on the file length

This is likely a result of either 1) the sinusoidal window bounds changing slightly or 2) the prior "initial guess" changing. Both of these cause the current estimation step to depend on the previous one.
If you have your test code easily scripted to include the cropping step, you could try cropping with slightly different bounds. For example you could try a few different 1 ms steps like .crop(9.501, 10.501), .crop(9.502, 10.502), or a few different 10 ms steps. The old code will probably break under some of those because it was fragile, whereas yours will probably remain stable. But if your code isn't set up to try this easily, don't worry about it.

Argh sorry I broke a test, please also cherry pick https://github.com/Eric89GXL/mne-python/commit/cf07506b0759e65168ba3fc6f9f415defe901a67

Hi Eric, one data file is prepared: I decided to take two seconds from the beginning of the original file and combine this with 15 seconds from 95 to 110 seconds. Interestingly, this strange file can also be processed by maxfilter, but MF throws an error at the border:
ERROR: arg -hpicons has too short parameter list!
HPI amplitude correlation 28.000: 0.600 (9 chnls > 0.95)
Warning: HPI amp extraction failed (t=28.000)!
Skipped 1 bad data tags.
But, it produces the pos and a log file nevertheless.
The MF time axis starts with t0=26s - the MNE-PYTHON, however, ignores this initial shift - which I find very good and reasonable, but how do you handle the difference in your tests?
Is it possible to deactivate the automatic removal of the test results? To inspect them once the test is finished?
How can I upload the test data?

The MF time axis starts with t0=26s - the MNE-PYTHON, however, ignores this initial shift - which I find very good and reasonable, but how do you handle the difference in your tests?

That is a bit odd, actually. This is caused by raw.first_samp, and I made it in one of my commits that this is actually added in _compute_chpi_positions.
How did you create the cropped file? If you do something like this, it should respect the first_samp:
raw1 = raw_orig.copy().crop(0, 1).load_data()
raw2 = raw_orig.copy().crop(95, 110).load_data()
raw1.append(raw2)
assert raw1.first_samp == raw_orig.first_samp

Is it possible to deactivate the automatic removal of the test results? To inspect them once the test is finished?

I'm not quite sure what you mean. You mean the logging output? If you do nosetests -s then it won't capture stdout/stderr. If you just run the script like python mne/tests/test_chpi.py it will also give full output.

How can I upload the test data?

By PR to mne-testing-data. I guess here would be the proper location:
https://github.com/mne-tools/mne-testing-data/tree/master/SSS

How did you create the cropped file? If you do something like this, it should respect the first_samp:

I had used mne.concatenate_raws. But now, I followed your hint, however, the protocol is the same. More careful check of the code yields, that chpi first maps the time (seconds) to samples - and then samples back to time, by  fit_time does not make use of first_samp.

assert raw1.first_samp == raw_orig.first_samp

was not needed.

by fit_time does not make use of first_samp.

I'm not sure what you mean by fit_time... it does add first_samp in this PR here:
https://github.com/mne-tools/mne-python/pull/3875/files#diff-fa2418899264b70a3b2a18fc2314d643R583

was not needed.

Yes it's not necessary for the code to run, I put it in there to make sure it worked properly though. Does that assertion pass?

... if you look at the tests as they stand in your PR, the first_time is taken into account in _calculate_chpi_positions in the same way as MaxFilter -- you shouldn't need to mess with raw.first_samp in your testing to get it to match.

I'm not sure what you mean by fit_time... it does add first_samp in this PR here:

now I see it - you added the first_samp to the quaternion first column. I added it to the complete code - now also the log-files use the correct time (including first_samp) - however, with the current code - the pos-file has first_samp added twice. Question is - shall we remove your line adding something to the quaternions only - or shall we remove my edit - changing the fit_time?

I did not use the assert command - I used a debugger and saw it is correct - then I continued.

I added it to the complete code

Feel free to remove my line, then

Hi there,
I added a test of _calculate_chpi_positions which simulates data with ground truth head positions in https://github.com/hsonntag/mne-python/commit/a75c5d504660e9d4b40985e893b17118a88c8d4e.
nosetests mne/tests/test_simulation_chpi.py fails on the master branch and passes on https://github.com/bmaess/mne-python/commit/2dc059d2bbfe954109d7b82201544ee0acd25c63.
The distance criterion is set to 2mm.
Hope this is useful.

Hi Eric, on my way to get the test running... i have forked the repository mne-testing-data, cloned it to my local computer and created a branch cHPI there. My current problem is that mne-python does not accept the setting for the testing data path. If I run ipython:

ipython
In [1]: import mne
In [2]:  mne.datasets.testing.data_path(download=False)
Out[2]: ''
In [3]: mne.datasets.testing.data_path('/scr/eber4/anthony/mne_test_data/mne-testing-data/',download=False)
Out[3]: ''
if I allow the download, then it starts downloading, but this makes no sense as I already cloned the directory. What is the way out? Many thanks in advance,
Burkhard

@hsonntag that's great. Can you add it as part of test_chpi, just as a new function instead of a new file?
@bmaess this one is a bit tricky -- you need to pass a directory name that contains the subdirectory MNE-testing-data, so rename mne-testing-data to MNE-testing-data (or symlink) and then do
mne.datasets.testing.data_path('/scr/eber4/anthony/mne_test_data', download=False, verbose=True)

@Eric89GXL I wanted to add it as part of test_chpi.py originally but I saw that you changed the file quite a bit already in your cHPI branch. I liked the Idea of just switching branches in order to see the effect in the tests. If you merge your changes of test_chpi.py back to eg. master I would conveniently add my test to test_chpi.py. What is your opinion on that?

I liked the Idea of just switching branches in order to see the effect in the tests.

I'm not quite sure what you mean, unless you're talking about wanting to run a specific test. All tests should stay passing with a given version of code, so stacking multiple changes like this should be fine as long as they're all relevant to the changes at hand. You'd just want to make sure whatever new branch you're working from was created from where this one is currently in order to avoid merge conflicts, like:
git remote add bmaess git://github.com/bmaess/mne-python.git
git fetch bmaess
git checkout -b hpitest bmaess/cHPI
<make changes>
git commit -am 'ENH: Better tests'
git push origin chpitest

Keep in mind you can always run a specific test with e.g. nosetests mne/tests/test_chpi.py:test_calculate_chpi, so from a test development perspective this should be fine.

If you merge your changes of test_chpi.py back to eg. master I would conveniently add my test to test_chpi.py. What is your opinion on that?

Actually some of my changes don't work on master because they rely on @bmaess's changes, so they fit as part of this PR/branch.

What I mean with "just switching branches in order to see the effect" is basically your suggestion

Check to ensure that test fails on master (it should easily fail)

I can integrate it in a way that we can check the test on master

This should do the job: https://github.com/hsonntag/mne-python/commit/1d6e5a5ec865bd283867bd5b42c8ca50bc7b12e7
:)

@bmaess I sent a PR to your branch for some minor changes to @hsonntag's code.
Hopefully with those Travis will be happy and we can review/merge. Was there anything else you wanted to add or change @bmaess? If not, change the title of the PR from WIP: to MRG: to indicate you're ready for review and merge

I am ready for MRG, however I cannot edit the headline since Alex opened this PR.

now it's MRG !

LGTM :)

@bmaess FYI the reason _calculate_chpi_positions is private (prefixed by _) is because AFAIK nobody has had time to properly vet it across a large variety of cases. So if you start gathering evidence that it is as good as or better than MaxFilter, let me know, and we can work on making it public for 0.15 (0.14 will be released in a week or two, so that's too soon).
Head position estimation is also the last thing we use MaxFilter for at UW, so it would be great if we could get it working well in Python.

@bmaess https://github.com/bmaess/mne-python/pull/2 should get tests passing here (and much faster) hopefully

Ahh nuts, merge conflict due to mne/datasets/utils.py. Since the commit history is so long and has merges it'll probably be a pain to rebase. @bmaess do you want to try rebasing or would you prefer if I do it?

I could give it a try - although I have almost no idea how this should
be done.
Am 19.01.2017 um 17:47 schrieb Eric Larson:

Ahh nuts, merge conflict due to |mne/datasets/utils.py|. Since the
commit history is so long and has merges it'll probably be a pain to
rebase. @bmaess https://github.com/bmaess do you want to try
rebasing or would you prefer if I do it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mne-tools/mne-python/pull/3875#issuecomment-273829920,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHg6-QS8FUArzFHrsD2YtVmSxrjlkxc5ks5rT5OOgaJpZM4LUCMk.

Closing for #3920


15-12-18 06:00 being parsed as 2018, Dec 15

'15-12-18 06:00' becomes datetime.datetime(2018,12,15, 6, 0, 0) which IMHO should not be the first pattern it would match. I ended up using strptime instead of dateparser.parse.

Just to explain the situation, day-month-year format is the prevailing one and I can imagine applications that prefer future dates.
To obtain the proper date, you should use dateparser.parse("15-12-18 06:00", date_formats=["%y-%m-%d %H:%M"]), which, I admit, would not be far from directly using strptime.
That said, I think this problem deserves a resolution in setting, so the issue is still valid.

So that is something that is not documented, nowhere in your readme does it
show you can pass date formats
On Dec 18, 2015 6:50 PM, "Artur Sadurski"<EMAIL_ADDRESS>wrote:

Just to explain the situation, day-month-year format is the prevailing one
and I can imagine applications that prefer future dates.
To obtain the proper date, you should use dateparser.parse("15-12-18
06:00", date_formats=["%y-%m-%d %H:%M"]), which, I admit, would not be
far from directly using strptime.
That said, I think this problem deserves a resolution in setting, so the
issue is still valid.
—
Reply to this email directly or view it on GitHub
https://github.com/scrapinghub/dateparser/issues/134#issuecomment-165914727
.

Last think I want to say, mm/dd/yy is probably prevailing but I am not sure
mm-dd-yy would be. (Slash vs dash)
On Dec 18, 2015 10:31 PM, "Hatem Nassrat"<EMAIL_ADDRESS>wrote:

So that is something that is not documented, nowhere in your readme does
it show you can pass date formats
On Dec 18, 2015 6:50 PM, "Artur Sadurski"<EMAIL_ADDRESS>wrote:

Just to explain the situation, day-month-year format is the prevailing
one and I can imagine applications that prefer future dates.
To obtain the proper date, you should use dateparser.parse("15-12-18
06:00", date_formats=["%y-%m-%d %H:%M"]), which, I admit, would not be
far from directly using strptime.
That said, I think this problem deserves a resolution in setting, so the
issue is still valid.
—
Reply to this email directly or view it on GitHub
https://github.com/scrapinghub/dateparser/issues/134#issuecomment-165914727
.

A comment about the docs - what about the part in https://github.com/scrapinghub/dateparser/blob/master/README.rst#popular-formats?

@pykler we will keep ordering based on delimiters in our bucket list. Though, for now that #187 is merged, you can use new DATE_ORDER setting to address similar issues.
In [4]: parse('15-12-18 06:00', settings={'DATE_ORDER': 'YMD'})
Out[4]: datetime.datetime(2015, 12, 18, 6, 0)