Length of year: UDUNITS Gregorian or astronomical year?

Reported by ansley b manke on 15 Apr 2013 18:21 UTC
Below is some of a discussion in the THREDDS mailing list brought up the fact the Ferret "month" definition differs a little from some java code.   The  significant issue it raises is that Ferret differs from udunits in our definition of a year.  Ferret uses 364.2425 days, whereas udunits uses<PHONE_NUMBER>81 days.  (Each defines "month" as 1/12 of their respective "year".)  This makes for a % difference of 8E-7.   In 100 years this would be  .0003 days or .43 hours.
Shall we tweak our definition of year to be aligned with the astronomical year in udunits, or stay with the Gregorian year that Ferret has been using?
Things to check will be how time translations in years are affected; and the definition of month boundaries on the MONTH_IRREG climatological axis (do we change the length of February, and the boundaries of all the subsequent months?)
Here is the UDUNITS document:http://www.unidata.ucar.edu/software/udunits/udunits-1/udunits.txt
which has

...
# Interval between 2 successive passages of sun through vernal equinox
#<PHONE_NUMBER>81 days -- see
# http://www.ast.cam.ac.uk/pubinfo/leaflets/,
# http://aa.usno.navy.mil/AA/
# and http://adswww.colorado.edu/adswww/astro_coord.html):
tropical_year		P 3.15569259747e7 second
lunar_month		P 29.530589 day

common_year		P 365 day		# exact: 3.153600e7 seconds
leap_year		P 366 day		# exact
Julian_year		P 365.25 day		# exact
Gregorian_year		P 365.2425 day		# exact
...

==========================
The discussion is in the thredds mailing list at

thredds mailing list
<EMAIL_ADDRESS>For list information or to unsubscribe,  visit: http://www.unidata.ucar.edu/mailing_lists/

-------- Original Message --------
Subject: 	Re: [thredds] WMS time dimension different from OPeNDAP
Date: 	Fri, 12 Apr 2013 10:18:29 -0400
From: 	Benno Blumenthal<EMAIL_ADDRESS>To: 	John Caron<EMAIL_ADDRESS>CC: 	THREDDS<EMAIL_ADDRESS>we interpret months as implying calendar 360,  with a calendar month like conversion between calendar 360 and standard calendars, which amounts to the same thing as John's code.   So there are definitely useful interpretations of months, it is just the udunits interpretation which is useless.  And we have even a modified version of udunits, which keeps second-based and year-based time separately, and avoids messing things up.
In any case, it is a shame that a flaw in udunits incapacitates CF.
On Fri, Apr 12, 2013 at 3:52 AM, John Caron<EMAIL_ADDRESS>wrote:
Hi all:

I assume that someone who uses "months" really wants "calendar months", which manipulates the calendar fields, rather than using a fixed duration in seconds. The CDM implements this :

  http://www.unidata.ucar.edu/software/netcdf-java/CDM/CalendarDateTime.html

and i think i still have to make a formal proposal to CF for it.

However, concerns about backwards compatibility have scuttled most attempts to extend datetime handling in CF.

John

On 4/10/2013 11:59 PM, Signell, Richard wrote:

Patrick, Lansing & Co:

I note that the time units for this dataset are:
 "units: months since 1901-01-01 00:00:00"
with
"calendar: standard"

In the CF documentation
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/ch04s04.html
it warns about units of "month":

" We recommend that the unit year be used with caution. The Udunits
package defines a year to be exactly<PHONE_NUMBER>81 days (the interval
between 2 successive passages of the sun through vernal equinox). It
is not a calendar year. Udunits includes the following definitions for
years: a common_year is 365 days, a leap_year is 366 days, a
Julian_year is 365.25 days, and a Gregorian_year is 365.2425 days.

For similar reasons the unit month, which is defined in udunits.dat to
be exactly year/12, should also be used with caution."

Perhaps the WMS and OPeNDAP service are not handling units of month in
the same way?

-Rich

On Wed, Apr 10, 2013 at 5:45 PM, Lansing Madry<EMAIL_ADDRESS>wrote:

Hi Patrick,

I pulled down the file via http service and looked at it in ToolsUI.
ToolsUI (perhaps unsurprisingly) gives the "1951-07-16T12:12:04.339Z" value.
I'm not familiar with ferret as an OpenDAP client or viewer, have you heard
from anyone on that front?

Regards,
  Lansing Madry
  Unidata
  Boulder, Colorado

On 4/9/2013 11:08 AM, Patrick Brockmann wrote:

Hi all,

I have a strange behaviour with time dimension values exposed by the
GetCapabilities
and values get from an OPeNDAP access.
Could someone explain me what is the source of the problem ?

http://webportals.ipsl.jussieu.fr/thredds/wms/CARBON/CARBONES_PRIOR_orchidee_hadgem2-es_historical_nosoc_co2_npp_monthly_1951_2004_yearmonmean_3.nc?service=WMS&version=1.3.0&request=GetCapabilities

---> 1951-07-16T12:12:04.339Z, ....

And from the OPeNDAP access with ferret client, I get

ferret
yes? use
"http://webportals.ipsl.jussieu.fr/thredds/dodsC/CARBON/CARBONES_PRIOR_orchidee_hadgem2-es_historical_nosoc_co2_npp_monthly_1951_2004_yearmonmean_3.nc"
yes? sg/t/l=1:10 npp2
    GRID GTL2
 name       axis              # pts   start                end
 MYLON     LONGITUDE          720mr   0E                   0.5W
 LAT       LATITUDE           360 r   89.75S               89.75N
 normal    Z
 TIME      TIME                54 r   16-JUL-1951 12:33 15-JUL-2004 09:01

       L     T                   TBOX      TBOXLO TSTEP (months)
       1>  16-JUL-1951 12:33:59  12         14-JAN-1951 21:39:23 606.4526
       2>  15-JUL-1952 18:23:11  12         15-JAN-1952 03:28:35 618.4526
       3>  16-JUL-1953 00:12:23  12         14-JAN-1953 09:17:47 630.4526

--> Note the difference between 1951-07-16T12:12:04.339Z and 16-JUL-1951
12:33:59

My TDS version is Version 4.3.15 - 20121218.1126

Regards
Patrick

_______________________________________________
thredds mailing list
<EMAIL_ADDRESS>For list information or to unsubscribe,  visit:
http://www.unidata.ucar.edu/mailing_lists/

_______________________________________________
thredds mailing list
<EMAIL_ADDRESS>For list information or to unsubscribe,  visit: http://www.unidata.ucar.edu/mailing_lists/

--
Dr. M. Benno Blumenthal<EMAIL_ADDRESS>International Research Institute for climate and society
The Earth Institute at Columbia University
Lamont Campus, Palisades NY 10964-8000<PHONE_NUMBER>
Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/2061

Comment by andrew.wittenberg on 15 Apr 2013 19:06 UTC
I'd strongly recommend AGAINST changing the default Gregorian (365.2425-day) calendar, as it wouldn't help the vast majority of users, and would have severe and pervasive effects on existing scripts etc.
The drift problem is poorly defined anyhow, given the uncertainty in the actual lengths of days and years.  Even ENSO can alter the length of a day slightly, by temporarily changing Earth's moment of inertia.  See
http://en.wikipedia.org/wiki/Tropical_year#Calendar_year
If this is really needed by someone, I'd suggest adding a new (non-default) calendar called SOLAR or TROPICAL, which has leap-seconds, leap-hours, and/or leap-days added as necessary (according to some established convention), so that it averages out to the requisite<PHONE_NUMBER>81 days.

Comment by steven.c.hankin on 15 Apr 2013 19:16 UTC
Hi Andrew,
There's no eagerness to make this change, believe me, but we'd like to make sure we understand the lay of the land before finalizing any decision.  By my calculation the change would remove a "drift" of a mere 4.3 hours in a 1000 year model run.
The "up" side is small -- only the ability to claim compatibility with udunits.  It's only an issue of being good citizens wrt strict standards compliance.
Can you explain the "down" side -- how the 4.3 hour drift change would adversely effect backwards compatibility, and whether there's anything that might mitigate the negative impacts?

Comment by andrew.wittenberg on 15 Apr 2013 19:33 UTC
Presumably the drift would be adjusted by adding leap-sec/min/days, right?  So that's a different calendar from Gregorian, which is an international standard, and is by definition 365.2425 days on average.
I don't think we'd want to change the default calendar out from under everyone, especially if it broke backwards compatibility(by shifting dates around in existing files/scripts etc.
I'm fine with adding another ("SOLAR") calendar that removes the slight drift.  But it would probably have a small audience -- mainly those dealing with paleo runs.

Comment by ansley.b.manke on 15 Apr 2013 21:11 UTC
There is an argument to be made that we ARE using Udunits, which lists a Gregorian calendar at 365.2425 days, and that's the definition of the Ferret default calendar.  The way the Udunits document reads, the description of the year of length<PHONE_NUMBER>81 days is contained in a comment.
The audience for the change would be those who compare Ferret's results other software which applies another length for the default year.

Comment by andrew.wittenberg on 15 Apr 2013 21:43 UTC
I agree.  It seems the problem is that these folks are trying to operate on a dataset with insufficient metadata (missing calendar attribute), and these two pieces of software are each trying to apply a "reasonable" default -- which happens to differ between the two programs.
I happen to think that Ferret's default is the more "reasonable" one for most cases.  :-)  And it's even Udunits-compliant, as long as the calendar attribute is provided.

Comment by steven.c.hankin on 15 Apr 2013 23:10 UTC
OK.  This is a non-problem after all.  In fact, it is only the CF document that is in error.
I will take it up with the CF group.
Case closed.  Thanks.


[3.0] Implement SPACESHIP_VENV_GENERIC_NAMES, better naming for generically named virtualenvs

If a virtualenv folder has a generic name in the array SPACESHIP_VENV_GENERIC_NAMES, use its parent folder as its name instead.
Some people like to create a project specific virtualenv under the project folder with a generic name like "venv", but than Spaceship will show it as via venv, with this PR it will use the parent folder name instead which is often a much better name to show in such cases.
Another option is to grep the activate script for the user settable prompt (--prompt) and shop it with the surrounding parentheses and whitespace stripped. Though that might be a bit sensitive to virtualenv versions (And the builtin Python 3 venv).

Thanks for contributing. 3.0 has settled a list of features. I'll be glad to merge it after releasing 3.0.

Any update on this @denysdovhan @maximbaz ?


tensor2image and dataloder length

Hi,
I have two questions now

In pytorch, we can convert tensors to images and save on disk using from torchvision.utils import save_image. How to convert a tensor to image and save it on disk using burn ?
In pytorch, we can get batch numbers using len(dataloader). How to get the batch numbers in a dataloader in burn ? It seems that dataloader.num_items only returns the total number of samples, not the batch number.

There is no utility similar to the pytorch vision extension for save_image in Burn atm. But this should be fairly straightforward using the tensor data and constructing an image buffer to save (e.g., Rgb32FImage)
The DataLoader trait doesn't expose the number of batches in a dataloader. It returns an iterator to iterate over the batches any fixed-size assumption. The dataloader length can only be known for a fixed-size dataset, so even pytorch makes assumption for different dataset types and might report inaccurate length (as in their disclaimer). So for now we chose not to make any assumptions 🙂


Send tracer events to cli in test mode

Synchronously sends events to the cli when we detect a cli environment variable. The cli just collects these into an array, so doing this synchronously will have minimal impact on the test run.
Uses context variables to associate test id and test case hash. The test is setup so that it verifies when we have multiple test cases running at a time, everything is working correctly.

I did also test this in a python project (user feedback clustering)


API conformance check has problems with nullable types

The API conformance check didn't caught the divergance of the API which was fixed in this PR https://github.com/flutter/engine/pull/34428. Originally this divergance was submitted in https://github.com/flutter/engine/pull/32078.
A possible cause could be due to the check not knowing about null-safe code.

Will probably get fixed in https://github.com/flutter/engine/pull/34428

This got fixed in https://github.com/flutter/engine/pull/34428


CRS-875 Display reason for calculation in the view journey

Added the reason the to CalculatedReleaseDates so that it can be displayed in the view journey.
Updated the tests to reflect this change.

HI @dave-virden  - Could you add a screenshot or a video to this PR? It would be helpful to get an idea of the impact.


Harden the spo and bpf-recorder containers with custom apparmor profiles

What type of PR is this?

/kind feature
What this PR does / why we need it:
This pull request harden the security-profiles-operator and bpf-recorder containers as part of spod daemonset with custom apparmor profiles when apparmor is enabled.
These two containers run in privileged mode when the apparmor is activated.
Which issue(s) this PR fixes:

Fixes #65
Does this PR have test?

Yes
Special notes for your reviewer:
Does this PR introduce a user-facing change?

Harden the security-profiles-operator and bpf-recorder containers with custom apparmor profiles when apparmor is enabled.

cc @mhils


Add more linters to the lint_modern target.

Also delete some redundant config files.

@geeknoid: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

lint_istio
e291d82c45d842ca0a18ccfeb2ed6913b82c3274
link
/test lint

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@geeknoid: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

lint_istio
e291d82c45d842ca0a18ccfeb2ed6913b82c3274
link
/test lint

lintmodern_istio
a7a559193b94e568f3a99efe01c4573ea4172116
link
/test lintmodern

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@geeknoid: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

lint_istio
e291d82c45d842ca0a18ccfeb2ed6913b82c3274
link
/test lint

lintmodern_istio
485454ae657f1e8ba29a16089a7d6bf88683bfc8
link
/test lintmodern

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@geeknoid: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-simpleTests-cni_istio
c2c4cc1dfbc04ffe8989c2732b25f6df27f9d4b7
link
/test e2e-simpleTests-cni

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test e2e-simpleTests-cni


How to install latest ssh through sccop?

https://www.openssh.com/portable.html
I don't understand who care about main bucket and who control it?
Does it refer to official app repos or bucket's author have to prepare own repo and installation code for each app?
Who and how often updates app version in scoop buckets?
How can I install app that are not in known scoop buckets?

This github repo is the main bucket. The bucket directory contains text files called manifests that describe where and how to download and install applications. These manifests are updates programmatically for the most part, with fields called checkver and autoupdate in each manifest.
 Many people create third party buckets.
For further information, refer to the wiki.

@Deide Thanks
I have read wiki autoupdate and checkvers and seen several manifests.
I tried to execute .\bin\checkver.ps1 * command in powershell and get checkver list I suppose.
Obviously, there are no checkver and autoupdate in openssh.json
openssh manifest in main bucket is rather different than others and rather difficult for me to prepare correct new one manifest.
I am looking for something folk good solution.
What's the problem with openssh and other such apps? Why author or somebody goodman can't integrate checkver and autoupdate feauture? I try to figure it out to understand do I need to try update manifest and do pull request.
I saw that official repo is specific but there is also the github mirror

In the openssh case, it's updated manually. It's the msys2 OpenSSH. If you can work out a sensible checkver and autoupdate, by all means submit a pull request.
It's worth mentioning that Windows 10 comes with OpenSSH these days, and if not enabled, can usually be enabled under Optional Features. If it's more desirable to use a scoop managed version, I would recommend win32-openssh, which does have checkver and autoupdate. It is a port, so it's not entirely equal.
Maybe this website will be useful to you.


Incorrect failure message from toEqual

🐛 Bug Report
The following expectation fails in an unexpected way:
expect([]).toEqual([new MouseEvent('click')]);

gives:
TypeError: Cannot read property 'includes' of undefined

To Reproduce
Add the above expectation to any existing Jest test, or clone this minimal reproduction.
Expected behavior
I would expect the failure message to be something like:
    expect(received).toEqual(expected) // deep equality

    - Expected  - 3
    + Received  + 1

    - Array [
    -   MouseEvent {},
    - ]
    + Array []

Link to repl or repo (highly encouraged)
https://gitlab.com/markrian/jest-bug
envinfo

  System:
    OS: Linux 5.4 Ubuntu 20.04.2 LTS (Focal Fossa)
    CPU: (8) x64 Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
  Binaries:
    Node: 12.18.4 - ~/.asdf/installs/nodejs/12.18.4/bin/node
    Yarn: 1.22.5 - ~/.asdf/shims/yarn
    npm: 7.5.4 - ~/.asdf/installs/nodejs/12.18.4/.npm/bin/npm
  npmPackages:
    jest: ^26.6.3 => 26.6.3

Running into this too. Pretty annoying. Have you had a chance to look at where the includes call happens?

@gaearon Yes, and it's not pretty :smile: It's actually throwing in JSDOM!
TypeError: Cannot read property 'includes' of undefined
    at Object.exports.hasAttribute (~/jest-bug/node_modules/jsdom/lib/jsdom/living/attributes.js:15:33)
    at ~/jest-bug/node_modules/pretty-format/build/plugins/DOMElement.js:34:34
    at testNode (~/jest-bug/node_modules/pretty-format/build/plugins/DOMElement.js:38:9)
    at Object.test (~/jest-bug/node_modules/pretty-format/build/plugins/DOMElement.js:57:35)
    at deepCyclicCopyReplaceable (~/jest-bug/node_modules/jest-matcher-utils/build/deepCyclicCopyReplaceable.js:50:49)
    at ~/jest-bug/node_modules/jest-matcher-utils/build/deepCyclicCopyReplaceable.js:75:16
    at Array.reduce (<anonymous>)
    at deepCyclicCopyObject (~/jest-bug/node_modules/jest-matcher-utils/build/deepCyclicCopyReplaceable.js:67:5)
    at deepCyclicCopyReplaceable (~/jest-bug/node_modules/jest-matcher-utils/build/deepCyclicCopyReplaceable.js:53:14)
    at deepCyclicCopyArray (~/jest-bug/node_modules/jest-matcher-utils/build/deepCyclicCopyReplaceable.js:98:19)

It's a bit tricky to follow, but it looks like the toEqual matcher ends up traversing module objects, which eventually leads to this non-sensical call of hasAttribute("is", undefined) call:

The deepCyclicCopyObject ends up recursing through the properties of the mouse event. The chain of objects it goes through is long, but it goes something like this:
MouseEvent[Symbol(impl)] ->
MouseEventImpl['_globalObject'] ->
Window['globalThis'] ->
Window['jasmine'] ->
jasmine['process'] ->
process['mainModule'] -> # This is bin/jest.js! See where this is going? ;)
mainModule['children'] ->
[... snip! Many modules later ...]
moduleWhichIsJSDOMAttributes['exports']

The final object in that chain is the exports object from JSDOM's lib/jsdom/living/attributes.js file. It has hasAttribute as a key (i.e., it exports it), which the https://github.com/facebook/jest/blob/v26.6.2/packages/pretty-format/src/plugins/DOMElement.ts#L30 to determine incorrectly that the module is actually a DOMElement.
Having said all this, I just now discovered that this has likely been fixed by https://github.com/facebook/jest/commit/33c3b0718169ffcacce1d2a3ad6b7d77f27deec9 (https://github.com/facebook/jest/pull/11189)! :tada:
So, I think I'll close this. But, this investigation shows me that using toEqual on host objects like events is really quite expensive, and is doing a lot more than I would have expected.


Axes labels

Motivation and Context

Adding axes labels to the dashboard graphs to avoid confusion
Your Changes

Description:
added internationalized labels to the dashboard graphs using "options"
Type of change (select all that apply):

[x] New feature (non-breaking change which adds functionality)

Testing

tested using rspec and manually going through the UI
Questions and Comments (if applicable)

Checklist

[x] I have performed a self-review of my own code.
[x] I have fixed any Hound bot comments.
[x] I have verified that the CI tests have passed.
[x] I have reviewed the test coverage changes reported on Coveralls.

Required documentation changes (if applicable)

Thanks, @emilychangg! I ended up pulling out the common scales object into a separate helper file, just to make the code a bit cleaner.


Fix ENCODING_DTS_UHD_P2 Issue

During Direct Passthrough Audio Playback on platforms that supports AudioFormat.ENCODING_DTS_UHD_P2, we need to include C.ENCODING_DTS_UHD_P2 in DefaultAudioSink.getFramesPerEncodedSample() and DefaultAudioTrackBufferSizeProvider.getMaximumEncodedRateBytesPerSecond().

@microkatz Here is the new PR that merges into main branch. Thanks.

@cedricxperi
Thanks for creating the new PR! One quick point, I think you may have missed a spot in the DefaultAudioTrackBufferSizeProvider.getPassthroughBufferSizeInBytes method? Would ENCODING_DTS_UHD_P2 need the same multiple for the buffersize computation for passthrough as ENCODING_DTS_HD?

@cedricxperi
Thanks for creating the new PR! One quick point, I think you may have missed a spot in the DefaultAudioTrackBufferSizeProvider.getPassthroughBufferSizeInBytes method? Would ENCODING_DTS_UHD_P2 need the same multiple for the buffersize computation for passthrough as ENCODING_DTS_HD?

@microkatz Thanks for pointing that out. However, DTS:X Profile 2 has a smaller number of audio samples per channel per frame compared to DTS-HD (DTS Express). So it does not require a bigger multiple for buffersize computation.  :)

I'm going to send this for internal review now. You may see some more commits being added as I make changes in response to review feedback. Please refrain from pushing any more substantive changes as it will complicate the internal review - thanks!


Not strictly use angular 2 beta 0

I'm reluctant to do this because they're still bringing in lots of breaking changes at the moment

In fact, it seems that beta 1 brings in a few DI problems. Closing this for now, until I have a chance to resolve the problems.


separating into a library and a binary

This allows the library to be used in other projects

Looking good, thanks!


[WIP][SPARK-27366][CORE] Support GPU Resources in Spark job scheduling

What changes were proposed in this pull request?
This PR adds support to schedule tasks with GPU resource requirements on executors with available GPU resources. A few major features introduced in this PR:

Represent resources by ResourceInformation;
Use static config spark.[executor|driver].resource.${resourceName}.[count|address] to specify available resources on driver or executors;
Use static config spark.task.resource.${resourceName}.count to specify resources required by each task;
Provide TaskContext.resources() method so tasks can access available device indices allocated to them.

How was this patch tested?

Added new end-to-end test cases in SparkContextSuite;
Added new test case in CoarseGrainedSchedulerBackendSuite;
Added new test case in TaskSchedulerImplSuite;
Added new test case in SparkConfSuite;
Updated existing tests.

cc @tgravescs @squito @mengxr @cloud-fan

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9909/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9909/
Test PASSed.

Test build #104586 has started for PR 24374 at commit aa0d9ae.

Test build #104586 has finished for PR 24374 at commit aa0d9ae.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ResourceInformation(
  case class StatusUpdate(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104586/
Test FAILed.

This pr also contains more then what is for this particular jira. SPARK-27024 and SPARK-27374  We should either add those jira to the header here or split it apart. If its hard to split apart and others are ok with it, I'm ok with it.
Please note that I wrote a bunch of this code for the executor side and discovery changes, so someone else will have to officially approve this.

@jiangxb1987 I think we should reduce the scope of this PR:

Remove auto-discovery script and executor interface.
Do not consider gpu as a special hardcoded resource name.
Use static conf resource.accelerator_name.addresses to load accelerator addresses and resource.accelerator_name.count as request.

In this way, we can make this PR smaller and leave discovery and executor interface as follow-up work.

@mengxr your point 3 I think has to deal with the executor side and the configs, so if we split those it would apply there. I'm also not sure what you mean by using a static conf there, how do you have a static conf that isn't hardcoded?  You would have to build it on the fly to put in the resource type/accelerator_name.
If we want to remove gpu as known (hardcoded) resource name then we should go all the way generic on the scheduler pieces.  I was going to do that in a follow on PR but we could do it here as well if people prefer. Or are you just saying remove the hardcoded configs and still only do gpu for now?

I'll go ahead and try to split out the executor and discovery pieces from this.  I think that may be easier to PR first on its own and then build the scheduler pieces next since it will need the resource information from the executor to make it decision.  I'll try to apply everyone's comments above to that part and update here if it works out.

One general thought I have -- there seems to be a lot of changes to do general resource tracking, though only gpus are supported here.  These are all internal classes, so I'm wondering whether its useful to even put in those abstractions now.  Is FPGA support (or whatever other special hardware) still years away?  If nobody has at least experimented with it at all, are we sure that the generalizations you're putting in would even be useful in those cases?
I don't really know anything about other accelerators, so I don't have any strong feelings here, just a general concern about putting in abstractions too early.  Just wanted to mention it, I'll leave it up to you.

I do know there is a company that sells a product to run Spark on FPGAs. I've also seen multiple talks on it.  But I don't know how close it is for general user.
Personally I think it would be good to just do the generic thing as long as it doesn't add much overhead to the scheduler.  I can see people wanting to add in possibly other information like GPU type, memory, etc.  I can see a use for virtual GPU's if they want to share a GPU.
I was thinking if we put in the generic pieces people could more easily experiment without having to change core pieces.

ok that makes sense, like I said I'm willing to trust your judgement on that.  Might be helpful to reach out to that company to at least see if they see any red flags with this approach.

took me a bit longer to split the executor side stuff out and make fully generic, should have PR up tomorrow.  My thought was I could put that up which gives the base for for @jiangxb1987 to do the scheduler changes on top of.

I submitted PR https://github.com/apache/spark/pull/24394 to replace this.  @jiangxb1987  can we close this one?
thanks for everyone's reviews. I tried to apply everyone's comments from here that were applicable to that pr, if missed anything please comment there.

@tgravescs Sure I'm closing this one, thanks!

sorry for any confusion, the executor pr is https://github.com/apache/spark/pull/24406

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10029/
Test PASSed.

Test build #104716 has started for PR 24374 at commit a251551.

Test build #104716 has finished for PR 24374 at commit a251551.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104716/
Test FAILed.

Test build #104717 has started for PR 24374 at commit 40eaca0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10030/
Test PASSed.

Test build #104717 has finished for PR 24374 at commit 40eaca0.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104717/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10033/
Test PASSed.

Test build #104720 has started for PR 24374 at commit fe7ed1b.

Test build #104720 has finished for PR 24374 at commit fe7ed1b.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104720/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10035/
Test PASSed.

Test build #104722 has started for PR 24374 at commit 054806a.

Test build #104722 has finished for PR 24374 at commit 054806a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104722/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10388/
Test PASSed.

Test build #105098 has started for PR 24374 at commit 5d5bd46.

Test build #105098 has finished for PR 24374 at commit 5d5bd46.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ResourceInformation(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105098/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10658/
Test PASSed.

Test build #105395 has started for PR 24374 at commit 04c9816.

Test build #105395 has finished for PR 24374 at commit 04c9816.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105395/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10888/
Test PASSed.

Test build #105634 has started for PR 24374 at commit 691a121.

Test build #105634 has finished for PR 24374 at commit 691a121.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105634/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10896/
Test PASSed.

Test build #105642 has started for PR 24374 at commit 98a0309.

Test build #105642 has finished for PR 24374 at commit 98a0309.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105642/
Test FAILed.

Test build #105645 has started for PR 24374 at commit 5d099e1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10899/
Test PASSed.

Test build #105645 has finished for PR 24374 at commit 5d099e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105645/
Test PASSed.

@jiangxb1987 is this ready for another review?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10996/
Test PASSed.

Test build #105740 has started for PR 24374 at commit fcff234.

Test build #105740 has finished for PR 24374 at commit fcff234.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105740/
Test PASSed.

Test build #105885 has started for PR 24374 at commit 3782fdf.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11141/
Test PASSed.

Remaining works:

Add test in CoarseGrainedExecutorBackendSuite and CoarseGrainedSchedulerBackendSuite;
Add check to make sure the executor resources are a multiple of task requirements.

Test build #105885 has finished for PR 24374 at commit 3782fdf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105885/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11185/
Test PASSed.

Test build #105933 has started for PR 24374 at commit a63dcf4.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11187/
Test PASSed.

Test build #105935 has started for PR 24374 at commit 1ac1678.

Test build #105933 has finished for PR 24374 at commit a63dcf4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105933/
Test FAILed.

Test build #105935 has finished for PR 24374 at commit 1ac1678.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105935/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11228/
Test PASSed.

Test build #105977 has started for PR 24374 at commit 59bcb83.

Test build #105977 has finished for PR 24374 at commit 59bcb83.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105977/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11236/
Test PASSed.

Test build #105985 has started for PR 24374 at commit 8f51971.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11237/
Test PASSed.

Test build #105986 has started for PR 24374 at commit a3c1db5.

Test build #105985 has finished for PR 24374 at commit 8f51971.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105985/
Test PASSed.

Test build #105986 has finished for PR 24374 at commit a3c1db5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105986/
Test PASSed.

Discussed with @jiangxb1987 offline:

In WorkOffer, we put a copy of available addresses. Modification to it doesn't change actual booking.
At the same location where we update the booking of available CPU cores, we do the same for resources.
ExecutorResourceInfo will implement:

def acquire(addrs: Seq[String]): Unit
def release(addrs: Seq[String]): Unit
def availableAddrs: Seq[String]

Internally we use mutable.Map[String, Boolean] to record whether an address is used or not. Later we will switch to OpenHashMap for performance optimization.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11288/
Test PASSed.

Test build #106038 has started for PR 24374 at commit f388338.

Test build #106040 has started for PR 24374 at commit d15a51d.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11291/
Test PASSed.

Test build #106038 has finished for PR 24374 at commit f388338.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106038/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11292/
Test PASSed.

Test build #106042 has started for PR 24374 at commit dcc147e.

Test build #106040 has finished for PR 24374 at commit d15a51d.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106040/
Test PASSed.

Test build #106042 has finished for PR 24374 at commit dcc147e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106042/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11387/
Test PASSed.

Test build #106135 has started for PR 24374 at commit cd01cae.

Test build #106135 has finished for PR 24374 at commit cd01cae.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106135/
Test FAILed.

test this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11405/
Test PASSed.

Test build #106152 has started for PR 24374 at commit cd01cae.

Test build #106152 has finished for PR 24374 at commit cd01cae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106152/
Test PASSed.

Test build #106164 has started for PR 24374 at commit e539097.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11415/
Test PASSed.

Test build #106167 has started for PR 24374 at commit 82cd1e3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11418/
Test PASSed.

Test build #106164 has finished for PR 24374 at commit e539097.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106164/
Test PASSed.

Test build #106167 has finished for PR 24374 at commit 82cd1e3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106167/
Test PASSed.

LGTM and merged into master. Thanks!

Thanks very much! @tgravescs @mengxr @squito @srowen @viirya @kiszk


Symfony connection redis cluster

Hello,
Symfony connection redis cluster encountered an error
Error is as follows
Warning: call_user_func_array() expects parameter 1 to be a valid callback, no array or string given
I use the third party bundle
"snc/redis-bundle": "2.*",
"predis/predis": "^1.0"
This is my configuration
snc_redis:
clients:
default:
type: predis
alias: default
dsn: redis://<IP_ADDRESS>:6380
logging: %kernel.debug%
cache:
type: predis
alias: cache
dsn: redis://<IP_ADDRESS>:6381
options:
profile: 2.2
connection_timeout: 10
read_write_timeout: 30
cluster:
type: predis
alias: cluster
dsn:
- redis://<IP_ADDRESS>:6380
- redis://<IP_ADDRESS>:6381
- redis://<IP_ADDRESS>:6382
- redis://<IP_ADDRESS>:6383
- redis://<IP_ADDRESS>:6384
- redis://<IP_ADDRESS>:6385
it raises an exception in the predis client (predis/predis/src/Client.php:162):
I am eager to get help. In this first thank you

I'm experiencing the same issue and found out that snc's bundle sends "replication" => false in predis configuration (DependencyInjection/Configuration/Configuration.php:138)
But it seems that predis doesn't support replication:false parameter.
Predis documentation says that replication accepts only TRUE or callable object.
commenting out line
->booleanNode('replication')->defaultFalse()->end()
in DependencyInjection/Configuration/Configuration.php
makes my case up and running.

Try to reduce version of predis bundle:
change composer settings from:
"predis/predis": "^1.0"
to:
"predis/predis": "1.0.*"
predis 1.1 and 2.* don't support replication:false as a default value.

Thank you very much.I solved the problem of success.But a new error.
Error message:
ERROR - Command "INCR foo:bar" failed (MOVED 9094 <IP_ADDRESS>:6381)
CRITICAL - Uncaught PHP Exception Predis\Response\ServerException: "MOVED 9094 <IP_ADDRESS>:6381" at /home/fujie/www/mySymfony/vendor/predis/predis/src/Client.php line 365
I am study it.

My problem has been solved, is less a configuration.
options:
profile: 2.4
connection_timeout: 10
connection_persistent: true
read_write_timeout: 30
iterable_multibulk: false
throw_errors: true
cluster: redis-cluster
replication: false

No longer an issue with current versions.


changed text from fleet to edge

Description
Replace text from Fleet to Edge
Change route from /fleet-management to /edge-management
NOTE: Need to make another PR to cloud-services-config to get nav working properly
Fixes # (issue)
Type of change
What is it?

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Documentation update
[ ] Tests update

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[x] I run npm run lint:js:fix to check that my code is properly formatted

Codecov Report

Merging #388 (5c69b13) into master (b13309c) will decrease coverage by 3.17%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master     #388      +/-   ##
==========================================
- Coverage   34.30%   31.12%   -3.18%
==========================================
  Files         118      118
  Lines        2586     2586
  Branches      749      749
==========================================
- Hits          887      805      -82
- Misses       1557     1635      +78
- Partials      142      146       +4

Impacted Files
Coverage Δ

src/Routes.js
0.00% <ø> (ø)

src/Routes/Devices/Devices.js
0.00% <0.00%> (ø)

src/Routes/Groups/GroupTable.js
11.42% <ø> (-17.15%)
:arrow_down:

src/Routes/GroupsDetail/GroupsDetail.js
2.56% <0.00%> (-41.03%)
:arrow_down:

src/Routes/Repositories/RepositoryHeader.js
100.00% <ø> (ø)

src/hooks/useApi.js
4.76% <0.00%> (-66.67%)
:arrow_down:

src/components/general-table/ToolbarKebab.js
25.00% <0.00%> (-62.50%)
:arrow_down:

src/Routes/Groups/Groups.js
3.84% <0.00%> (-57.70%)
:arrow_down:

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b13309c...5c69b13. Read the comment docs.


Implement boundary conditions for CurvedMesh in 1D and 2D

Boundary conditions in 3D can be implemented once #513 is merged.
@sloede, should we leave this PR as is and create a new one for 3D boundary conditions, or should we wait until #513 is merged to add the 3D stuff here?

I adapted the mesh to allow for periodicity in only certain dimensions like the TreeMesh implementation does.
To do this, I created a function calc_boundary_flux_by_direction! like for TreeMesh.
The need to pass two different sets of indices to this function made it not as elegant as I hoped, but I have a lot less redundand code now.
I tested both versions for performance:
julia> @benchmark Trixi.calc_boundary_flux!($cache, $u, $t, $boundary_conditions, $equations, $mesh, $dg)

TreeMesh implementation
BenchmarkTools.Trial:
  memory estimate:  0 bytes
  allocs estimate:  0
  --------------
  minimum time:     6.600 μs (0.00% GC)
  median time:      6.680 μs (0.00% GC)
  mean time:        6.716 μs (0.00% GC)
  maximum time:     13.680 μs (0.00% GC)
  --------------
  samples:          10000
  evals/sample:     5

Old implementation without calc_boundary_flux_by_direction!
BenchmarkTools.Trial:
  memory estimate:  0 bytes
  allocs estimate:  0
  --------------
  minimum time:     6.680 μs (0.00% GC)
  median time:      6.760 μs (0.00% GC)
  mean time:        6.786 μs (0.00% GC)
  maximum time:     11.400 μs (0.00% GC)
  --------------
  samples:          10000
  evals/sample:     5

New implementation with calc_boundary_flux_by_direction!
BenchmarkTools.Trial:
  memory estimate:  0 bytes
  allocs estimate:  0
  --------------
  minimum time:     6.740 μs (0.00% GC)
  median time:      6.840 μs (0.00% GC)
  mean time:        6.874 μs (0.00% GC)
  maximum time:     11.560 μs (0.00% GC)
  --------------
  samples:          10000
  evals/sample:     5

Unfortunately, I have no idea how to improve this further.

I adapted the mesh to allow for periodicity in only certain dimensions like the TreeMesh implementation does.
To do this, I created a function calc_boundary_flux_by_direction! like for TreeMesh.
The need to pass two different sets of indices to this function made it not as elegant as I hoped, but I have a lot less redundand code now.
I tested both versions for performance:
Unfortunately, I have no idea how to improve this further.

The performance does not look too bad to me? You might try sprinkling some @inbounds function calc_boundary_flux_by_direction! to the definitions. From a quick look, the difference between your version of calc_boundary_flux_by_direction! for curved meshes and the one for TreeMeshes seems to be the level where they are called - lower level for curved meshes, so inlining could help a lot there, I guess

Hmmm, no difference for the advection simulations but a little faster for Euler (about 4%).

Hmmm, no difference for the advection simulations but a little faster for Euler (about 4%).

So it's even faster than tree-Trixi? That sounds good :+1:

Hmmm, no difference for the advection simulations but a little faster for Euler (about 4%).

I wouldn't be a big fan of using @inbounds, but the reported performance is already on par with TreeTrixi, so I would not necessarily change anything

Sorry, I'm doing too many things in parallel - I meant @inline and not @inbounds. Sorry!

Auto-merge seems to be blocked by a coverage increase (what the...!?!?), thus I am merging this manually.


Performance improvement in futures handling

As said in the following discussions:
https://github.com/emmett-framework/granian/discussions/30
Almost sure this optimization will make granian ASGI faster than uvicorn+uvloop+httptools+gunicorn (in local tests is)
PS the same improvement can be made in add_done_callback in src/runtime.rs, src/generic.rs and src/lib.rs

Looks good! Thank you @cirospaciari


WFLY-9037 Cluster topology updates are lossy

https://issues.jboss.org/browse/WFLY-9037
N.B. All ejb client clustering tests are now working, with the exception of the authentication one, which has more to do with my inability to setup the test correctly.
We can merge a fix for that later.
Farah has provided a workaround, now incorporated as a separate test using a global authentication context.

@pferraro Unfortunately we seem to have a failure in https://ci.wildfly.org/viewLog.html?buildId=66718&tab=buildResultsDiv&buildTypeId=WFPR

@kabir That run was outdated - I've kicked off another.

@pferraro Still seems problematic, unless the runs still are stale. For linux I've kicked off builds #12863-65 and for windows #7896-98.

@kabir The linux runs all passed like a champ.  The windows builds are failing all over the place and in the weirdest of places.  The downstream windows builds look fine.

Retest this please

Retest this please


feat(config): KP-SW-07 and KP-SW-08 Keemple wall switch templates added

Templates for Keemple wall switch devices:

KP-SW-07
KP-SW-08
added.

@zwave-js-bot fix lint

@zwave-js-bot fix lint

Why close?

I closed it because I had problems with run checks on mobile app. I planed open pull request today again, but I see I don't need to


Cancel any pending animation frames on stopRenderLoop

When calling stopRenderLoop, we weren't canceling any animation frames requested, this was causing a problem in the multiview scenario: https://forum.babylonjs.com/t/stop-render-loop-and-keep-last-frame-rendered-multiview/42847/3
Test PG: #LYSEQ9#2 (starts and stops a render loop and prints on begin/end frame), in the current version there's a log even after the stopRenderLoop:

Snapshot stored with reference name:
refs/pull/14102/merge
Test environment:
https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14102/merge/index.html
To test a playground add it to the URL, for example:
https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14102/merge/index.html#WGZLGJ#4600
Links to test babylon tools with this snapshot:
https://playground.babylonjs.com/?snapshot=refs/pull/14102/merge
https://sandbox.babylonjs.com/?snapshot=refs/pull/14102/merge
https://gui.babylonjs.com/?snapshot=refs/pull/14102/merge
https://nme.babylonjs.com/?snapshot=refs/pull/14102/merge
To test the snapshot in the playground with a playground ID add it after the snapshot query string:
https://playground.babylonjs.com/?snapshot=refs/pull/14102/merge#BCU1XR#0


Comparing installed versions of a package with the required version is not reliable

The code in in RedHat/installation.yml is:
echo -e "$required_pkg\n$(rpm -q {{ pkg[0] }} )"| sort -V | tail -1
In case of RHEL 7.7 GA (kernel 3.10.0-1062.el7), with a minimum required kernel version+release of  3.10.0-1062.1.1.el7, the following command:
echo -e "$required_pkg\n$(rpm -q {{ pkg[0] }} )"| sort -V
(or, emulated:
echo -e "kernel-3.10.0-1062.1.1.el7\n$(rpm -q kernel)"| sort -V
)
results in:
kernel-3.10.0-1062.1.1.el7
kernel-3.10.0-1062.el7.x86_64
but we want to have line 1 at the end (because kernel-3.10.0-1062.1.1.el7 is newer than kernel-3.10.0-1062.el7.x86_64).
We could use the following command to get the correct result:
echo -e "kernel-3.10.0-1062.1.1.el7\n$(rpm -q kernel)" | awk '{gsub ("\\.el", ".0.0."); print}' | sort -V

The test condition also has to be changed to include the string replacement:
if [ ${newest} == $(echo ${required_pkg}|awk '{gsub ("\.el", ".0.0."); print}') ]; then

or just:
if [ ${newest} == ${required_pkg/.el/.0.0.} ]; then

Fixed in pull request #100 .


Proper Login Screen

The HTTP Basic Authentication added in #75 was just a temporary solution to get the first version out the door.
We should probably implement a proper login screen just like RTL and others have it.

if we implement a proper login screen, do we then need individual passwords for each wallet inside?

Nope, this would be independent from wallet passwords. UX wise it would stay the same as it is with the current Basic Authentication setup.

then why bother? I mean you can't access the wallets tu unlock them. the only option you have is to create a new wallet (until now)
why bother the normal home user to locking kind of twice (even 3 times, if you go through umbrel login » apps)

Because some parts of the API (e.g. listing wallets, creating wallets) will be accessible for anyone on the internet otherwise. While, as you correctly said, funds won't be in danger since wallets are locked on Joinmarket level, imagine people harassing you by creating thousands of wallets on your node or inferring things about the way you use Joinmarket by looking at what/how many wallets you have. That's something we don't want.
We already have this flow in place. It uses HTTP Basic Authentication to implement the vertical line from the diagram. Basic Auth looks a bit old and dusted and there's no way for us to style it which is why we want to move towards an in-app login screen similar to what Thunderhub, RTL, etc. have.

I added a simple login screen and also a logout possibility on the settings screen
https://www.figma.com/file/kfejZJFlwBywvLEnPEmJo1/?node-id=3994%3A73345


Error while cross compiling freerdp.

Hello everyone,
I'm trying to cross-compile freerdp 3.2.0 on my Ubuntu 20 for Windows 64-bit using mingw-w64. So far, I've successfully completed the required prerequisites necessary for freerdp. For configuring freerdp, I've used below command:

cmake -GNinja
-DCMAKE_TOOLCHAIN_FILE=/home/deewan/freerdp_compilation/mingw-w64-amd64.cmake
-B freerdp-build
-S freerdp
-DCMAKE_BUILD_TYPE=Release
-DCMAKE_SKIP_INSTALL_ALL_DEPENDENCY=ON
-DCMAKE_INSTALL_PREFIX=freerdp-install
-DWITH_SERVER=ON
-DWITH_SAMPLE=ON
-DWITH_PLATFORM_SERVER=OFF
-DUSE_UNWIND=OFF
-DWITH_SWSCALE=OFF
-DWITH_FFMPEG=OFF
-DWITH_WEBVIEW=OFF
-DBUILD_SHARED_LIBS=OFF
-DCHANNEL_URBDRC=OFF
-DCMAKE_MSVC_RUNTIME_LIBRARY=MultiThreaded

This runs fine. But when I try to build, I'm getting the below error:

In function ‘winpr_CreateThreadpoolWork’:
/home/deewan/freerdp_compilation/freerdp/winpr/libwinpr/pool/work.c:82:36: error: invalid application of ‘sizeof’ to incomplete type ‘TP_WORK’ {aka ‘struct _TP_WORK’}
82 |  work = (PTP_WORK)calloc(1, sizeof(TP_WORK));
|                                    ^~~~~~~
/home/deewan/freerdp_compilation/freerdp/winpr/libwinpr/pool/work.c:92:7: error: dereferencing pointer to incomplete type ‘struct _TP_WORK’
92 |   work->CallbackEnvironment = pcbe;
|       ^~
/home/deewan/freerdp_compilation/freerdp/winpr/libwinpr/pool/work.c: In function ‘winpr_SubmitThreadpoolWork’:
/home/deewan/freerdp_compilation/freerdp/winpr/libwinpr/pool/work.c:146:61: error: invalid application of ‘sizeof’ to incomplete type ‘TP_CALLBACK_INSTANCE’ {aka ‘struct _TP_CALLBACK_INSTANCE’}
146 |  callbackInstance = (PTP_CALLBACK_INSTANCE)calloc(1, sizeof(TP_CALLBACK_INSTANCE));
|                                                             ^~~~~~~~~~~~~~~~~~~~
/home/deewan/freerdp_compilation/freerdp/winpr/libwinpr/pool/work.c:150:19: error: dereferencing pointer to incomplete type ‘struct _TP_CALLBACK_INSTANCE’
150 |   callbackInstance->Work = pwk;
|                   ^~
/home/deewan/freerdp_compilation/freerdp/winpr/libwinpr/pool/work.c:151:31: error: dereferencing pointer to incomplete type ‘struct _TP_POOL’
151 |   CountdownEvent_AddCount(pool->WorkComplete, 1);
|                               ^~
/home/deewan/freerdp_compilation/freerdp/winpr/libwinpr/pool/work.c: At top level:
/home/deewan/freerdp_compilation/freerdp/winpr/libwinpr/pool/work.c:197:1: fatal error: opening dependency file winpr/libwinpr/CMakeFiles/winpr.dir/pool/work.c.obj.d: No such file or directory
197 | }
| ^
compilation terminated.

It seems like the compiler is not able to find the structure definition of TP_WORK (Instead of struct S_TP_WORK, it saying struct _TP_WORK). Is there any solution for this issue or am I doing something wrong here?
Thanks.

@Jino26 looks like there is something wrong with platform detection, checking.

@Jino26 ok, it is a bug in combination with the mingw version on 20.04.

@akallabeth Thanks for the reply. So what would you recommend, upgrade my Ubuntu to 22.04 or will there be a fix coming later for the above issue?

@Jino26 we´re happy about pull requests, but mingw builds on older platforms are not my area of expertise.
If you don´t have strict requirements on your build machine an update is recommended.
(personally I do prefer mingw on fedora as they did an excellent job at packaging nearly everything you require whereas you need to compile all dependencies for freerdp on debian based systems

@hardening do you have any idea what is breaking here?

I've already hit that too, I think it was when cross compiling to mingw 32 bits. Related to some definitions in threadpool.h or so, got look at my various repos...

Rebuilding the vcpkg port after an update from 3.0.0 to 3.3.0, I notice this build error. I guess this is caused by #9423.
winnt.h from mingw still defines
typedef struct _TP_POOL TP_POOL,*PTP_POOL;

etc., so depending on order of including headers, TP_POOL may not refer to struct S_TP_POOL as expected.
My local build succeeded after preprending
#ifdef __MINGW32__
#define _TP_CALLBACK_INSTANCE S_TP_CALLBACK_INSTANCE
#define _TP_POOL S_TP_POOL
#define _TP_WORK S_TP_WORK
#endif

to winpr/libwinpr/pool/pool.c and winpr/libwinpr/pool/work.c.


MSID: 98768 Version: 1 DOI: 10.1101/2<IP_ADDRESS>6831

MSID: 98768
Version: 1
Preprint DOI: https://doi.org/10.1101/2<IP_ADDRESS>6831
Step 1. Awaiting reviews
Editorial to post reviews via hypothesis
Useful links:

DocMap: https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v2/by-publisher/elife/get-by-manuscript-id?manuscript_id=98768
New model tracking: https://docs.google.com/spreadsheets/d/1_fHaoOy7hjyocptKtVJRijeNpUY4hBS7Ck_aVmx6ZJk/
Reviews on sciety: https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>6831

For trouble shooting (e.g. no Docmaps available):

DocMap issue addressing: https://miro.com/app/board/uXjVNCwK6EI=/
Explore DataHub DocMaps API: https://lookerstudio.google.com/reporting/4c2f0368-babb-4beb-b5b3-497e7e7b0f08/page/ejphD
Unmatched submissions and preprints: https://lookerstudio.google.com/u/0/reporting/9f86204f-3bf7-477c-9b18-5c5ef141bf69/page/p_gxi57ha93c
Unmatched manuscripts spreadsheet: https://docs.google.com/spreadsheets/d/15QcK8w-ssB7109RQEDtFpJPZ0J5HTGxoHa_2TtpMBbg/edit#gid=1336081641

Step 2. Preview reviewed preprint
Production QC content ahead of publication
Instructions:

QC preview: https://prod--epp.elifesciences.org/previews/98768v1
Update ticket with any problems (add blocked label)
When QC OK, add QC OK label to ticket and add publication date and time to https://docs.google.com/spreadsheets/d/1amAlKvdLcaDp5W8Z8g77NmkwbMF5n_u89ArSqPMO8jg
Move card to next column
(At end of the day post link in #enhanced-preprint and ask for PDF to be generated)

Useful links:

Preprint DOI :  https://doi.org/10.1101/2<IP_ADDRESS>6831
Confirm reviews returned by EPP: https://prod--epp.elifesciences.org/api/reviewed-preprints/98768/v1/reviews
To update the MECA path in the docmap: https://docs.google.com/spreadsheets/d/1mctCQuNFBjSn97Lihy7_vBO6z7-N-oqyLv4clyi6zHg

Step 3: Awaiting search reindex
This step adds the reviewed preprint to the homepage: https://elifesciences.org
The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.
Useful links:

Jenkins pipeline to reindex search can be triggered sooner or monitored here: https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 4: Published! PDF requested
Waiting for PDF to be generated
Useful links:

PDF tracking: https://docs.google.com/spreadsheets/d/106_XeDjmuBae7gexOTNzg60lapeqjl2aRn9DzupGyS8/

Step 5: Introduce PDF to data folder and git repo
Upload PDF to relevent folder in git repo https://github.com/elifesciences/enhanced-preprints-data/
Step 6: Done!

[ ] Kettle is on!

I think the wrong version of the preprint was loaded initially, updating meca bucket with correct version; hopefully that'll fix the missing reviews.

Hi @JGilbert-eLife, noticed this was published on 25 July but has pub date of 24 July, will this cause any issues?

Fiddlesticks -- I mustn't have changed the date on the sheet properly (thought I did but clearly not).
@fred-atherden I assume we can just update the sheet and ask Graham to resubmit to crossref?

I've updated the sheet, so once the docmap catches up, Graham will be able to redeposit it. I'll ping him on slack.


Removed future imports

Following #3457, all imports from __future__ have been removed, as have all tests for these imports and six imports.

This is now being addressed by #3505, which is pulling into a branch of master for the sake of safely merging with other similar pull requests.


Implement full builder tree configuration

After node builders, this PR adds a few features allowing the creation of a full configuration tree through chained methods. Also brings some constraint refactors.
Builders

Node builders have methods to begin value configuration (allows chaining).
Every builder has a finish method that builds the value and returns the parent builder (maximum chaining).

finish methods can take a Consumer argument

Aggregate builders allow null component types. For those cases where you cannot give the component type at compile time (eg. bad generics).
Value builders now check that the default value conforms to specified constraints when building.

Snakey snake result:
    /* you can either store intermediate builders and use 'build()',
       or use some kind of box to retrieve the nodes and values. */
    AtomicReference<ConfigValue<Integer>> a = new AtomicReference<>();
    ConfigNode config = new ConfigNodeBuilder()
            .beginValue("version", "1.0.0")
                .withFinality()
                .beginConstraints() // checks the default value
                    .regex("\\d+\\.\\d+\\.\\d+")
                .finishConstraints()
            .finishValue()
            .fork("child")
                .beginValue("A", 10)
                    .beginConstraints()
                        .composite(CompositeType.OR)
                            .atLeast(3)
                            .atMost(0)
                        .finishComposite()
                    .finishConstraints()
                .finishValue(a::set)
            .finishNode()
            .build();`
    this.a = a.get(); // do whatever you want with the setting

A possibility I explored was to give to every builder its source as a generic parameter. This allows slightly better compile time safety, at the cost of very painful type declarations when you put an intermediate builder in a variable. Probably not worth it until we get var for everyone.
Constraints

Constraints can be instantiated with a null type. This results in deferred type checking, allowing more flexibility while building the configuration tree (main use being bad generics).
Component constraints now are ValuedConstraints. Probably useful when we start serializing those.
Finality is now implemented as a Constraint.
Property has a new accepts method that's like a setValue dry run.

Miscellaneous

Nodes just store their items as a Map, instead of a Map and a Set.
More tests!

note: this PR is built on #24

Ignore my review I put for half a second, I didn't read properly lol


Better link between API and Extension

If you hit record in the extension and then try to start recording from js it says "busy", which is good.
But if you are recording from the extension and ask for the state it says inactive.  So it is not possible to tell if a user started recording with the button.
Similarly it isn't possible to add a payload to a recording started via the extension.

That's intentional behaviour, the API only allows to control recordings that it started.
I'll contact you offline to learn more about your use-case.


Window resizing seems clean the document

How to reproduce:

Draw some random things
Resize window by double clicking title bar or dragging corner.
The things are disappeared..

Thanks for reporting.
I am not exactly sure how tackle it for now, so any ideas/help are welcome.
There is a function that called on resize: https://github.com/StaffEngineer/velo/blob/main/src/ui_plugin/systems/resize_window.rs
Position of nodes updated in https://github.com/StaffEngineer/velo/blob/main/src/ui_plugin/systems/update_rectangle_position.rs by setting left and bottom values.

@piaoger interesting.
I think this code was added for fixing arrow positions on wasm. If you comment code and run velo in browser and resize window I think position of arrows gonna be broken? If commenting code fixes the issue maybe we have to revisit approach how arrows on wasm was fixed and find another way.

@piaoger this should be fixed by https://github.com/StaffEngineer/velo/pull/153, please re-open the issue if not.


fix(ci): remove unused dev deps from pasaport and std pkgs

Description

Removed unused dev dependency vite-tsconfig-paths from packages/std and apps/pasaport
See comments on this PR for more detail https://github.com/kamp-us/monorepo/pull/702

Checklist

[x] discord username: muroyoe#9144
[x] Closes #663 (see also #702)
[x] PR must be created for an issue from issues under "In progress" column from our project board.
[x] A descriptive and understandable title: The PR title should clearly describe the nature and purpose of the changes. The PR title should be the first thing displayed when the PR is opened. And it should follow the semantic commit rules, and should include the app/package/service name in the title. For example, a title like "docs(@kampus-apps/pano): Add README.md" can be used.
[x] Related file selection: Only relevant files should be touched and no other files should be affected.
[x] I ran npx turbo run at the root of the repository, and build was successful.
[x] I installed the npm packages using npm install --save-exact <package> so my package is pinned to a specific npm version. Leave empty if no package was installed. Leave empty if no package was installed with this PR.

How were these changes tested?
No need for tests.

since we removed the "pasaport" folder for now, this is not needed.
Thanks for the PR and would be happy to see your future contributions :) 🚀 🎉


[DEPRECATED] library Snackbar should be removed

Hello,
Why a deprecated library like Snackbar is still in the list?
Shouldn't it be removed?

Hello,
I completely agree, it's written in their repository - deprecated.

Shall I make a Pull request ?

I think yes

#218 Done!

#219 Done


Connection String example code doesn't build

When selecting the Connection String option to create a new .net application, the documentation is telling you to only install this Nuget package:
Install-Package Azure.Messaging.EventHubs
however the C# application for Connection String starts with:
using Azure.Identity;
using Azure.Messaging.EventHubs;
using Azure.Messaging.EventHubs.Producer;
using System.Text;

The Azure.Identity package of course is missing when following instructions and a build error is produced.
When removing the using Azure.Identity requirement, the application Builds, but throws an error anyways:

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: c4c6d63e-95f5-de2c-972b-4da5fe71a6a6
Version Independent ID: 338bbd1b-d86d-ff48-0b8a-1d48cb9547f2
Content: Send or receive events from Azure Event Hubs using .NET (latest) - Azure Event Hubs
Content Source: articles/event-hubs/event-hubs-dotnet-standard-getstarted-send.md
Service: event-hubs
GitHub Login: @spelluru
Microsoft Alias: spelluru

@powershellpr0mpt
I've delegated this to content author @spelluru, who will review it and offer his insightful opinions.

In the connection string scenario, no need to import the Azure.Identity package. Not sure about the error message. I haven't seen it before.

@powershellpr0mpt - Thanks for the feedback. Appreciate it. I have removed that unnecessary using Azure.Identity for the connection string scenario.
The error you are receiving is not related to the quickstart. Please restart VS and/or install SQL Server Data Tools and retry again.
#please-close


Wait for frames and its state in frames test

Change FramesPageUnitTest to wait for the frames (instead of sleep)
before asserting its state, also, wait for the expected state as the
state might not have been changed yet.
Part of #344 - Make functional tests more reliable

Fixed the code format.


NIFI-12017 add ability to choose to output to single line for base32 base64 contents

Summary
NIFI-12017
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x ] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[x] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[x] Build completed using mvn clean install -P contrib-check
[x] JDK 21

Licensing

[x] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[x] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[x] Documentation formatting appears as expected in rendered files

Thanks again for your work on these new configuration properties @knguyen1. As it has been several months since the last update, I am closing this pull request for now. When you are ready to follow up, feel free to reopen the pull request and address the feedback, then we can move forward.

I opened the PR at: https://github.com/apache/nifi/pull/8417 @exceptionfactory
I didn't see a button to re-open this one.


Java 21 compatibility

Java 21 is the next LTS release, expected September 2023. We'd like to officially support this.
It isn't out yet, but we can start with Java 20.

I'll share my notes since I've been slowly working my way through this!
Equalsverifier based tests fail with:
Java 20 (64) is not supported by the current version of Byte Buddy which officially supports Java 18 (62) - update Byte Buddy or set net.bytebuddy.experimental as a VM property

which is resolved by manually adding the newest version of byte-buddy to the pom (I also updated both equalsverifier and mockito to latest versions which also have transient dependencies on byte-buddy).
Before updating easymock we also see some failures of this form
[ERROR] org.apache.druid.segment.writeout.FileWriteOutBytesTest.writeBufferSmallerThanCapacityShouldIncrementSizeCorrectly  Time elapsed: 0 s  <<< ERROR!
java.lang.IllegalArgumentException: Unsupported class file major version 64

which go away after updating easymock, however I am still seeing a failure on the latest version of easymock:
[ERROR] org.apache.druid.query.aggregation.constant.LongConstantBufferAggregatorTest.testAggregate  Time elapsed: 0.018 s  <<< ERROR!
java.lang.IllegalArgumentException: Could not create type
	at org.easymock.bytebuddy.TypeCache.findOrInsert(TypeCache.java:170)
	at org.easymock.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:399)
	at org.easymock.internal.ClassProxyFactory.createProxy(ClassProxyFactory.java:136)
	at org.easymock.internal.MocksControl.createMock(MocksControl.java:108)
	at org.easymock.internal.MocksControl.createMock(MocksControl.java:81)
	at org.easymock.IMocksControl.mock(IMocksControl.java:44)
	at org.easymock.EasyMock.mock(EasyMock.java:70)
...
Caused by: java.lang.IllegalStateException: java.lang.IncompatibleClassChangeError: class org.easymock.mocks.ByteBuffer$$$EasyMock$1 cannot inherit from sealed class java.nio.ByteBuffer

which is as far as i've got so far, I think maybe easymock hasn't been updated to be cool with java 20, as https://github.com/easymock/easymock/releases has an older version of bytebuddy than i recently added.

For LongConstantBufferAggregatorTest specifically, I think we can replace buffer = EasyMock.mock(ByteBuffer.class) with buffer = null. It's just verifying that the LongConstantBufferAggregator is not calling any methods on buffer. Can't call methods on null 😉

Want to make a branch on apache/druid that various of us can commit to? Once things are buildable someone can make a PR from it. At that point I would suggest running most tests on 8 (oldest supported), 17 (latest LTS supported), and 20.

Yeah, will try to do later today :+1:
fwiw druid itself seems to run fine in java 20, i've been using it for all of my dev work running a local cluster for a couple of months now (though disclaimer i mainly use indexers instead of middle managers since easier to debug stuff), so i think it is just test problems.
I let some more tests run while i was eating lunch and the next failure is in druid-server:
[ERROR] org.apache.druid.curator.CuratorModuleTest.exitsJvmWhenMaxRetriesExceeded  Time elapsed: 0 s  <<< ERROR!
java.lang.UnsupportedOperationException: The Security Manager is deprecated and will be removed in a future release
	at java.base/java.lang.System.setSecurityManager(System.java:429)
	at org.junit.contrib.java.lang.system.ProvideSecurityManager.before(ProvideSecurityManager.java:39)

and later in druid-sql are some failures like:
[ERROR] org.apache.druid.sql.calcite.rule.DruidLogicalValuesRuleTest$GetValueFromLiteralSimpleTypesTest.testGetValueFromLiteral[BIGINT, class java.lang.Long]  Time elapsed: 0 s  <<< FAILURE!
java.lang.AssertionError:
Unable to mock the literal for test.
Exception: java.lang.NoSuchFieldException: value
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.druid.sql.calcite.rule.DruidLogicalValuesRuleTest$GetValueFromLiteralSimpleTypesTest.makeLiteral(DruidLogicalValuesRuleTest.java:109)
	at org.apache.druid.sql.calcite.rule.DruidLogicalValuesRuleTest$GetValueFromLiteralSimpleTypesTest.testGetValueFromLiteral(DruidLogicalValuesRuleTest.java:91)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)

For DruidLogicalValuesRuleTest, we can remove the mocks from makeLiteral and make real literals instead:
    private static RexLiteral makeLiteral(Comparable<?> val, SqlTypeName typeName)
    {
      return (RexLiteral) new RexBuilder(DruidTypeSystem.TYPE_FACTORY).makeLiteral(
          typeName == SqlTypeName.DECIMAL && val != null ? new BigDecimal(String.valueOf(val)) : val,
          DruidTypeSystem.TYPE_FACTORY.createSqlType(typeName),
          false
      );
    }

The CuratorModuleTest case exitsJvmWhenMaxRetriesExceeded relies on NoExitSecurityManager to trap System.exit. Security managers are being removed gradually from Java 18+ (https://openjdk.org/jeps/411). The test will need to be rewritten or deleted.
IMO a good option would be to adjust CuratorModuleTest to subclass CuratorModule and override shutdown(lifecycle). The overridden method should set an AtomicBoolean in the subclass rather than calling System.exit(1). Then, the test case should check the AtomicBoolean rather than using ExpectedSystemExit.

I was able to make unit tests pass in #15014. The biggest issue I see is that we need mockito 5.x to run tests with Java 21, however that version no longer supports Java 8. We might be able to do some workaround where we run tests with different mockito versions and restrict ourselves to the shared APIs, but that is also somewhat tricky since mockito-inline is now merged into mockito-core.

The biggest issue I see is that we need mockito 5.x to run tests with Java 21, however that version no longer supports Java 8

Now that we've dropped support for Hadoop 2, maybe we can consider doing the same for Java 8? Afaik that was the primary reason we needed to keep supporting it... though we'd need to decide on a new minimum version.

IMO we should only target LTS Java versions, so 11 should be the new minimum version

Current versions of Kafka and Spark still support Java 8; I'd rather not be on the vanguard of dropping support, unless we get something big in return.
Kafka has deprecated it for removal in Kafka 4.0. Maybe we can do something similar?

Wonder how this is going, is Druid ready to allow Java 21 without DRUID_SKIP_JAVA_CHECK? 😄