BUG: handle inconsistencies in factorial functions and their extensions

In the context of #15351 and reviewing #15299, I noticed a bunch of inconsistencies that should be fixed IMO.
TL;DR
Courtesy of @tirthasheshpatel:

factorial, factorial2, and factorialk behave inconsistently for several different combinations of inputs. (for example, factorial2 returning 0-dim arrays for scalars and, factorial2 and factorialk aren't vectorized). See the first table for more details. This issue can be tacked by adding a new parameter like legacy which, by default, preserves the old behaviour but when false, handles the current inconsistent behavior correctly.
Support for float and complex inputs can easily be added using this extension. This enhancement adds a new boolean parameter extend='zero'|'complex' (and perhaps 'recurrence' down the line) that specifies whether to use the extension formula or the default formula. See the second table for more details.

Design Space
There are several dimensions that come into play:

scalar vs. array inputs
for integers: approximation or exact computation (exact=)
extensions to negative integers
extensions to floats
extensions to complex numbers
extensions to poles (possible e.g. for factorial2)
for the array-case, failing soft (e.g. inserting NaNs) or hard (raising error)

Current Behaviour (as of 1.8.0)

Inputs
factorial(exact=False)
factorial(exact=True)
factorial2(exact=False)
factorial2(exact=True)
factorialk(exact=False)
factorialk(exact=True)

Scalar positive integer
float
int
!! 0-dimfloat array !!
int
NotImpl Error
int

Scalar negative integer
0.0
0
!! array(0.) !!
0
NotImpl Error
0

Scalar positive float
float
ValueError (but text misleading)
TypeError
TypeError
NotImpl Error
TypeError

Scalar negative float
0.0
0
TypeError
TypeError
NotImpl Error
TypeError

Scalar complex
TypeError
TypeError
TypeError
TypeError
NotImpl Error
TypeError

Scalar NaN
NaN
NaN
TypeError
TypeError
NotImpl Error
TypeError

Positive integer in array
float
int
float
!! TypeError !!
NotImpl Error
TypeError

Negative integer in array
0.0
0
0.0
!! TypeError !!
NotImpl Error
TypeError

Positive float in array
float
ValueError (but text misleading)
TypeError
TypeError
NotImpl Error
TypeError

Negative float in array
0.0
0
TypeError
TypeError
NotImpl Error
TypeError

Complex in array
TypeError
TypeError
TypeError
TypeError
NotImpl Error
TypeError

NaN in array
NaN
NaN
TypeError
TypeError
NotImpl Error
TypeError

Empty array
array([])
IndexError
TypeError
TypeError
NotImpl Error
TypeError

Note that:

factorial2(<scalar>, exact=False) returns a zero-dimensional array instead of a scalar.
All the TypeErrors are IMO implementation oversights (and in case of "!!", against documentation) - i.e. we should provide better error messages.
There is a full generalization of factorialk which is basically just a fraction of two gamma functions, which would work for all complex n & k (up to poles), as well as arrays.

Extension Issues
In addition to this hodgepodge of behaviour, some people would like to have the float/complex extensions of these functions (e..g. https://github.com/scipy/scipy/pull/15349). I tend to agree, but that raises another problem:
Complex extensions are incompatible with current inputs ≤0
First off, everything that's 0 currently on the negative axis gets a different value. But also note the "≤"; namely, 0!! is 1 in the regular definition, but ~0.79=sqrt(2/pi) in the extension.
Extensions based on recurrence are incompatible with current status as well as complex extension
This came up for example in #15299. It's possible (for example) to invert the recurrence relation n!! = n * (n-2)!!, and so extend the double-factorial to negative integers. The resulting value would be neither 0 (currently) nor the pole in the complex extension.
Design Considerations
If we (or at least I) were designing from scratch:

Changing from the exact=False to exact=True should never fundamentally change the result (e.g. going from 0 to non-zero, or casting inputs to integers as seen in #15351).

As such, the exact keyword only really matters to positive scalar integers; in all other cases, it could be retired, resp. raise, or simply be turned into a no-op (cf. #15359)
Open question whether it's worth to properly implement exact=True for array-case (currently, only factorial does).

Complex extensions  (together with non-positive floats) should be behind a keyword, roughly because:

0-for-negative-values is a valid approach, and has been in scipy ~forever.
Since the values being produced are incompatible (for certain inputs), the complex extension should be explicitly opt-in.

Using recurrence relations for extending into poles is an open question; not sure if worth it, but if so, probably with another ~keyword~ value for the extend keyword.
Use the most general approximation, except when alternatives with better accuracy / performance exist.
Probably need to think about dtype casting rules for empty arrays (keep dtype? always float? ...)

Possible Future Behaviour (note extend=, not exact=)

Inputs
factorial(extend='zero')
factorial(extend='complex')
factorial2(extend='zero')
factorial2(extend='complex')
factorialk(extend='zero')
factorialk(extend='complex')

Scalar positive integer w/ exact=True
n!
n!
n!!
n!!
n!(k)
n!(k)

Scalar positive integer
Γ(n+1)
Γ(n+1)
(1)
(1)
(2)
(2)

Scalar negative integer / poles
0.0
NaN
0.0
NaN
0.0
NaN

Scalar positive float
Γ(n+1)
Γ(n+1)
(1)
(1)
(2)
(2)

Scalar negative float
0.0
Γ(n+1)
0.0
(1)
0.0
(2)

Scalar complex
ValueError
Γ(n+1)
ValueError
(1)
ValueError
(2)

Scalar NaN
NaN
NaN
NaN
NaN
NaN
NaN

Positive integer in array
Γ(n+1)
Γ(n+1)
(1)
(1)
(2)
(2)

Negative integer in array
0.0
NaN
0.0
NaN
0.0
NaN

Positive float in array
Γ(n+1)
Γ(n+1)
(1)
(1)
(2)
(2)

Negative float in array
0.0
Γ(n+1)
0.0
(1)
0.0
(2)

Complex in array
ValueError
Γ(n+1)
ValueError
(1)
ValueError
(2)

NaN in array
NaN
NaN
NaN
NaN
NaN
NaN

Empty array
array([])
array([])
array([])
array([])
array([])
array([])

The formulas in question are:

(1): z!! = 2^((z-1)/2) Γ(z/2 + 1) / Γ(1/2 + 1)
(2): z!(α) =  α^((z-1)/α) Γ(z/α + 1) / Γ(1/α + 1)

I'd very much welcome feedback on this - there are some trade-offs to be made (e.g. whether negative floats should also raises without extend='complex' or whether the errors in the array-case should just map to NaN; also there are different approximations for n!! in use currently), but I think the above would be reasonable. In particular, the scalar and array case behave the same way.
Implementation Plan
I would propose to introduce another keyword (sorry...) legacy, defaulting to True, but implement legacy=False with all the corrected behaviour. Where there's any planned change in behaviour, we raises a deprecation warning recommending to switch to legacy=False. After 2 releases, we switch the default from legacy=True to legacy=False, after two more releases we raise on legacy=True, and after two more we remove the keyword.
Looking more closely at relevant changes in (currently non-raising) behaviour, it really only affects the zero-dimensional arrays for factorial2(scalar_val, exact=False). In particular, if we let exact=True be a no-op for non-integers (the less "noisy" options from the point of UX), we would keep essentially all current non-raising behaviour intact.
CC @steppi @tirthasheshpatel @j-bowhay @tupui
Edit History:

v4 (2022-03-10): Make exact=True a no-op for non-integers (less noisy), following #15359. Extend deprecation period of legacy-keyword along the same lines as proposed in https://github.com/scipy/scipy/issues/15722.
v3 (2022-02-16; up to rev 20): Added (current & proposed) behaviour for NaN and empty arrays; good catch @tirthasheshpatel!
v2 (2022-02-16; up to rev 15): Added TL;DR, clarification on impact of legacy, changed extend from boolean argument to take an enum.
v1 (2022-02-16; up to rev 10): Original

Thank you this makes it a lot easier to follow because, as you say, there are a lot of inconsistencies. I will try to have a look over this at the weekend.

Thanks for the incredibly detailed report @h-vetinari :exploding_head:! I suggest adding a TLDR; something like this if I understand the issue correctly:

factorial, factorial2, and factorialk behave inconsistently for several different combinations of inputs. (for example, factorial2 returning 0-dim arrays for scalars). See the first table for more details. This issue can be tacked by adding a new parameter like legacy which, by default, preserves the old behaviour but when false, handles the current inconsistent behavior correctly.
Support for float and complex inputs can easily be added using this extension. This enhancement adds a new boolean parameter extend that specifies whether to use the extension formula or the default formula. See the second table for more details.

Can this issue be split into two: one for addressing inconsistencies and one for adding the extension?

factorial2(<scalar>, exact=False) returns a zero-dimensional array instead of a scalar.

This is somewhat common in stats too. Although it's an inconsistency, such outputs mostly behave like scalars (and also get coerced into one if operated with scalars). I think there are more important things we should tackle first: for example, vectorizing factorial2 and factorialk (these functions throwing for array inputs is very unintuitive).

As such, the exact keyword only really matters to positive scalar integers; in all other cases, it could be retired, resp. raise.

I agree that raising would be nice but that would again be a backward-incompatible change. Rather, we can add a keyword, similar to nan_policy, that asks the user whether to coerce the inputs to integers, raise, or fill NaNs for invalid cases in array inputs (e.g. exact_policy='raise' | 'coerce' | 'fill-nan'). For factorial, the default could be 'coerce'. For factorial2 and factorialk, raise.

Complex extensions  (together with non-positive floats) should be behind a keyword

This sounds good.

I would propose to introduce another keyword (sorry...) legacy, defaulting to True, but implement legacy=False with all the corrected behaviour. Where there's any planned change in behaviour, we raises a deprecation warning recommending to switch to legacy=False. After 2 releases, we switch the default from legacy=True to legacy=False, and after two more releases we remove the keyword.

I am not against this but maybe we can also consider an alternative approach like adding exact_policy?

Thanks for the incredibly detailed report @h-vetinari 🤯!

🙃

I suggest adding a TLDR; something like this if I understand the issue correctly:

Done, thanks!

Can this issue be split into two: one for addressing inconsistencies and one for adding the extension?

Fine by me, and effectively adopted with your TL;DR.

factorial2(<scalar>, exact=False) returns a zero-dimensional array instead of a scalar.

This is somewhat common in stats too. Although it's an inconsistency, such outputs mostly behave like scalars (and also get coerced into one if operated with scalars). I think there are more important things we should tackle first: for example, vectorizing factorial2 and factorialk (these functions throwing for array inputs is very unintuitive).

Prioritization aside, I think we should clean this up. 0-dim arrays used to be a bit of a wild west thing, but it's getting cleaned up in lots of APIs (not just numpy/scipy, also other downstreams like cvxpy).

As such, the exact keyword only really matters to positive scalar integers; in all other cases, it could be retired, resp. raise.

I agree that raising would be nice but that would again be a backward-incompatible change. Rather, we can add a keyword, similar to nan_policy, that asks the user whether to coerce the inputs to integers, raise, or fill NaNs for invalid cases in array inputs (e.g. exact_policy='raise' | 'coerce' | 'fill-nan'). For factorial, the default could be 'coerce'. For factorial2 and factorialk, raise.

I'm not a fan, to be honest. It has the same high price (another new keyword) for very little gain IMO. In particular, I think coerce is basically never the right thing, and if so, then people should get the warning/error and explicitly cast.

Complex extensions  (together with non-positive floats) should be behind a keyword

This sounds good.

We'll have to do some bikeshedding on this. I proposed extend=True|False more or less as a stand-in, because extend_complex seemed too long), but if we want to leave the recurrence thing on the table, we could actually do extend='zero'|'complex'|'recurrence', which to me looks even better than boolean switches.

I would propose to introduce another keyword (sorry...) legacy, defaulting to True, but implement legacy=False with all the corrected behaviour. Where there's any planned change in behaviour, we raises a deprecation warning recommending to switch to legacy=False. After 2 releases, we switch the default from legacy=True to legacy=False, and after two more releases we remove the keyword.

I am not against this but maybe we can also consider an alternative approach like adding exact_policy?

I understand that it's not attractive, but I had a closer look and added this to the OP:

Looking more closely at relevant changes in (currently non-raising) behaviour, it really only affects the zero-dimensional arrays for factorial2(scalar_val, exact=False). In particular, we could choose to let exact=True only raise if extend=True as well, keeping essentially all current non-raising behaviour intact.

Some more inconsistencies: factorial only works with empty arrays when exact=False. For exact=True, it fails:
from scipy.special import factorial, factorial2, factorialk

factorial([])  # array([], dtype=float64)
factorial([], exact=True)  # IndexError: index -1 is out of bounds for axis 0 with size 0

factorial2 fails for NaN inputs:
factorial2(nan, exact=False)
# TypeError: ufunc 'bitwise_and' not supported for the input types, and the inputs could not be safely coerced to any supported
# types according to the casting rule ''safe''

Some more inconsistencies

Good points! Added to the OP. Do we also want to consider None?

Another possibility mentioned in the review for #15299 is making factorial and factorial2 thin wrappers around factorialk. I'm not sure if it is a good idea but probably worth discussing.

@j-bowhay: [...] making factorial and factorial2 thin wrappers around factorialk.

That's partly what I was referring to with:

Use the most general approximation, except when alternatives with better accuracy / performance exist.

But to do that, we need some benchmarks against the current most important code paths (haven't verified current status of factorial in benchmarks/), and ideally some as-tight-as-possible accuracy tests.

But to do that, we need some benchmarks against the current most important code paths (haven't verified current status of factorial in benchmarks/), and ideally some as-tight-as-possible accuracy tests.

Agreed, this was my hesitation in making them thin wrappers is a potential loss in performance and accuracy as some cases have more optimal ways of performing the calculation that don't generalise.
In benchmarks/benchmarks/special.py there are currently no benchmarks for any of the factorial stuff. Maybe a good first step would be to add some benchmarks?

Maybe a good first step would be to add some benchmarks?

Absolutely! :)
IMO we should aim basically for all the cases in the first table that don't raise (except NaN/empty array).

Since it would be good if we can do this at the same time with the changes in comb, I'm tentatively milestoning this for 1.9.

While looking at the code for this, I realized that the array-path for factorial(exact=True) depends on the fact that np.unique returns sorted unique values, and so a recent behaviour change in numpy has at least disabled the type-setting logic. It doesn't seem to break too loudly though, so I'm not raising a separate issue, but thought I'd note it here.

Also, I found that - contrary to their documentation - factorial2 and factorialk return wrong values for small negative integers. I'm not planning to go through a multi-release deprecation cycle for this, and will just fix it outright to return zero, as intended/documented.
>>> factorial2(0)
array(1.)
>>> factorial2(-1)
array(1.)
>>> factorial2(-2)
array(0.)

Note factorial2(-1) should be 0.
>>> factorialk(0, k=3)
1
>>> factorialk(-1, k=3)
1
>>> factorialk(-2, k=3)
1
>>> factorialk(-3, k=3)
0

Same here, and it's actually dependent on k (i.e. it's falsely non-zero until -(k-1)).

While working on this, I just stumbled over test_factorial_0d_return_type, and thus https://github.com/scipy/scipy/issues/7400 & https://github.com/scipy/scipy/pull/10954. This shows that the 0d array-output for factorial2 was already undesired a long time ago, and was merely overlooked in #10954.
The reason this is relevant is that if we treat it as a bugfix (similar to #10954), then we'd have no deprecations to do for the other points in this issue.
However, I wanted to tag the participants of those old discussion, because I think #10954 went too far in fixing #7400 - I don't think the factorial-functions should be changing array-inputs (even zero-dimensional ones) to scalars. IMO it should be scalar->scalar, array->array (and the latter with identical dimensions).
CC @WarrenWeckesser @person142 @rgommers @larsoner

Also, I found that - contrary to their documentation - factorial2 and factorialk return wrong values for small negative integers. I'm not planning to go through a multi-release deprecation cycle for this, and will just fix it outright to return zero, as intended/documented.

I just realised that this statement was true for 1.8.0 (which were the sources I was looking at), but this was changed for 1.9 in https://github.com/scipy/scipy/issues/15589, https://github.com/scipy/scipy/pull/15590, https://github.com/scipy/scipy/issues/15592.
I strongly believe that the 1.8. docstrings were correct behaviour, and we should match the code, rather than document broken behaviour. CC @AtsushiSakai @tupui @j-bowhay

Next problem - the "universal" approximations for n!! is very bad for even numbers (the odd ones work well). For example, the formula (1) from the OP, calculated to high precision with mpmath gives: 10!! = 3063.8767134830029.
However, the true value is very different:
>>> factorial2(10)
3840.

and the current approximation actually does an extremely good job (down to a single eps even for n=100)
>>> factorial2(10) / float(factorial2(10, exact=True)) -1
0.0
>>> factorial2(100) / float(factorial2(100, exact=True)) - 1
2.220446049250313e-16
>>> float(factorial2(100, exact=True))
3.4243224702511973e+79

However, the formula picked up by @j-bowhay in #15349 seems to do a much better job also for even integers.

I strongly believe that the 1.8. docstrings were correct behaviour, and we should match the code, rather than document broken behaviour.

Regarding the small-negative-integers for factorialk, I'll note that a common definition does include n!!=1 for small negative n, but I claim that's purely for the mathematical convenience of simplifying the recursion (which is - mixing metaphors - a kind of mathematical implementation detail, but not required for an unambiguous and consistent definition**).
This makes outright changing the behaviour slightly harder, but I still think that extend='zero' would be by far the most sane default for negative numbers. However, if people believe it necessary, we can go through the full legacy= deprecation dance for that.

** alternative definition
The definition could be rewritten equivalently for all non-negative values:
n!(α) = n * (n - α)!(α) if n ≥ α
      = 1 if 0 ≤ n < α
      = 0 otherwise

I have a WIP PR that addresses some of the issues mentioned in the OP and recent comments. It's not finished yet, and I expect that I'll have to cut it into smaller pieces, but if someone's interested, please take a look: https://github.com/scipy/scipy/pull/15841

Almost in every software I used, if we have a multiple function doing similar but different versions of a functionality 5 years down the line we struggle deprecating/removing them.
I would have liked this function to be in NumPy namespace since it is a very common math operation or from standard math library but it is now living in special functions territory because reasons.
So at least can we combine them into a single signature since they are special cases of k=1, 2 and get rid of the others eventually in the future?

So at least can we combine them into a single signature since they are special cases of k=1, 2 and get rid of the others eventually in the future?

I sympathise with the sentiment, though I'm not sure I agree... Aside from IMO quite well-established names (not just in scipy), the functions have some pretty subtantial differences, despite how closely related they are - e.g. factorialk doesn't have a well-known meromorphic continuation (meaning we cannot interpolate to floats; contrary to what I claimed in the OP), factorial does not permit something like extend='recurrence' but factorial2 does, etc. etc.
I mean, yes, of course it's possible, but effectively it'd be three different implementations in a trenchcoat pretending to be one function.
I cannot comment on whether this belongs to numpy or scipy, but I'd be open to shepherd the PR wherever necessary.

I mean, yes, of course it's possible, but effectively it'd be three different implementations in a trenchcoat pretending to be one function.

Though if I can pull of the array-path of factorial2 / factorialk for exact=True as I imagine it (based on a generalized _exact_factorial_array), then I think it might be possible after all to share most of the surrounding code.

Though if I can pull off the array-path of factorial2 / factorialk for exact=True as I imagine it (based on a generalized _exact_factorial_array), then I think it might be possible after all to share most of the surrounding code.

I agree it think the not exact path is too different for each to be one function.

What I mean is that from the user's perspective nobody is interested in how the code traverses the internal paths. It just needs a proper factorial function which is already buried under a special module whereas it is always included in the primary namespace in other (commercial/OSS) tools and unfortunately not in NumPy.
pinv2, lsim2 and so on shared also very much the same thinking back then but probably we should have been a bit more careful (20/20 hindsight). I am wondering if we are doing the same with these factorial functions

pinv2, lsim2 and so on shared also very much the same thinking back then but probably we should have been a bit more careful (20/20 hindsight)

I'm not sure the case is fully comparable. pinv/pinv2 are doing the same thing conceptually but were using different implementations - as you wrote in #13831:

Historically, pinv2 and pinv were using different solution methods however for a very long time already both use the same SVD based solvers. Hence duplicate code for no apparent reason in the namespace.

Here however, factorial2 does not mean "we're adding a 2 to not break the old API/behaviour", but it is a very different (though obviously related) function that has its use-cases independently from factorial.
That being said, I don't mind in principle to fold everything into factorial and deprecate factorial2 / factorialk, though I don't think it's a huge benefit API- and especially maintenance-wise (the code will need to branch a lot to deal with different behaviours, extensibility, etc. of these functions)

a proper factorial function which is already buried under a special module whereas it is always included in the primary namespace in other (commercial/OSS) tools and unfortunately not in NumPy.

Just stumbled over this in the git-blame - seems the factorial functions used to be in an even more obscure place until https://github.com/scipy/scipy/pull/3151.

I have an almost fully passing PR (some issues with GCC6) for all the fixes & behaviour changes in #15841. I'd appreciate if people who are interested could take a look. 🙃
It does not yet include the complex extension, but is very thoroughly tested and should get rid of basically all the spurious TypeErrors etc. from the OP.
It also does not yet attempt to unify everything under factorial (cf. the most recent comments here with @ilayn).

I think I should concede my comments this is indeed a quite complicated issue and I don't want to muddy it with philosophical issue. We can come back to it later and since there is no user noise around this apparently people are not too unhappy about it.

@h-vetinari what state is your pr in? It would be nice to try get this in 1.10.0 so that the comb and factorial tidy ups end at roughly the same time.

There was an open question (mainly for @ilayn) if we're fine with using the canonical extension for the double-factorial, which is off by a factor for even integers. I have some change requests pending but wanted to have some clarity on that before I start changing things around.
Agreed with 1.10, though that's >6 months away, so hopefully enough time.

My bad. Sorry for ignoring, reading it a bit more, it seems to me that I shouldn't be the one judging this as I am not 100% on every detail for the use case. Let me ping @WarrenWeckesser and @person142 just in case they want to have a look.

I consider this issue solved by #15841. For the open points regarding the extensions, I've opened a follow-up issue: https://github.com/scipy/scipy/issues/18409


Integration with Vanilla+ mods.

https://www.curseforge.com/minecraft/mc-mods/simple-emerald-tools-fabric

I think I'm going to change this Issue into a sort of mega-issue that will have a list of all the Vanilla+ material mods I want to guarantee integration with.
https://github.com/CrimsonDawn45/Basic-Shields-Fabric/issues/21
https://github.com/CrimsonDawn45/Basic-Shields-Fabric/issues/24
https://github.com/CrimsonDawn45/Basic-Shields-Fabric/issues/20
https://github.com/CrimsonDawn45/Basic-Shields-Fabric/issues/19
https://github.com/CrimsonDawn45/Basic-Shields-Fabric/issues/18
https://github.com/CrimsonDawn45/Basic-Shields-Fabric/issues/15

Enriched:
https://www.curseforge.com/minecraft/mc-mods/enriched
https://github.com/CamoMano/Enriched
Mod is now called enriched materials are:

Steel
Obsidian Alloy
Ruby
Sapphire
Tanzanite
Emerald
Copper

Copper Armor & Tools:
https://www.curseforge.com/minecraft/mc-mods/copper-armor-tools-fabric
https://github.com/agorasim20/coppermod

Copper

Amethyst:
https://www.curseforge.com/minecraft/mc-mods/amethyst-equipment

Amethyst

More Vanilla+ mods:
https://www.curseforge.com/minecraft/mc-mods/copper-equipment
https://www.curseforge.com/minecraft/mc-mods/exlines-emerald-equipment
https://www.curseforge.com/minecraft/mc-mods/obsidian-equipment

Current list of needed materials:

Steel**(Done)**
Obsidian Alloy
Ruby**(Done)**
Sapphire**(Done)**
Tanzanite
Emerald**(Done)**
Copper**(Done)**
Amethyst**(Done)**
Obsidian**(Done)**

This is completed for now.


Improve target aggregate speed

Add stream methods
Call to Strict on some Seq
Add Atom on Skeleton building of ancestors and descendants
Fix iteration in GetComposing

Tests running ok


[Bug] : <can't play anything-{iframeResponse.selectFirst(vid eoListSelector()) must not be null}>

PLEASE READ THIS
I acknowledge that:

I have updated:

To the latest version of the app (stable is v0.10.12)
All extensions

I have tried the troubleshooting guide: https://tachiyomi.org/help/guides/troubleshooting-problems/
If this is an issue with the app itself, that I should be opening an issue in https://github.com/jmir1/aniyomi
I have searched the existing issues and this is new ticket NOT a duplicate or related to another open issue
For source requests, I have checked the list of existing extensions including the multi-source spreadsheet: https://tachiyomi.org/extensions/
I will fill out the title and the information in this template

Note that the issue will be automatically closed if you do not fill out the title or requested information.
DELETE THIS SECTION IF YOU HAVE READ AND ACKNOWLEDGED IT

Device information

Tachiyomi version: ?
Android version: ?

Source information

Name: ?
Extension version: ?

Steps to reproduce

First Step
Second Step

Expected behavior
Describe what should have happened
Actual behavior
Describe what happens instead
Other details

The site removed their video server wait till they put it back again

@9MoIssam9 I think you should close this issue as the site returned to work again


something unclear

Please clarify this part of the splinter documentation

@objarni thank you for your feedback. The Makefile that the docs refers is https://github.com/cobrateam/splinter/blob/master/Makefile
Do you think if we add a mention to clone the project, enter in the project dir and then run make test will fix this problem?

@andrewsmedina yes I think that would be enough, if that is enough to make "make" work :).
:+1:

@andrewsmedina what this issue fixed? I'm trying to clean up our PRs :P


Emojis in files crashes 'packing files' stage

Hi there, just started testing repopack-py!
Some of my files have emoji in them, which made the process crash. After removal of said emojis the command works fine again.
Error (I changed the paths to remove some irrelevant information) :
`⠙ Packing files...INFO: Total files to process: 6334
⠧ Packing files...ERROR: Error processing files: Error processing file './../password_reset_key_message.txt': Error processing file ./../password_reset_key_message.txt': 'charmap' codec can't decode byte 0x8d in position 144: character maps to <undefined>
✖ Error during packing: File processing error: Error processing file ./../password_reset_key_message.txt': Error processing file ./../password_reset_key_message.txt': 'charmap' codec can't decode byte 0x8d in position 144: character maps to <undefined>
ERROR: File processing error: Error processing file './../password_reset_key_message.txt': Error processing file './../password_reset_key_message.txt': 'charmap' codec can't decode byte 0x8d in position 144: character maps to <undefined>

@yunusdemir
chardet detection is not that accurate it seems, added an option to fallback to utf-8 upon failure.
Just released a new version to fix this bug.  https://github.com/abinthomasonline/repopack-py/commit/06828dbf729191dd896d02c0f1cc0cf196c1ea44
do pip install repopack --upgrade
Thank you for pointing out!


tutorial, data binding, step 4: missing "resourceroots"

OpenUI5 version: 1.78.1 and above
Browser/version (+device/version): Firefox, Chromium
URL (minimal example if possible):
https://openui5.hana.ondemand.com/topic/c72b922fdb59422496661000165d7ff1
Steps to reproduce the problem:

follow the tutorial steps 1 to 4
implement the changes described in the chapter but do NOT download ready-made examples

What is the expected result?
step 4 works as described
What happens instead?
"Error: resource ....view/App.view.xml could not be loaded from resources/....view/App.view.xml. Check for 'file not found' or parse errors. Reason: Not Found -  "
Any other information? (attach screenshot if possible)
the code sample in the tutorial chapter itself is missing the data-sap-ui-resourceroots keyword, which is only present in the complete downloadable sample, consequently the view is searched in the wrong path

Hello @AndreaBorgia-Abo ,
Thank you for sharing this finding. I've created an internal incident<PHONE_NUMBER>. The status of the issue will be updated here in GitHub.
Regards,
Niya

Issue has been fixed from 1.84
https://openui5.hana.ondemand.com/topic/4cde849dc18949b69bbe152a0aaa19c6


WIP Update deps + changelog to v3.7.1

Related to https://github.com/camunda/camunda-modeler/issues/1729

There is one more fix that I'd like to include in this release https://github.com/bpmn-io/bpmn-js/pull/1293.

@philippfromme ready for review


compose: Improve error message for mentioning a user who isn't subscr…

Fixes: #8550  Changed error message and styled it according to different screen sizes

I had the same concern as @lonerz, and also think the CSS for the line-wrapping could be better.  I asked @synicalsyntax to help with that, and she submitted #8603, which I've now merged.  Thanks for your efforts @ihsavru!


Generate multiple projects

Hi
I want to use CRA to create multiple brands in same application.
Basically, source code will render two folders inside dist folder (one folder for each brand) with the same code but different theming and some minor changes.
Let's say I have two shopping websites ABC and XYZ with almost same functionality and different theming and I want to generate two output folder, one for each website.
Can I achieve this in CRA?
If yes with eject or without eject?

CRA 2.0 will have support for sharing source via yarn workspace monorepos , so you could have multiple CRA apps in a monorepo using the same source app component.  Maybe combine that with .env vars and you can achieve brand customization?


Isolate m.prop() and m.deferred() implementations (mostly)

This mostly isolates the implementations for both of these. Now, everything
here calls the method itself, not any of the external methods.
Few driveby fixes as well:

Git now ignores archive/ again (it's a build artifact, and can be removed
when updating master)
Since I had to rewrite most of the Deferred implementation, the new version
passes one of the skipped tests, so it is now enabled.

Prepares for #800.

@impinball, what are the performance implications of this?

About the same. The bottleneck is still in the rendering, which probably
needs rewritten to reduce GC pressure.
On Fri, Nov 20, 2015, 11:50 Richard Eames<EMAIL_ADDRESS>wrote:

@impinball https://github.com/impinball, what are the performance
implications of this?
—
Reply to this email directly or view it on GitHub
https://github.com/lhorie/mithril.js/pull/853#issuecomment-158457253.

Promises should be slightly faster, but I haven't benchmarked it much.
On Fri, Nov 20, 2015, 19:16 Isiah Meadows<EMAIL_ADDRESS>wrote:

About the same. The bottleneck is still in the rendering, which probably
needs rewritten to reduce GC pressure.
On Fri, Nov 20, 2015, 11:50 Richard Eames<EMAIL_ADDRESS>wrote:

@impinball https://github.com/impinball, what are the performance
implications of this?
—
Reply to this email directly or view it on GitHub
https://github.com/lhorie/mithril.js/pull/853#issuecomment-158457253.

I would also like to note that this is slightly breaking. m.prop() no longer returns a Mithril promise when wrapping a third-party promise, which could break those depending on this behavior.

That sound like a passing assertion :)
What happens in author-land resulting in user error with 3rd party integration should stay there. Mithril already documents its disparities with the spec. There is only so much protection you can reasonably offer!

@barneycarroll And a conforming Promises/A+ implementation would see the m.prop() as a thenable, and would grab the then out of it. If the promise implementation raises a fuss over that, it's a problem with the implementation.
Another breaking change: m.then() doesn't return a memoized value (which it shouldn't). It's another deviation from spec, but it's the expected behavior - you're changing the internal value of the thenable when you call the function.

:shipit:

/cc @lhorie

Merging now. Looking at it again, I don't think that it's a broad enough use case yet that the breaking change would cause problems.

really nice work @isiahmeadows !

@tinchoz49 Thanks!


[F5] [F5.1] Attempts details

Attempts details
Related High-Level Requirement: History Service of User Attempts
Description
Attempts of sessions to be added, updated and displayed when session ends for a specific user.
Features

Create Attempt after session ends
Update specific user's attempt's notes
Delete attempt of specific user
Get all user's attempts

Closing - completed


fix 500 error on bad requests

The following HTTP GET request returns 500:
https://oscar.wmo.int/surface/rest/api/stations/approvedStations/wmoIds
The WADL in https://oscardevt.meteoswiss.ch/surface/rest/api?_wadl says to include the pageSize, q, and page parameters are part of the request.
For missing parameters, the HTTP response should be 400, not 500

The acceptable endpoints are listed in the wadl, e.g., .../wigosIds, .../gawIds.


⚠ Samsung has degraded performance

In 4439092, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2686 ms

Resolved: Samsung performance has improved in 191ad10.


Custom sampler policy

Motivation:
Replace Add custom sampling strategy  to provide a custom sampling strategy
Result:
Fixes #49
If there is no issue then describe the changes introduced by this PR.

Pull Request Test Coverage Report for Build 362

26 of 56 (46.43%)  changed or added relevant lines in 9 files are covered.
5 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.1%) to 64.039%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

tracer-core/src/main/java/com/alipay/common/tracer/core/reporter/digest/AbstractDiskReporter.java
1
2
50.0%

tracer-sofa-boot-starter/src/main/java/com/alipay/sofa/tracer/boot/listener/SofaTracerConfigurationListener.java
3
5
60.0%

tracer-core/src/main/java/com/alipay/common/tracer/core/configuration/SofaTracerConfiguration.java
0
4
0.0%

tracer-core/src/main/java/com/alipay/common/tracer/core/samplers/SamplerFactory.java
11
16
68.75%

tracer-sofa-boot-starter/src/main/java/com/alipay/sofa/tracer/boot/properties/SofaTracerProperties.java
4
10
40.0%

tracer-samples/tracer-sample-with-springmvc/src/main/java/com/alipay/sofa/tracer/examples/springmvc/sampler/CustomOpenRulesSamplerRuler.java
0
12
0.0%

Files with Coverage Reduction
New Missed Lines
%

tracer-core/src/main/java/com/alipay/common/tracer/core/samplers/SofaTracerPercentageBasedSampler.java
5
71.05%

Totals

Change from base Build 352:
-0.1%

Covered Lines:
2835

Relevant Lines:
4427

💛  - Coveralls


Report only latest log folder to CrashReporter

With these changes only the logging subfolder that is actually used is reported to CrashReporter.
Fixes https://github.com/MovingBlocks/CrashReporter/issues/19

Refer to this link for build results (access rights to CI server needed):
http://jenkins.terasology.org/job/TerasologyPRs/124/
Hooray Jenkins reported success with all tests good!

Neat! Checked it out, looks good :-)


File Explorer preview feature request

📝 Provide a description of the new feature
File Explorer preview feature to assign my own simple previewer (or reassign an existing one) for ANY file type. Eg: *.url files -- I'd like to reassign the preview to simply look at the text of the file. Now THAT would be useful my friends. Instead, I get just SVG and MD previews... that's like promising a meal and getting a plate with a peanut in the middle.  Also, btw, techrepublic said it would do PDF preview - but doesn't.

If you'd like to see this feature implemented, add a 👍 reaction to this post.

That could be possible but i'd group this in with the https://github.com/microsoft/PowerToys/issues/1527 idea since that itself would have to do a lot of dynamic registry work

Thanks guys. I selected "suggested feature somewhere but this seems to have gone into the prob slot. Keep up the good work.

We should be able to see the files with code (python, ruby, css, js ,...) with the colors as a vim color and format preview


Use valid Java package names

Updated the Android code to use valid Java packages names. The package names aren't a problem when using older versions of Java but newer versions of Java validate the names and cause build failures.
Resolves #18

I will take a look at CI soon and figure out why its failing. The failures I am seeing do not look related to your changes.

Once https://github.com/rustls/rustls-platform-verifier/pull/20 merges and fixes CI, this should be good to merge.


How to analysis tdeserver log?

Hi,
I noticed that after analysis，there is not a result is related to tdeserver.
How can I analysis tdeserver log?

Are you using version 2.1? Wasn't the TDE analysis retired in the patch notes for version 2.1? You may want to use 2.0 if you're still using a non-Hyper Tableau Version.
Copied from the notes on 2.1
"Logshark 2.1 includes numerous fixes and improvements made since the previous release, but the most notable changes are:
Support for log sets generated by new versions of Tableau Server (including Tableau Server on Linux)
New plugin: Hyper
Retired plugin: DataEngine"

I am using version 2.0.Here's the result:

Hi,
In version 2.0 DataEngine plugin is present, but disabled by default (due to poor performance). If you would like to run it on your log set, you'll have to either add it to the list in LogShark.config, or specify this plugin with "--plugin" parameter in command prompt


ARTEMIS-4922 remove spurious warn msg when OpenWire client disconnects

I tested this locally and the errors are gone now


Hide Incline when there is just one window

Hello there!
I'm loving the plugin, it's really handy with the new global status bar.
My only problem with it is that I see no point in showing it when there is just one window (expected behaviour for me is: 1 window => Hide | 2 or more windows => Show)
I have cloned it and tried to implement it myself but not there yet 😅
Maybe you feel this is not necessary, feel free to close it then!
Have a nice day and thanks for the plugin!

  hide = {
    focused_win = true,
  },

should take care of single window, shouldn't it?

Yeah the option @gegoune mentioned should work. Give it a try and let me know if that’s what you’re looking for?

Just to chime in, I'm not currently using this plugin (though I am planning to give it a shot), but I can imagine situations where I wouldn't want it to be hidden every time a window is focused, only when there's only one window open. For example, when the top window of a horizontal split is focused, just looking at the bottom of that window is easier and more intuitive than looking all the way at the bottom of the screen, so I probably wouldn't want the status line hidden in that case. But if there's only one window open, the floating status line is going to be redundant, and the bottom of the window is where the main status line is anyway. So I still think it would be helpful for this to be an additional option.

That is a great point and I agree, @pianocomposer321. I’ll add this to the backlog.

Does anyone have thoughts on how this proposed option should behave when there is more than one window, but only one non-ignored window?
For example, if you open the file foobar.lua in one window and nvim-tree in another window, and if you have ignore.buftypes = "special", then there would effectively be only one non-ignored window. If the proposed option was enabled, should Incline be shown or hidden on the foobar.lua window?

Hmm...that is an interesting question.
TBH, I feel like this would be good as two separate options. Or an option with an "enum"-style value. But if you don't want to implement two separate options (or just one at a time), I think that it should be hidden when it's the last non-ignored window. The main issue with this would be if a window is ignored with incline but not for the main statusline (meaning that when it's focused you wouldn't be able to tell the name of the file in the non-ignored window), but leaving that up to the users to fix doesn't seem entirely unreasonable.
Of course, having two separate options still seems like the best approach to me.

Just jumping in regarding the two separate options. I'm currently using the plugin and really enjoying it with the current focused win option I'd much rather retain the current behaviour, i.e. the buffer I'm in should never have a label since my statusline will always show it. Trying not to muddy the waters but advocate for a different option that does what @pianocomposer321 proposed but also keeping the ability to have the plugin behave as it currently does.

@akinsho I just want to clarify, because I know this has gotten a bit confusing, and I think my last comment probably is the source of a lot of it. There are actually a total of three options being discussed right now - focused_win, which already exists, and two more (I'll just make up names because they haven't been implemented yet): last_win and last_nonignored_win. The final two are the ones that I was saying I think should be two separate options, with the latter being the one I thought was more important if only one of them gets implemented. I completely agree that focused_win should be left as-is.


Expected a supported JSON media type but got "application/octet-stream".

Expected Behavior
When publishing and receiving events through pubsub component using rawPayload=true, receiving subscriber endpoint should be able to deserialize response body into a valid object.
Actual Behavior
When receiving events from topic subscriber where enableRawPayload=true and message was sent with rawPayload=true with Content-Type of application/json, the receiving end throws an exception from the useCloudEvents middleware.

Microsoft.AspNetCore.Http.BadHttpRequestException: Expected a supported JSON media type but got "application/octet-stream".
at Microsoft.AspNetCore.Http.RequestDelegateFactory.Log.UnexpectedJsonContentType(HttpContext httpContext, String contentType, Boolean shouldThrow)
at Microsoft.AspNetCore.Http.RequestDelegateFactory.g__TryReadBodyAsync|90_0(HttpContext httpContext, Type bodyType, String parameterTypeName, String parameterName, Boolean allowEmptyRequestBody, Boolean throwOnBadRequest)
at Microsoft.AspNetCore.Http.RequestDelegateFactory.<>c__DisplayClass90_2.<b__2>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Dapr.CloudEventsMiddleware.ProcessBodyAsync(HttpContext httpContext, String charSet)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

Or when the app.UseCloudEvents() middleware is completely removed the source binding binds the data as null. When using streamReader to read the request.Body it is:

{
"data_base64": "ew0KICAgICJvcmRlcklkIjogIjMiLA0KICAgICJyZXF1ZXN0SWQiOiAiMSIsDQogICAgImZvb2JhcnMiOiBbDQogICAgICAgIHsNCiAgICAgICAgICAgICJuYW1lIjogIkZvbyIsDQogICAgICAgICAgICAibGFzdG5hbWUiOiAiQmFyIg0KICAgICAgICB9DQogICAgXQ0KfQ\u003d\u003d",
"datacontenttype": "application/octet-stream",
"id": "60071be7-1379-45c4-8f3a-fea2202b8b89",
"pubsubname": "servicebus-pubsub",
"source": "Dapr",
"specversion": "1.0",
"topic": "sbt-foo-bar-topic",
"type": "com.dapr.event.sent"
}

Steps to Reproduce the Problem
Configure startup with app.UseCloudEvents();
Publish messages where rawPayload=true set up a subscriber with enableRawPayload = true.
Questions
How raw payloads should be deserialized? There is so little information about how to use pubsub without CloudEvents as rawPayload messages, especially for the AspNetCore SDK. I am starting to feel like using ServiceBus SDK to receive messages outside of dapr cluster.

The UseCloudEvents middleware is strongly tied to the event actually being the exact format of CloudEvent that Dapr sends. When I've used raw messages in the past, what I've done is just have the endpoint take an object that the data can be deserialzied into and accessed data in the handler and done the deserialization there. You could do this with your own middleware too if you want a slightly nicer experience.

Ok, thanks for the tip. That's what I thought, ended up writing custom handler with ServiceBus SDK so we can have indented communication between dapr instances using pubsub with CloudEvents utilizing UseCloudEvents middleware.
As a developer It would be nice if UseCloudEvents could co-exist with endpoints using rawPayloads. Then you would have the flexibility to use rawPayloads with some lecagy brokers without known json schema and CloudEvents normally if you don't explicitly tell to Dapr SDK to listen subscripiption with rawPayload.


Merge Workflow Patch

Description
Fixes the path to the data_management requirements file in the merge workflow.

🚀 [bumpr] Bumped! New version:v1.0.1 Changes:v1.0.0...v1.0.1


Simple Typo

https://github.com/masayuki0812/c3/blob/master/c3.js#L720
Says 'categor' and should be 'category'

@GordonSmith Hi! Good spot!
Want to do an easy Pull Request? Go here, save and follow the directions. Make sure to put "Fixes #1007" in the commit message! :smile:


Will not let me "Run Test" even though my code is correct

Challenge Waypoint: Give a Background Color to a Div Element has an issue.
User Agent is: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:
<link href="http://fonts.googleapis.com/css?family=Lobster" rel="stylesheet" type="text/css">

CatPhotoApp

Click here for cat photos.

Things cats love:

cat nip

laser pointers

lasagna

Top 3 things cats hate:

flea treatment

thunder

other cats

<form action="/submit-cat-photo">
  <input type="radio" name="indoor-outdoor" checked> Indoor
  <input type="radio" name="indoor-outdoor"> Outdoor
  <input type="checkbox" name="personality" checked> Loving
  <input type="checkbox" name="personality"> Lazy
  <input type="checkbox" name="personality"> Energetic
  <input type="text" placeholder="cat photo URL" required>
  <button type="submit">Submit</button>
</form>

Looks like you are using a different spelling of gray than the challenge is asking for. You used grey when it is looking for gray.
Closing as not an issue with the tests.
Thanks and happy coding!


ZBMINI recognized as LIGHT being the correct SWITCH

When Turning on, it displays the following error:
Failed to call service light / turn_on. 'EWeLinkToggle' object has no attribute 'supported_color_modes'
Turning off works normally, when connected with eWeLink.

Duplicate #444


BWA MEM yields an empty alignment

I have ran bwaMem through cannoli, but all I'm getting are empty sam files with only the header.
Note that I only have single-end reads which I had labeled as .ifq because cannoli wouldn't take .fq or .fastq.
time cannoli-submit bwaMem $PWD/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.output  -sample_id HG00513 -index $PWD/human_g1k_v37.fasta -sequence_dictionary $PWD/human_g1k_v37.dict  -add_files
Using SPARK_SUBMIT=/usr/local/Cellar/apache-spark/3.1.2/libexec/bin/spark-submit
21/10/06 00:51:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
log4j:WARN No appenders could be found for logger (org.bdgenomics.cannoli.cli.Cannoli).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/10/06 00:51:37 INFO SparkContext: Running Spark version 3.1.2
21/10/06 00:51:37 INFO ResourceUtils: ==============================================================
21/10/06 00:51:37 INFO ResourceUtils: No custom resources configured for spark.driver.
21/10/06 00:51:37 INFO ResourceUtils: ==============================================================
21/10/06 00:51:37 INFO SparkContext: Submitted application: bwaMem
21/10/06 00:51:37 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
21/10/06 00:51:37 INFO ResourceProfile: Limiting resource is cpu
21/10/06 00:51:37 INFO ResourceProfileManager: Added ResourceProfile id: 0
21/10/06 00:51:37 INFO SecurityManager: Changing view acls to: yassinesouilmi
21/10/06 00:51:37 INFO SecurityManager: Changing modify acls to: yassinesouilmi
21/10/06 00:51:37 INFO SecurityManager: Changing view acls groups to:
21/10/06 00:51:37 INFO SecurityManager: Changing modify acls groups to:
21/10/06 00:51:37 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yassinesouilmi); groups with view permissions: Set(); users  with modify permissions: Set(yassinesouilmi); groups with modify permissions: Set()
21/10/06 00:51:37 INFO Utils: Successfully started service 'sparkDriver' on port 61655.
21/10/06 00:51:37 INFO SparkEnv: Registering MapOutputTracker
21/10/06 00:51:37 INFO SparkEnv: Registering BlockManagerMaster
21/10/06 00:51:37 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/10/06 00:51:37 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/10/06 00:51:37 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
21/10/06 00:51:37 INFO DiskBlockManager: Created local directory at /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/blockmgr-091e049e-7bcb-4cb4-91a3-a34c92be9835
21/10/06 00:51:37 INFO MemoryStore: MemoryStore started with capacity 366.3 MiB
21/10/06 00:51:37 INFO SparkEnv: Registering OutputCommitCoordinator
21/10/06 00:51:37 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/10/06 00:51:37 INFO SparkUI: Bound SparkUI to <IP_ADDRESS>, and started at http://yassines-ipro.lan:4040
21/10/06 00:51:37 INFO SparkContext: Added JAR file:/usr/local/Cellar/cannoli/0.13.0/libexec/bin/../repo/cannoli-assembly-spark3_2.12-0.13.0.jar at spark://yassines-ipro.lan:61655/jars/cannoli-assembly-spark3_2.12-0.13.0.jar with timestamp<PHONE_NUMBER>260
21/10/06 00:51:38 INFO Executor: Starting executor ID driver on host yassines-ipro.lan
21/10/06 00:51:38 INFO Executor: Fetching spark://yassines-ipro.lan:61655/jars/cannoli-assembly-spark3_2.12-0.13.0.jar with timestamp<PHONE_NUMBER>260
21/10/06 00:51:38 INFO TransportClientFactory: Successfully created connection to yassines-ipro.lan/<IP_ADDRESS>:61655 after 23 ms (0 ms spent in bootstraps)
21/10/06 00:51:38 INFO Utils: Fetching spark://yassines-ipro.lan:61655/jars/cannoli-assembly-spark3_2.12-0.13.0.jar to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/fetchFileTemp4452330962860272692.tmp
21/10/06 00:51:38 INFO Executor: Adding file:/private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/cannoli-assembly-spark3_2.12-0.13.0.jar to class loader
21/10/06 00:51:38 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 61657.
21/10/06 00:51:38 INFO NettyBlockTransferService: Server created on yassines-ipro.lan:61657
21/10/06 00:51:38 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/10/06 00:51:38 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, yassines-ipro.lan, 61657, None)
21/10/06 00:51:38 INFO BlockManagerMasterEndpoint: Registering block manager yassines-ipro.lan:61657 with 366.3 MiB RAM, BlockManagerId(driver, yassines-ipro.lan, 61657, None)
21/10/06 00:51:38 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, yassines-ipro.lan, 61657, None)
21/10/06 00:51:38 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, yassines-ipro.lan, 61657, None)
21/10/06 00:51:38 INFO ADAMContext: Loading /Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq as interleaved FASTQ and converting to Fragments.
21/10/06 00:51:38 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 314.8 KiB, free 366.0 MiB)
21/10/06 00:51:39 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 27.2 KiB, free 366.0 MiB)
21/10/06 00:51:39 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on yassines-ipro.lan:61657 (size: 27.2 KiB, free: 366.3 MiB)
21/10/06 00:51:39 INFO SparkContext: Created broadcast 0 from newAPIHadoopFile at ADAMContext.scala:3001
21/10/06 00:51:40 INFO BwaMem: Piping RDDBoundFragmentDataset with 0 reference sequences, 0 read groups, and 0 processing steps to bwa with command: [bwa, mem, -t, 1, -R, @RG\tID:HG00513\tSM:HG00513\tLB:HG00513\tPL:ILLUMINA\tPU:0, -p, $0, -] files: [file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta, file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.amb, file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.ann, file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.bwt, file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.pac, file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.sa]
21/10/06 00:51:40 INFO SparkContext: Added file file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta at file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta with timestamp<PHONE_NUMBER>638
21/10/06 00:51:40 INFO Utils: Copying /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/human_g1k_v37.fasta
21/10/06 00:51:52 INFO SparkContext: Added file file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.amb at file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.amb with timestamp<PHONE_NUMBER>807
21/10/06 00:51:52 INFO Utils: Copying /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.amb to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/human_g1k_v37.fasta.amb
21/10/06 00:51:52 INFO SparkContext: Added file file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.ann at file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.ann with timestamp<PHONE_NUMBER>818
21/10/06 00:51:52 INFO Utils: Copying /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.ann to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/human_g1k_v37.fasta.ann
21/10/06 00:51:52 INFO SparkContext: Added file file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.bwt at file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.bwt with timestamp<PHONE_NUMBER>831
21/10/06 00:51:52 INFO Utils: Copying /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.bwt to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/human_g1k_v37.fasta.bwt
21/10/06 00:51:59 INFO SparkContext: Added file file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.pac at file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.pac with timestamp<PHONE_NUMBER>320
21/10/06 00:51:59 INFO Utils: Copying /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.pac to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/human_g1k_v37.fasta.pac
21/10/06 00:52:02 INFO SparkContext: Added file file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.sa at file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.sa with timestamp<PHONE_NUMBER>418
21/10/06 00:52:02 INFO Utils: Copying /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.sa to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/human_g1k_v37.fasta.sa
21/10/06 00:52:06 INFO FileInputFormat: Total input files to process : 1
21/10/06 00:52:06 INFO ADAMContext: Loading /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.dict as HTSJDK sequence dictionary.
21/10/06 00:52:06 INFO RDDBoundAlignmentDataset: Saving data in SAM/BAM/CRAM format
21/10/06 00:52:06 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 53.8 KiB, free 365.9 MiB)
21/10/06 00:52:06 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 4.4 KiB, free 365.9 MiB)
21/10/06 00:52:06 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on yassines-ipro.lan:61657 (size: 4.4 KiB, free: 366.3 MiB)
21/10/06 00:52:06 INFO SparkContext: Created broadcast 1 from broadcast at AlignmentDataset.scala:735
21/10/06 00:52:06 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:06 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:06 INFO SparkContext: Starting job: runJob at SparkHadoopWriter.scala:83
21/10/06 00:52:06 INFO DAGScheduler: Got job 0 (runJob at SparkHadoopWriter.scala:83) with 12 output partitions
21/10/06 00:52:06 INFO DAGScheduler: Final stage: ResultStage 0 (runJob at SparkHadoopWriter.scala:83)
21/10/06 00:52:06 INFO DAGScheduler: Parents of final stage: List()
21/10/06 00:52:06 INFO DAGScheduler: Missing parents: List()
21/10/06 00:52:06 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[4] at keyBy at AlignmentDataset.scala:819), which has no missing parents
21/10/06 00:52:06 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 107.8 KiB, free 365.8 MiB)
21/10/06 00:52:06 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 40.4 KiB, free 365.8 MiB)
21/10/06 00:52:06 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on yassines-ipro.lan:61657 (size: 40.4 KiB, free: 366.2 MiB)
21/10/06 00:52:06 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1388
21/10/06 00:52:06 INFO DAGScheduler: Submitting 12 missing tasks from ResultStage 0 (MapPartitionsRDD[4] at keyBy at AlignmentDataset.scala:819) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11))
21/10/06 00:52:06 INFO TaskSchedulerImpl: Adding task set 0.0 with 12 tasks resource profile 0
21/10/06 00:52:06 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (yassines-ipro.lan, executor driver, partition 0, PROCESS_LOCAL, 4620 bytes) taskResourceAssignments Map()
21/10/06 00:52:06 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1) (yassines-ipro.lan, executor driver, partition 1, PROCESS_LOCAL, 4620 bytes) taskResourceAssignments Map()
21/10/06 00:52:06 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2) (yassines-ipro.lan, executor driver, partition 2, PROCESS_LOCAL, 4620 bytes) taskResourceAssignments Map()
21/10/06 00:52:06 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3) (yassines-ipro.lan, executor driver, partition 3, PROCESS_LOCAL, 4620 bytes) taskResourceAssignments Map()
21/10/06 00:52:06 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4) (yassines-ipro.lan, executor driver, partition 4, PROCESS_LOCAL, 4620 bytes) taskResourceAssignments Map()
21/10/06 00:52:06 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5) (yassines-ipro.lan, executor driver, partition 5, PROCESS_LOCAL, 4620 bytes) taskResourceAssignments Map()
21/10/06 00:52:06 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6) (yassines-ipro.lan, executor driver, partition 6, PROCESS_LOCAL, 4620 bytes) taskResourceAssignments Map()
21/10/06 00:52:06 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7) (yassines-ipro.lan, executor driver, partition 7, PROCESS_LOCAL, 4620 bytes) taskResourceAssignments Map()
21/10/06 00:52:06 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8) (yassines-ipro.lan, executor driver, partition 8, PROCESS_LOCAL, 4620 bytes) taskResourceAssignments Map()
21/10/06 00:52:06 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9) (yassines-ipro.lan, executor driver, partition 9, PROCESS_LOCAL, 4620 bytes) taskResourceAssignments Map()
21/10/06 00:52:06 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10) (yassines-ipro.lan, executor driver, partition 10, PROCESS_LOCAL, 4620 bytes) taskResourceAssignments Map()
21/10/06 00:52:06 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11) (yassines-ipro.lan, executor driver, partition 11, PROCESS_LOCAL, 4620 bytes) taskResourceAssignments Map()
21/10/06 00:52:06 INFO Executor: Running task 7.0 in stage 0.0 (TID 7)
21/10/06 00:52:06 INFO Executor: Running task 11.0 in stage 0.0 (TID 11)
21/10/06 00:52:06 INFO Executor: Running task 4.0 in stage 0.0 (TID 4)
21/10/06 00:52:06 INFO Executor: Running task 9.0 in stage 0.0 (TID 9)
21/10/06 00:52:06 INFO Executor: Running task 3.0 in stage 0.0 (TID 3)
21/10/06 00:52:06 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/10/06 00:52:06 INFO Executor: Running task 8.0 in stage 0.0 (TID 8)
21/10/06 00:52:06 INFO Executor: Running task 5.0 in stage 0.0 (TID 5)
21/10/06 00:52:06 INFO Executor: Running task 6.0 in stage 0.0 (TID 6)
21/10/06 00:52:06 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
21/10/06 00:52:06 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
21/10/06 00:52:06 INFO Executor: Running task 10.0 in stage 0.0 (TID 10)
21/10/06 00:52:06 INFO Executor: Fetching file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.ann with timestamp<PHONE_NUMBER>818
21/10/06 00:52:06 INFO Utils: /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.ann has been previously copied to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/human_g1k_v37.fasta.ann
21/10/06 00:52:06 INFO Executor: Fetching file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.pac with timestamp<PHONE_NUMBER>320
21/10/06 00:52:07 INFO Utils: /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.pac has been previously copied to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/human_g1k_v37.fasta.pac
21/10/06 00:52:07 INFO Executor: Fetching file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.bwt with timestamp<PHONE_NUMBER>831
21/10/06 00:52:11 INFO Utils: /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.bwt has been previously copied to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/human_g1k_v37.fasta.bwt
21/10/06 00:52:11 INFO Executor: Fetching file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.amb with timestamp<PHONE_NUMBER>807
21/10/06 00:52:11 INFO Utils: /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.amb has been previously copied to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/human_g1k_v37.fasta.amb
21/10/06 00:52:11 INFO Executor: Fetching file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.sa with timestamp<PHONE_NUMBER>418
21/10/06 00:52:13 INFO Utils: /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta.sa has been previously copied to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/human_g1k_v37.fasta.sa
21/10/06 00:52:13 INFO Executor: Fetching file:/Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta with timestamp<PHONE_NUMBER>638
21/10/06 00:52:27 INFO Utils: /Users/yassinesouilmi/Downloads/parallelized_bwa/human_g1k_v37.fasta has been previously copied to /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad/userFiles-b163806c-98af-4319-a086-83df910f2d9c/human_g1k_v37.fasta
21/10/06 00:52:27 INFO NewHadoopRDD: Input split: file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq:0+33554432
21/10/06 00:52:27 INFO NewHadoopRDD: Input split: file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq:335544320+33554432
21/10/06 00:52:27 INFO NewHadoopRDD: Input split: file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq:167772160+33554432
21/10/06 00:52:27 INFO NewHadoopRDD: Input split: file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq:67108864+33554432
21/10/06 00:52:27 INFO NewHadoopRDD: Input split: file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq:301989888+33554432
21/10/06 00:52:27 INFO NewHadoopRDD: Input split: file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq:268435456+33554432
21/10/06 00:52:27 INFO NewHadoopRDD: Input split: file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq:369098752+23902525
21/10/06 00:52:27 INFO NewHadoopRDD: Input split: file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq:100663296+33554432
21/10/06 00:52:27 INFO NewHadoopRDD: Input split: file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq:201326592+33554432
21/10/06 00:52:27 INFO NewHadoopRDD: Input split: file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq:234881024+33554432
21/10/06 00:52:27 INFO NewHadoopRDD: Input split: file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq:33554432+33554432
21/10/06 00:52:27 INFO NewHadoopRDD: Input split: file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.ifq:134217728+33554432
21/10/06 00:52:28 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:28 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:28 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:28 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:28 INFO FileOutputCommitter: Saved output of task 'attempt_202110060052064203496344395362105_0004_r_000011_0' to file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.sam/_temporary/0/task_202110060052064203496344395362105_0004_r_000011
21/10/06 00:52:28 INFO SparkHadoopMapRedUtil: attempt_202110060052064203496344395362105_0004_r_000011_0: Committed
21/10/06 00:52:28 INFO Executor: Finished task 11.0 in stage 0.0 (TID 11). 935 bytes result sent to driver
21/10/06 00:52:28 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 22154 ms on yassines-ipro.lan (executor driver) (1/12)
21/10/06 00:52:28 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:28 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:28 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:28 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: Saved output of task 'attempt_202110060052064203496344395362105_0004_r_000010_0' to file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.sam/_temporary/0/task_202110060052064203496344395362105_0004_r_000010
21/10/06 00:52:29 INFO SparkHadoopMapRedUtil: attempt_202110060052064203496344395362105_0004_r_000010_0: Committed
21/10/06 00:52:29 INFO Executor: Finished task 10.0 in stage 0.0 (TID 10). 935 bytes result sent to driver
21/10/06 00:52:29 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 22626 ms on yassines-ipro.lan (executor driver) (2/12)
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: Saved output of task 'attempt_202110060052064203496344395362105_0004_r_000009_0' to file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.sam/_temporary/0/task_202110060052064203496344395362105_0004_r_000009
21/10/06 00:52:29 INFO SparkHadoopMapRedUtil: attempt_202110060052064203496344395362105_0004_r_000009_0: Committed
21/10/06 00:52:29 INFO Executor: Finished task 9.0 in stage 0.0 (TID 9). 935 bytes result sent to driver
21/10/06 00:52:29 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 23006 ms on yassines-ipro.lan (executor driver) (3/12)
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: Saved output of task 'attempt_202110060052064203496344395362105_0004_r_000008_0' to file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.sam/_temporary/0/task_202110060052064203496344395362105_0004_r_000008
21/10/06 00:52:29 INFO SparkHadoopMapRedUtil: attempt_202110060052064203496344395362105_0004_r_000008_0: Committed
21/10/06 00:52:29 INFO Executor: Finished task 8.0 in stage 0.0 (TID 8). 935 bytes result sent to driver
21/10/06 00:52:29 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 23090 ms on yassines-ipro.lan (executor driver) (4/12)
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: Saved output of task 'attempt_202110060052064203496344395362105_0004_r_000007_0' to file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.sam/_temporary/0/task_202110060052064203496344395362105_0004_r_000007
21/10/06 00:52:29 INFO SparkHadoopMapRedUtil: attempt_202110060052064203496344395362105_0004_r_000007_0: Committed
21/10/06 00:52:29 INFO Executor: Finished task 7.0 in stage 0.0 (TID 7). 935 bytes result sent to driver
21/10/06 00:52:29 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 23161 ms on yassines-ipro.lan (executor driver) (5/12)
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: Saved output of task 'attempt_202110060052064203496344395362105_0004_r_000006_0' to file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.sam/_temporary/0/task_202110060052064203496344395362105_0004_r_000006
21/10/06 00:52:29 INFO SparkHadoopMapRedUtil: attempt_202110060052064203496344395362105_0004_r_000006_0: Committed
21/10/06 00:52:29 INFO Executor: Finished task 6.0 in stage 0.0 (TID 6). 935 bytes result sent to driver
21/10/06 00:52:29 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 23222 ms on yassines-ipro.lan (executor driver) (6/12)
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: Saved output of task 'attempt_202110060052064203496344395362105_0004_r_000005_0' to file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.sam/_temporary/0/task_202110060052064203496344395362105_0004_r_000005
21/10/06 00:52:29 INFO SparkHadoopMapRedUtil: attempt_202110060052064203496344395362105_0004_r_000005_0: Committed
21/10/06 00:52:29 INFO Executor: Finished task 5.0 in stage 0.0 (TID 5). 935 bytes result sent to driver
21/10/06 00:52:29 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 23277 ms on yassines-ipro.lan (executor driver) (7/12)
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: Saved output of task 'attempt_202110060052064203496344395362105_0004_r_000004_0' to file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.sam/_temporary/0/task_202110060052064203496344395362105_0004_r_000004
21/10/06 00:52:29 INFO SparkHadoopMapRedUtil: attempt_202110060052064203496344395362105_0004_r_000004_0: Committed
21/10/06 00:52:29 INFO Executor: Finished task 4.0 in stage 0.0 (TID 4). 935 bytes result sent to driver
21/10/06 00:52:29 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 23337 ms on yassines-ipro.lan (executor driver) (8/12)
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: Saved output of task 'attempt_202110060052064203496344395362105_0004_r_000003_0' to file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.sam/_temporary/0/task_202110060052064203496344395362105_0004_r_000003
21/10/06 00:52:29 INFO SparkHadoopMapRedUtil: attempt_202110060052064203496344395362105_0004_r_000003_0: Committed
21/10/06 00:52:29 INFO Executor: Finished task 3.0 in stage 0.0 (TID 3). 935 bytes result sent to driver
21/10/06 00:52:29 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 23379 ms on yassines-ipro.lan (executor driver) (9/12)
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: Saved output of task 'attempt_202110060052064203496344395362105_0004_r_000002_0' to file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.sam/_temporary/0/task_202110060052064203496344395362105_0004_r_000002
21/10/06 00:52:29 INFO SparkHadoopMapRedUtil: attempt_202110060052064203496344395362105_0004_r_000002_0: Committed
21/10/06 00:52:29 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 935 bytes result sent to driver
21/10/06 00:52:29 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 23407 ms on yassines-ipro.lan (executor driver) (10/12)
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: Saved output of task 'attempt_202110060052064203496344395362105_0004_r_000001_0' to file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.sam/_temporary/0/task_202110060052064203496344395362105_0004_r_000001
21/10/06 00:52:29 INFO SparkHadoopMapRedUtil: attempt_202110060052064203496344395362105_0004_r_000001_0: Committed
21/10/06 00:52:29 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 935 bytes result sent to driver
21/10/06 00:52:29 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 23463 ms on yassines-ipro.lan (executor driver) (11/12)
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
21/10/06 00:52:29 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/10/06 00:52:29 INFO FileOutputCommitter: Saved output of task 'attempt_202110060052064203496344395362105_0004_r_000000_0' to file:/Users/yassinesouilmi/Downloads/parallelized_bwa/HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.sam/_temporary/0/task_202110060052064203496344395362105_0004_r_000000
21/10/06 00:52:29 INFO SparkHadoopMapRedUtil: attempt_202110060052064203496344395362105_0004_r_000000_0: Committed
21/10/06 00:52:29 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 935 bytes result sent to driver
21/10/06 00:52:29 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 23509 ms on yassines-ipro.lan (executor driver) (12/12)
21/10/06 00:52:29 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
21/10/06 00:52:29 INFO DAGScheduler: ResultStage 0 (runJob at SparkHadoopWriter.scala:83) finished in 23.580 s
21/10/06 00:52:29 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
21/10/06 00:52:29 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
21/10/06 00:52:29 INFO DAGScheduler: Job 0 finished: runJob at SparkHadoopWriter.scala:83, took 23.621515 s
21/10/06 00:52:29 INFO SparkHadoopWriter: Job job_202110060052064203496344395362105_0004 committed.
21/10/06 00:52:29 INFO SparkUI: Stopped Spark web UI at http://yassines-ipro.lan:4040
21/10/06 00:52:29 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/10/06 00:52:29 INFO MemoryStore: MemoryStore cleared
21/10/06 00:52:29 INFO BlockManager: BlockManager stopped
21/10/06 00:52:29 INFO BlockManagerMaster: BlockManagerMaster stopped
21/10/06 00:52:29 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/10/06 00:52:30 INFO SparkContext: Successfully stopped SparkContext
21/10/06 00:52:30 INFO ShutdownHookManager: Shutdown hook called
21/10/06 00:52:30 INFO ShutdownHookManager: Deleting directory /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-de5d0eaf-b24c-46f3-9459-0a70dab892b5
21/10/06 00:52:30 INFO ShutdownHookManager: Deleting directory /private/var/folders/7z/q0j48jb10hg3b6_wlr2ww3sh0000gn/T/spark-1be01dea-a053-4024-9e54-2585704425ad

________________________________________________________
Executed in   54.86 secs    fish           external
   usr time   45.06 secs  216.00 micros   45.06 secs
   sys time   21.40 secs  781.00 micros   21.40 secs

yassinesouilmi@yassines-ipro ~/D/parallelized_bwa> ll HG00513-unpaired-reads-corrupt-15Mreads-cut-damaged.canolli.output                                                                                (base)
total 288
-rw-r--r--  1 yassinesouilmi  staff     0B  6 Oct 00:52 _SUCCESS
-rw-r--r--  1 yassinesouilmi  staff    11K  6 Oct 00:52 part-r-00000.sam
-rw-r--r--  1 yassinesouilmi  staff    11K  6 Oct 00:52 part-r-00001.sam
-rw-r--r--  1 yassinesouilmi  staff    11K  6 Oct 00:52 part-r-00002.sam
-rw-r--r--  1 yassinesouilmi  staff    11K  6 Oct 00:52 part-r-00003.sam
-rw-r--r--  1 yassinesouilmi  staff    11K  6 Oct 00:52 part-r-00004.sam
-rw-r--r--  1 yassinesouilmi  staff    11K  6 Oct 00:52 part-r-00005.sam
-rw-r--r--  1 yassinesouilmi  staff    11K  6 Oct 00:52 part-r-00006.sam
-rw-r--r--  1 yassinesouilmi  staff    11K  6 Oct 00:52 part-r-00007.sam
-rw-r--r--  1 yassinesouilmi  staff    11K  6 Oct 00:52 part-r-00008.sam
-rw-r--r--  1 yassinesouilmi  staff    11K  6 Oct 00:52 part-r-00009.sam
-rw-r--r--  1 yassinesouilmi  staff    11K  6 Oct 00:52 part-r-00010.sam
-rw-r--r--  1 yassinesouilmi  staff    11K  6 Oct 00:52 part-r-00011.sam

I haven't tried using single end reads with Cannoli bwaMem before, but I imagine something like this might happen.
In ADAM, which Cannoli builds on, single end reads can be Reads or unaligned Alignments.
Paired end reads are Fragments, pairs (or higher cardinality) of unaligned Alignments grouped by the DNA fragment that was targeted by the sequencer (read name, typically).
Schema are here in the bdg-formats repo
https://github.com/bigdatagenomics/bdg-formats/blob/master/src/main/resources/avro/bdg.avdl
Mapping flat file formats to these record types:

FASTA → Sequence
single end FASTQ → Read
pair of FASTQ files → Fragment
interleaved FASTQ file → Fragment
unaligned BAM/CRAM/SAM file → Fragment
aligned BAM/CRAM/SAM file → Alignment

Then tools wrapped in Cannoli look like functions, e.g. minimap2 via multiple classes currently supports paired end reads (pair of FASTQ files, interleaved FASTQ file, or unaligned BAM/CRAM/SAM file)

Minimap2 is a function mapping FragmentDataset → AlignmentDataset

single end reads (single end FASTQ file)

SingleEndMinimap2 is a function mapping ReadDataset → AlignmentDataset

and long reads (FASTA file)

LongMinimap2 is a function mapping SequenceDataset → AlignmentDataset

There currently is no similar SingleEndBwaMem to go from single end reads to alignments, only the present one that expects paired end reads.  If that is your use case, we can add such fairly easily.
Hope this helps, sorry for being so long-winded!

 && tail CEUTrio.HiSeq.WGS.b37.NA12878.20.21.tiny.unaligned_1.sam
@HD	VN:1.6	SO:unsorted
@SQ	SN:1	LN:249250621	M5:1B22B98CDEB4A9304CB5D48026A85128	UR:file:/data/hs37d5.fa
@SQ	SN:2	LN:243199373	M5:A0D9851DA00400DEC1098A9255AC712E	UR:file:/data/hs37d5.fa
@SQ	SN:3	LN:198022430	M5:FDFD811849CC2FADEBC929BB925902E5	UR:file:/data/hs37d5.fa
@SQ	SN:4	LN:191154276	M5:23DCCD106897542AD87D2765D28A19A1	UR:file:/data/hs37d5.fa
@SQ	SN:5	LN:180915260	M5:0740173DB9FFD264D728F32784845CD7	UR:file:/data/hs37d5.fa
...
20FUKAAXX100202:2:23:9749:11047	16	20	10138382	60	101M	*	0	0	CCAGGCATTTCACTGTTAATTATCAGCAGCCTGGCTGCTATGGCACCTTGCTGTGCCAATTTGAGAGATGAGATTCTTTACACTGCTTATTAGGCTCAGCT	HHIHFEIHHHHGHHHHHHHHHFHHHHHHHHHHHHHHHHHHHHHHGHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH	LB:Z:HG00513	MD:Z:101	RG:Z:HG00513	NM:i:0	AS:i:101	XS:i:0	PU:Z:0
20FUKAAXX100202:2:23:9792:150347	16	1	142893226	0	101M	*	0	0	TTTTGGATTCTTGATTTATTGGTAAAGAAATACAGTTATTTTTGTATACCAATCTTATATAGTGTTACATTCTTAAGTTTGTTCATGAGTCCTAATACTTT	HHHHHHHHHHHHHHHHHHHHHHGGHHHHIHHHHHGGGGGHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH	LB:Z:HG00513	MD:Z:101	RG:Z:HG00513NM:i:0	AS:i:101	XS:i:101	PU:Z:0

That pull request may need additional cleanup before merging, and I may want to add single end bwa-mem2 while I'm at it.  Please let me know if this works for you!

I'm having build issues this time around, I am running maven v3.8.3 and jdk v1.8.0_152-release, but when trying to install I get the following error:
> mvn clean install                                                                                (base)
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Cannoli_2.12                                                       [pom]
[INFO] Cannoli_2.12: Builder                                              [jar]
[INFO] Cannoli_2.12: Core                                                 [jar]
[INFO] Cannoli_2.12: CLI                                                  [jar]
[INFO] Cannoli_2.12: Assembly                                             [jar]
[INFO]
[INFO] ---------< org.bdgenomics.cannoli:cannoli-parent-spark3_2.12 >----------
[INFO] Building Cannoli_2.12 0.15.0-SNAPSHOT                              [1/5]
[INFO] --------------------------------[ pom ]---------------------------------
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/3.1.0/maven-clean-plugin-3.1.0.pom
...
Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-inject-bean/1.4.2/sisu-inject-bean-1.4.2.jar
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.sonatype.oss</groupId>
    <artifactId>oss-parent</artifactId>
    <version>9</version>
  </parent>

  <groupId>org.bdgenomics.cannoli</groupId>
  <artifactId>cannoli-parent-spark3_2.12</artifactId>
  <version>0.15.0-SNAPSHOT</version>
  <packaging>pom</packaging>
  <name>Cannoli_${scala.version.prefix}</name>
  <description>Big Data Genomics ADAM Pipe API wrappers for bioinformatics tools.</description>
  <url>https://github.com/bigdatagenomics/cannoli</url>
  <inceptionYear>2017</inceptionYear>

  <properties>
    <adam.version>0.36.0</adam.version>
    <scala.version>2.12.10</scala.version>
    <scala.version.prefix>2.12</scala.version.prefix>
    <spark.version>3.1.2</spark.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.build.targetEncoding>UTF-8</project.build.targetEncoding>
Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.0.4/plexus-utils-2.0.4.jar (222 kB at 180 kB/s)
...
Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7-noaop.jar (472 kB at 238 kB/s)
[INFO]
[INFO] --- maven-enforcer-plugin:1.2:enforce (enforce-maven) @ cannoli-parent-spark3_2.12 ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.2:enforce (enforce-java) @ cannoli-parent-spark3_2.12 ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Cannoli_2.12 0.15.0-SNAPSHOT:
[INFO]
[INFO] Cannoli_2.12 ....................................... FAILURE [ 15.649 s]
[INFO] Cannoli_2.12: Builder .............................. SKIPPED
[INFO] Cannoli_2.12: Core ................................. SKIPPED
[INFO] Cannoli_2.12: CLI .................................. SKIPPED
[INFO] Cannoli_2.12: Assembly ............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15.763 s
[INFO] Finished at: 2021-10-07T11:04:56+10:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.2:enforce (enforce-java) on project cannoli-parent-spark3_2.12: Execution enforce-java of goal org.apache.maven.plugins:maven-enforcer-plugin:1.2:enforce failed: An API incompatibility was encountered while executing org.apache.maven.plugins:maven-enforcer-plugin:1.2:enforce: java.lang.ExceptionInInitializerError: null
[ERROR] -----------------------------------------------------
[ERROR] realm =    plugin>org.apache.maven.plugins:maven-enforcer-plugin:1.2
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/Users/yassinesouilmi/.m2/repository/org/apache/maven/plugins/maven-enforcer-plugin/1.2/maven-enforcer-plugin-1.2.jar
[ERROR] urls[1] = file:/Users/yassinesouilmi/.m2/repository/org/apache/maven/reporting/maven-reporting-api/2.0.9/maven-reporting-api-2.0.9.jar
[ERROR] urls[2] = file:/Users/yassinesouilmi/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.0-alpha-10/doxia-sink-api-1.0-alpha-10.jar
[ERROR] urls[3] = file:/Users/yassinesouilmi/.m2/repository/commons-cli/commons-cli/1.0/commons-cli-1.0.jar
[ERROR] urls[4] = file:/Users/yassinesouilmi/.m2/repository/org/codehaus/plexus/plexus-interactivity-api/1.0-alpha-4/plexus-interactivity-api-1.0-alpha-4.jar
[ERROR] urls[5] = file:/Users/yassinesouilmi/.m2/repository/org/codehaus/plexus/plexus-utils/1.5.8/plexus-utils-1.5.8.jar
[ERROR] urls[6] = file:/Users/yassinesouilmi/.m2/repository/commons-lang/commons-lang/2.3/commons-lang-2.3.jar
[ERROR] urls[7] = file:/Users/yassinesouilmi/.m2/repository/org/apache/maven/enforcer/enforcer-api/1.2/enforcer-api-1.2.jar
[ERROR] urls[8] = file:/Users/yassinesouilmi/.m2/repository/org/apache/maven/enforcer/enforcer-rules/1.2/enforcer-rules-1.2.jar
[ERROR] urls[9] = file:/Users/yassinesouilmi/.m2/repository/org/apache/maven/shared/maven-common-artifact-filters/1.2/maven-common-artifact-filters-1.2.jar
[ERROR] urls[10] = file:/Users/yassinesouilmi/.m2/repository/org/beanshell/bsh/2.0b4/bsh-2.0b4.jar
[ERROR] urls[11] = file:/Users/yassinesouilmi/.m2/repository/org/apache/maven/shared/maven-dependency-tree/2.0/maven-dependency-tree-2.0.jar
[ERROR] urls[12] = file:/Users/yassinesouilmi/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.jar
[ERROR] urls[13] = file:/Users/yassinesouilmi/.m2/repository/org/codehaus/plexus/plexus-i18n/1.0-beta-6/plexus-i18n-1.0-beta-6.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import  from realm ClassRealm[maven.api, parent: null]]
[ERROR]
[ERROR] -----------------------------------------------------
[ERROR] : begin 0, end 3, length 2
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginContainerException

Not sure what I'm doing wrong here.
Thanks a lot for your help!
Yassine

Running this on Mac, I had to do the following:

Install JDK using home-brew, the conda version didn't cut it for some reason:

$ brew install --cask AdoptOpenJDK/openjdk/adoptopenjdk8

Install jenv to select the java version:

$ brew install jenv

based on which shell you're running, initialise jenv using the following documentation: https://www.jenv.be

then select jdk 8:

$ jenv add /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home

$ jenv global 1.8

This did the trick for me.
Thanks!
Yassine