Tweak wildcard signatures for non-useful type parameters

See https://github.com/ceylon/ceylon/issues/1617#issuecomment-232073764 for the start of the discussion.

Alright, I just ran into the problem with Collectors.asList() myself. We need to fix this ASAP, because it's going to affect almost anyone who tries to use Java's streams API.

I think changing the signature of this method is going to impact anyone implementing this API, no?

To recap, we'd change:
interface Stream<T>{
<R,A> R collect(Collector<? super T,A,R> collector);
}

Signatures into:
interface Stream<T>{
<R,A> R collect(Collector<? super T,?,R> collector);
}

I have to check that this doesn't apply to many existing methods.

The problem of doing this in the model loader is that it changes the signature for call-sites and for declarations, so it's rather heavy-handed.
Do I also change signatures like <A> void foo(A) into <A> void foo(?)? Or only if the type parameter is a type argument?

Yeah, I think that's right. The idea is: if the type param only occurs in one location in the parameter list, and never in the return type, it can be replaced by a wildcard.
@FroMage if this looks hard to do in the backend, I will take a look at doing it in the typechecker.

Do I also change signatures like <A> void foo(A) into <A> void foo(?)?

Sure, I suppose so.

Or only if the type parameter is a type argument?

WDYM?

I will take a look at doing it in the typechecker.

FTR, the reason it doesn't really belong in the typechecker is that this (wildcard capture) isn't a sound thing to do in Ceylon, because we have reified type args. Furthermore, in full generality, it's a significantly more complex thing than what we're proposing here, and we need to be careful about certain undecidabilities that Ross found.
So I still think it's better to just pre-process the signatures of certain badly-written Java methods.

I understand. Just saying the typechecker can do this only for inference. Where the model loader applies it to declarations, and thus causes issues when trying to implement the method too. I don't think it's difficult to do it in the model loader, I'm just worried of potential far-reaching consequences. Like, I wonder how many existing methods this will en up affecting.

Still, I'll try and see where this leads.
This is just for Java methods or also Ceylon methods?

Only for Java methods.

OK

FTR: using those substitution rules only on Collectors yields these replacements:
Substituting given Stream<T>.collect<R,A>.A from function Stream<T>.collect<R,A>(Collector<in T,A,R>? arg0) => R
Substituting given Collectors.mapping<T,U,A,R>.A from function Collectors.mapping<T,U,A,R>(Function<in T,out U>? arg0, Collector<in U,A,R>? arg1) => Collector<T,out Object,R>
Substituting given Collectors.mapping<T,U,A,R>.A from function Collectors.mapping<T,U,A,R>(U(T)? arg0, Collector<in U,A,R>? arg1) => Collector<T,out Object,R>
Substituting given Collectors.mapping<T,U,A,R>.A from function Collectors.mapping<T,U,A,R>(Function<in T,out U>? arg0, Collector<in U,A,R>? arg1) => Collector<T,out Object,R>
Substituting given Collectors.groupingBy<T,K,A,D>.A from function Collectors.groupingBy<T,K,A,D>(Function<in T,out K>? arg0, Collector<in T,A,D>? arg1) => Collector<T,out Object,Map<K,D>>
Substituting given Collectors.groupingBy<T,K,A,D>.A from function Collectors.groupingBy<T,K,A,D>(K(T)? arg0, Collector<in T,A,D>? arg1) => Collector<T,out Object,Map<K,D>>
Substituting given Collectors.groupingBy<T,K,D,A,M>.K from function Collectors.groupingBy<T,K,D,A,M>(Function<in T,out K>? arg0, Supplier<M>? arg1, Collector<in T,A,D>? arg2) => Collector<T,out Object,M>
Substituting given Collectors.groupingBy<T,K,D,A,M>.D from function Collectors.groupingBy<T,K,D,A,M>(Function<in T,out K>? arg0, Supplier<M>? arg1, Collector<in T,A,D>? arg2) => Collector<T,out Object,M>
Substituting given Collectors.groupingBy<T,K,D,A,M>.A from function Collectors.groupingBy<T,K,D,A,M>(Function<in T,out K>? arg0, Supplier<M>? arg1, Collector<in T,A,D>? arg2) => Collector<T,out Object,M>
Substituting given Collectors.groupingBy<T,K,D,A,M>.K from function Collectors.groupingBy<T,K,D,A,M>(K(*Tuple<T,T,[]>)? arg0, M()? arg1, Collector<in T,A,D>? arg2) => Collector<T,out Object,M>
Substituting given Collectors.groupingBy<T,K,D,A,M>.D from function Collectors.groupingBy<T,K,D,A,M>(K(*Tuple<T,T,[]>)? arg0, M()? arg1, Collector<in T,A,D>? arg2) => Collector<T,out Object,M>
Substituting given Collectors.groupingBy<T,K,D,A,M>.A from function Collectors.groupingBy<T,K,D,A,M>(K(*Tuple<T,T,[]>)? arg0, M()? arg1, Collector<in T,A,D>? arg2) => Collector<T,out Object,M>
Substituting given Collectors.groupingByConcurrent<T,K,A,D>.A from function Collectors.groupingByConcurrent<T,K,A,D>(Function<in T,out K>? arg0, Collector<in T,A,D>? arg1) => Collector<T,out Object,ConcurrentMap<K,D>>
Substituting given Collectors.groupingByConcurrent<T,K,A,D>.A from function Collectors.groupingByConcurrent<T,K,A,D>(K(T)? arg0, Collector<in T,A,D>? arg1) => Collector<T,out Object,ConcurrentMap<K,D>>
Substituting given Collectors.groupingByConcurrent<T,K,A,D,M>.K from function Collectors.groupingByConcurrent<T,K,A,D,M>(Function<in T,out K>? arg0, Supplier<M>? arg1, Collector<in T,A,D>? arg2) => Collector<T,out Object,M>
Substituting given Collectors.groupingByConcurrent<T,K,A,D,M>.A from function Collectors.groupingByConcurrent<T,K,A,D,M>(Function<in T,out K>? arg0, Supplier<M>? arg1, Collector<in T,A,D>? arg2) => Collector<T,out Object,M>
Substituting given Collectors.groupingByConcurrent<T,K,A,D,M>.D from function Collectors.groupingByConcurrent<T,K,A,D,M>(Function<in T,out K>? arg0, Supplier<M>? arg1, Collector<in T,A,D>? arg2) => Collector<T,out Object,M>
Substituting given Collectors.groupingByConcurrent<T,K,A,D,M>.K from function Collectors.groupingByConcurrent<T,K,A,D,M>(K(*Tuple<T,T,[]>)? arg0, M()? arg1, Collector<in T,A,D>? arg2) => Collector<T,out Object,M>
Substituting given Collectors.groupingByConcurrent<T,K,A,D,M>.A from function Collectors.groupingByConcurrent<T,K,A,D,M>(K(*Tuple<T,T,[]>)? arg0, M()? arg1, Collector<in T,A,D>? arg2) => Collector<T,out Object,M>
Substituting given Collectors.groupingByConcurrent<T,K,A,D,M>.D from function Collectors.groupingByConcurrent<T,K,A,D,M>(K(*Tuple<T,T,[]>)? arg0, M()? arg1, Collector<in T,A,D>? arg2) => Collector<T,out Object,M>
Substituting given Collectors.partitioningBy<T,D,A>.A from function Collectors.partitioningBy<T,D,A>(Predicate<in T>? arg0, Collector<in T,A,D>? arg1) => Collector<T,out Object,Map<Boolean,D>>
Substituting given Collectors.partitioningBy<T,D,A>.A from function Collectors.partitioningBy<T,D,A>(Boolean(T)? arg0, Collector<in T,A,D>? arg1) => Collector<T,out Object,Map<Boolean,D>>
Substituting given Collectors.toMap<T,K,U,M>.K from function Collectors.toMap<T,K,U,M>(Function<in T,out K>? arg0, Function<in T,out U>? arg1, BinaryOperator<U>? arg2, Supplier<M>? arg3) => Collector<T,out Object,M>
Substituting given Collectors.toMap<T,K,U,M>.K from function Collectors.toMap<T,K,U,M>(K(T)? arg0, U(T)? arg1, U(U, U)? arg2, M()? arg3) => Collector<T,out Object,M>
Substituting given Collectors.toConcurrentMap<T,K,U,M>.K from function Collectors.toConcurrentMap<T,K,U,M>(Function<in T,out K>? arg0, Function<in T,out U>? arg1, BinaryOperator<U>? arg2, Supplier<M>? arg3) => Collector<T,out Object,M>
Substituting given Collectors.toConcurrentMap<T,K,U,M>.K from function Collectors.toConcurrentMap<T,K,U,M>(K(T)? arg0, U(T)? arg1, U(U, U)? arg2, M()? arg3) => Collector<T,out Object,M>

Surprisingly, when compiling the Ceylon SDK only those come up:
Substituting given IoUtils.cast<I,O>.I from function IoUtils.cast<I,O>(IoFuture<I>? arg0, Class<O>? arg1) => IoFuture<out O>
Substituting given Collections.sort<T>.T from function Collections.sort<T>(List<T>? arg0) => Anything
Substituting given Collections.sort<T>.T from function Collections.sort<T>(List<T>? arg0) => Anything
Substituting given OptionMap.Builder.parse<T>.T from function OptionMap.Builder.parse<T>(Option<T>? arg0, String? arg1) => OptionMap.Builder
Substituting given OptionMap.Builder.parse<T>.T from function OptionMap.Builder.parse<T>(Option<T>? arg0, String? arg1, ClassLoader? arg2) => OptionMap.Builder
Substituting given OptionMap.Builder.parse<T>.T from function OptionMap.Builder.parse<T>(Option<T>? arg0, String? arg1) => OptionMap.Builder
Substituting given ChannelListeners.initiateTransfer<I,O>.I from function ChannelListeners.initiateTransfer<I,O>(I? arg0, O? arg1, Pool<ByteBuffer>? arg2) => Anything
Substituting given ChannelListeners.initiateTransfer<I,O>.O from function ChannelListeners.initiateTransfer<I,O>(I? arg0, O? arg1, Pool<ByteBuffer>? arg2) => Anything
Substituting given ChannelListeners.initiateTransfer<I,O>.I from function ChannelListeners.initiateTransfer<I,O>(I? arg0, O? arg1, Pooled<ByteBuffer>()? arg2) => Anything
Substituting given ChannelListeners.initiateTransfer<I,O>.O from function ChannelListeners.initiateTransfer<I,O>(I? arg0, O? arg1, Pooled<ByteBuffer>()? arg2) => Anything
Substituting given ChannelListeners.initiateTransfer<I,O>.I from function ChannelListeners.initiateTransfer<I,O>(I? arg0, O? arg1, Pool<ByteBuffer>? arg2) => Anything
Substituting given ChannelListeners.initiateTransfer<I,O>.O from function ChannelListeners.initiateTransfer<I,O>(I? arg0, O? arg1, Pool<ByteBuffer>? arg2) => Anything
Substituting given Arrays.copyOf<T,U>.U from function Arrays.copyOf<T,U>(ObjectArray<U>? arg0, Integer arg1, Class<out ObjectArray<T>>? arg2) => ObjectArray<T>
Substituting given Arrays.copyOfRange<T,U>.U from function Arrays.copyOfRange<T,U>(ObjectArray<U>? arg0, Integer arg1, Integer arg2, Class<out ObjectArray<T>>? arg3) => ObjectArray<T>

@FroMage I think whoever wrote the Collectors class doesn't understand how to use variance. ;-)

Actually, those look like abstraction methods, lemme skip them.

Ah no, same.

@FroMage I think it's wrong that Collections.sort<T>() occurs in the list.

Well, it's signature is void: it's in-place.

Oh, wait, no, that's wrong. There are two Collections.sort() and one of them indeed belongs on this list :-)

So this is good. Previously we could not have called Collections.sort() on a List<out Object>. Now we can.

Just let's be clear that if you try to implement one of those methods in Ceylon I wash my hands clear of the clusterfuck ;)
I've already explained that it's too much work doing that for fake method types.

Mmm, I have tests that fail because I applied those to T[]. I suppose I should avoid arrays, no?

Ah no, just type parameters that have bounds.

Ah no, just type parameters that have bounds.

For type parameters that have bounds you could use out UpperBound or in Nothing as the wildcard.

Nope, I run into issues in tests that spread values into <T> f(T...).

I'll just skip sequenced parameters.

I'll just skip sequenced parameters.

That's fine, probably.

Let's call this fixed then.

yay, thanks!

Reverted since it prevents us from implementing those types in Ceylon.

Reopening if you want to try this in the typechecker

Alright, refixed!


[Feature]: Change the twitter icon to the updated icon of twitter

Is your feature request related to a problem? Please describe.
The footer contains the social icons, where the twitter icon is outdated.

Describe the solution you'd like
I would like to update that to the new icon.

kindly assign it to me

@anamelahi go ahead


More not_null<> improvements, some general improvements

Add macros gsl_noreturn, gsl_nodiscard, gsl_deprecated() (https://github.com/martinmoene/gsl-lite/commit/66f62f8ddf77e218771aa2fc0e195e922c46a144)

CMake package now installs only <gsl/gsl-lite.hpp> by default (https://github.com/martinmoene/gsl-lite/commit/ff9bd3411adac51040db0b8b82863b0c22104c39)

Fix implicit conversion operators and operators -> and * of not_null<> (https://github.com/martinmoene/gsl-lite/commit/4c060db5100c873020142ecb8d11305c499f033f, https://github.com/martinmoene/gsl-lite/commit/dcc811e1c2829706152d39fce3dc802abcfd2b0b, https://github.com/martinmoene/gsl-lite/commit/9122aa558e3bfdc57269787ceab1cd8e34f12c6b, https://github.com/martinmoene/gsl-lite/commit/ae5ea3e136060c5aaa5d88214019cb2186aa9066)

not_null<> comparison operators are now transparent and use SFINAE (https://github.com/martinmoene/gsl-lite/commit/c67b5373090781528b49531195aa08e23d33641c)

Add configuration option gsl_CONFIG_NOT_NULL_TRANSPARENT_GET (https://github.com/martinmoene/gsl-lite/commit/a84560ef71816e4b44d5d8a18b34fbf071e23ee6)

Add deduction guides for not_null<> (https://github.com/martinmoene/gsl-lite/commit/7a6278305c8ff644736c5d798bc5591dc15e5051)

Add make_not_null() (https://github.com/martinmoene/gsl-lite/commit/042d4749891ccac2deca844d9d331b44c7b47e2d)

...and many minor fixes (https://github.com/martinmoene/gsl-lite/commit/c23820dab29bad198b909c754cd6ad6e9f3807ec, https://github.com/martinmoene/gsl-lite/commit/bb6532a75f1959b657e56855cef87d7ad21810bb, https://github.com/martinmoene/gsl-lite/commit/867ad12fb0489dafe34ef03f05cae06d460d8f7b, https://github.com/martinmoene/gsl-lite/commit/a8409320d0ea8451b70cbb84eae2cfb6a257b59e) and improvements (https://github.com/martinmoene/gsl-lite/commit/44812ef990342c999621f3e547e80f74125c5d36, https://github.com/martinmoene/gsl-lite/commit/75c5d15fe47ff6018bffd5450153e35b50210fe7, https://github.com/martinmoene/gsl-lite/commit/b81d806aeddc9d4b0c475241f92135e35b283b8b, https://github.com/martinmoene/gsl-lite/pull/202/commits/19a880e9ccf85424cb2f90bf980c683189c39dd6, https://github.com/martinmoene/gsl-lite/pull/202/commits/bdace076649fcb9170b442a073e5b4d19952ea73, https://github.com/martinmoene/gsl-lite/pull/202/commits/55769e3524138b2c7bce09ac17e08585dbf8d032).

I messed up and have written a few comments before I figured out how I can send a review. Sorry for the confusion!

No problem. Thanks for reviewing! :)

My main problem is that the relation of gsl_CONFIG_NOT_NULL_GET_BY_CONST_REF and gsl_CONFIG_NOT_NULL_TRANSPARENT_GET is not apparent from neither the documentation, nor the code. Can they be both set to 1? If so, what happens? If not, why don't we have an error? (Like with the contract violation configs)

The docs state that gsl_CONFIG_NOT_NULL_GET_BY_CONST_REF affects the return value of get() "and friends". The latter is the reason why the options can be combined; even with gsl_CONFIG_NOT_NULL_TRANSPARENT_GET=1, gsl_CONFIG_NOT_NULL_GET_BY_CONST_REF still affects the conversion operator (if function ref qualifiers are not available) and operator->().
I'll try to clarify this.

I'll try to clarify this.

I think operator ->() can safely return T const & in every case. That would leave the return value of get() (unless gsl_CONFIG_NOT_NULL_TRANSPARENT_GET=1) and the conversion operator (unless function ref qualifiers are available).
At this point, I'd like to reconsider the option of removing the option entirely, with both the pre-C++11 conversion operator and the legacy get() returning T const & unconditionally too. Could this possibly break any code?

At this point, I'd like to reconsider the option of removing the option gsl_CONFIG_NOT_NULL_GET_BY_CONST_REF entirely, with both the pre-C++11 conversion operator and the legacy get() returning T const & unconditionally too. Could this possibly break any code?

I think for 0.x, we should leave the return values as they were in the "default" case, then in 1.0, remove  gsl_CONFIG_NOT_NULL_GET_BY_CONST_REF, make gsl_CONFIG_NOT_NULL_TRANSPARENT_GET=1 the default, and if it's 0, then the code should behave as if gsl_CONFIG_NOT_NULL_GET_BY_CONST_REF=1 was defined. This way, we could drop lots of ifdefs in 1.0, have the most modern version as the default, but could  fall back to a more-or-less compatible older solution. What do you think?

Please do. remove_cv<> shouldn't be needed unless there is a compiler bug we need to work around.

I could not reproduce the problem; I think it might have happened when I was using U&&, and I just misremembered it persisting with U. So consider this resolved. :)
(Is there a way I can reply inline? I can't find a button I could use around your comment...)

I think for 0.x, we should leave the return values as they were in the "default" case, then in 1.0, remove gsl_CONFIG_NOT_NULL_GET_BY_CONST_REF, make gsl_CONFIG_NOT_NULL_TRANSPARENT_GET=1 the default, and if it's 0, then the code should behave as if gsl_CONFIG_NOT_NULL_GET_BY_CONST_REF=1 was defined. This way, we could drop lots of ifdefs in 1.0, have the most modern version as the default, but could fall back to a more-or-less compatible older solution. What do you think?

That is what I will do if I cannot convince myself that unconditional gsl_CONFIG_NOT_NULL_GET_BY_CONST_REF=1 cannot possibly break code.


NO-ISSUE: Add en-US locale to yargs

Adding this prevent yargs of using the machine locale.

@tiagobento yargs --help is used to write the the image-builder and run-script-if README files. By default yargs use the machine locale, so in my case it was en-BR, which re-writed the files using Portuguese.


[BEAM-1920] Add a Spark 2.x support in the Spark runner

Follow this checklist to help us incorporate your contribution quickly and easily:

[X] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[X] Each commit in the pull request should have a meaningful subject line and body.
[X] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[X] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[X] Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[X] If this contribution is large, please file an Apache Individual Contributor License Agreement.

R: @aviemzur
R: @amitsela
CC: @iemejia
CC: @peihe

This is a work in progress: I'm working to fix the failing tests.
I have some questions for you guys:

Is is better to rename package with spark2 instead of spark ?
I introduced a complete new runner as I was not able to run on both Spark 1.x and Spark 2.x with the same runner (the API changed). I think it's better to have two runners instead only to upgrade the existing one.

NB: this PR superseded #3739

I checked out this PR and looked at the diff between spark and spark2 directories - it is tiny, only 189 lines of diff (compared to 24371 lines total in spark2/). There has to be a better way to do this than copy-paste of 99.2% of the code. Can you extract a common module, e.g. runners/spark-common, and have spark and spark2 have minimal hooks on top of the common module?

Like I mentioned in #3739 the main difference (that affects compilation) is switching Iterables with Iterators in Spark's various functions, e.g., when extending SparkGroupAlsoByWindowViaOutputBufferFn implements FlatMapFunction or when applying to/from Pair (KV).
I agree with @jkff that we don't need a copy/paste of the runner module.
I guess we should try to use a common parent module (spark-runner-common ?) and have two sub-modules spark-runner1 and spark-runner2 each re-implementing ~20 files (maybe less).

I agree guys: it's what I plan to use a spark-common. I do it today.

Thanks! If any amount of copy-paste remains, I recommend to make it easier for review by splitting in 2+ commits: 1) commit that just copy-pastes the files with zero changes 2) commits that actually change the files.

Good idea. I will.

I also agree that such duplication of the code is dangerous and could lead to problems in the future.
IIRC the plan was we would use the same runner for both Spark 1.6.x and Spark 2.x utilizing something like SPeL (or other methods of reflection) to bridge the gaps where the API was broken. Instead of having to maintain two separate runners.
An example for where we do something like this today is KafkaIO - where we support multiple versions with conflicting APIs. we detect the version by finding specific method signatures using SPeL and call the correct one, abstracting this from away from KafkaIO using a class called ConsumerSPeL
Couldn't we create such a facade for the methods which interact with the APIs in Spark which have breaking changes between 1.6.x and 2.x?
Say the original method was part of the RDD class, and its signature is foo(Iterator i) vs foo(Iterable i).
Can we create a class RDDFacade that wraps an RDD and has method foo(Iterable i) which translates from Iterable to Iterator depending on the signature of the method on the delegate RDD instance at runtime?

Guys, don't get me wrong: my intention is not to have two duplicated runners. The first target is to have an unique runner for both Spark 1.x and Spark 2.x, and if I fail to do it, I will introduce a spark-common.
I pushed the PR to have more visibility to the users (as asked on the mailing list).
So, I agree with all your comments, and I will move forward as @aviemzur suggested first.
Stay tuned ;)

It's been about 1.5 months on this PR with no activity, and the original PR is taking a direction that, we have consensus, is not the right one. Would you mind closing this PR, and reopening a new one taking a better direction whenever it's ready?

We are working on a private branch where we gather both Spark 1.x and 2.x in the same runner. The PR will be updated soon.

Here's the update with:

rebased
unique runner addressing both Spark 1.x and Spark 2.x
profile to deal with dependencies

I'm doing the final touches and polishing.

@jkff @amitsela @aviemzur even if all the final polish has not been done, you can already take a look.

Oh nice! Can you elaborate on how the two profiles will work? Will we have continuous test coverage (unit tests and ValidatesRunner tests) both for Spark 1 and Spark 2? What should a user do to select which of the implementations is used in their pipeline?

Yeah, I will document a bit this in a README. Basically, the user can use -Pspark2 to enable Spark 2. For the itests, it's the same: running with both -Pvalidate-local-runner,spark2 will run with Spark2. The only small issue is that the profile is required: -Pspark1 or -Pspark2.
That's the part I would like to polish and document.

@staslev thanks for the comments ! Yeah, I tested with a local Spark 2 cluster. I'm preparing some docker images to do more loading tests.

Hm, not sure I understand how the profiles thing will work in practice.
For Beam contributors:

When I make a change to Beam and run the usual "mvn clean verify", will that run unit tests against Spark1 dependencies, or against Spark2 dependencies, or both?
If I say "Run Spark ValidatesRunner" on a PR: same question?
Which will the default Java precommit run?

For Beam users:

If I'm an external user and I specify a dependency on beam-runners-spark, will I get an artifact that supports launching jobs against only Spark1 or only against Spark2 or both? (an external user, of course, can not specify a build profile for Beam, because they are not building Beam, they are just fetching an already built artifact from the repo)

I think a better solution than profiles is something @echauchot did to support different versions of Elasticsearch.

Have a module beam-runners-spark-common containing almost all the current code of beam-runners-spark with a pom.xml that treats Spark deps as provided; beam-runners-spark (for Spark 1.x) that depends on -common and on Spark 1.x, beam-runners-spark2 (for Spark 2.x) that depends on -common and Spark 2.x. This way users can explicitly depend on either one or the other.

Craft the pom.xml's so that runners-spark1 will run unit tests of spark-common against spark1, and runners-spark2 will run unit tests of spark-common against spark2. This way "mvn clean verify" will run both.

Change Jenkins setup to refactor "Spark ValidatesRunner" into "Spark 1 ValidatesRunner" and "Spark 2 ValidatesRunner"

The purpose was to have a unique module as the code is exactly the same. We can have different modules, all code in common and just a pom with the dependencies set (no code). But it's basically a profile. I can draft a multi module layer but let me start with documenting profiles first.

I think before documenting this solution we need to make sure that this is the right solution, otherwise the documentation effort will be wasted work. To me it seems that the issues raised in my comment above are show-stoppers and warrant changing this to a multi-module implementation; am I missing something?

So having a module common with code and two modules just with a pom for dependencies is what you are proposing ? What's the pros comparing to profiles ?

Anyway I do what you propose but I think we can achieve the same just providing -Pspark1 or -Pspark2.

Per offline discussion, the plan is to do the multi-module approach.
Motivational quote from chat:
Hi. I feel like there is a misunderstanding about how profiles work - perhaps it is a misunderstanding on my side. Am I correct in thinking that, if I am an external user of Beam and I am simply specifying Beam as a dependency to be fetched from maven-central, then I can not specify -Pspark1 or -Pspark2?
AFAIK profiles are for the person who is building the project, not for the person who is using it as a dependency; you can not specify a  on "beam-runners-spark with -Pspark1"
And if I build my own pom.xml with -Pspark1, it will have no effect at all, or it will fail, because my pom.xml won't have a profile named "spark1"

Yeah as discussed it's more convenient for user. I'm doing the change now. Thanks.

I am not sure I follow the latest comments.
The profiles only add the 'provided' dependencies so for the end user they should not really do any different of what they already do (with the exception of putting the correct version of the spark dependencies). So one runner can cover in principle both versions and in that case we can just adapt the examples/tests/ci to do so. And with this cover all the cases listed by Eugene.
Also I am not sure if the comparison with ElasticsearchIO makes sense considering that there is only one implementation of the IO (not a common and two shared submodules). What Etienne did was to create two independent modules only for the tests of each version, is this what you propose @jkff ?
The 'big' advantage of having separated versions I see is if we want to include in the future a different TransformTranslator based on a different API (e.g. Structured Streaming) but at the time this will make sense Spark 1 will be already deprecated (Notice that Cloudera already started doing this).
Ping @amitsela to see if he has any preference on this ?

The idea is to simplify the experience for end users. I don't think that profiles approach is convenient for the user.
So basically, currently, I'm evaluating two approaches:

the user defines beam-spark-runner (unique) in his dependencies set in the pom.xml. Then, he will have to specify the Spark version related dependencies (or use a BoM).
the user has a choice of beam-spark1-runner and beam-spark2-runner acting as a BoM, providing a default coherent dependencies set.
I'm in favor of having:

common with the code (both base and test)
spark1 with a pom, basically a BoM with dependencies
spark2 with a pom, basically a BoM with dependencies
Thanks to that, we can step by step improve spark2 to smoothly add dataframe support.

By the way, I already started to refactore to use three modules.

@jkff has a point about profiles..
Even if dependencies are provided, the runner code compiles against whichever spark version it is told (but one has to be specified) - defaults to 1.6.
Assuming that we've handled all the "weldings" in the code, we still might end-up running a runner compiled against spark2 that has spark1 in the classpath pr vice-versa, and who knows what'll happen then..
I don't think it makes sense thinking about it like this, especially not when the alternative is simple and clean.
We'll have the entire (now adapted) source code is part of the beam-spark-runner-common module, and the actual runner modules only contain the pom.
It'll also be easier to keep code duplication to minimum, if eventually necessary.
Beam users will now depend on the spark-runner version that suits them, and contributors will trigger SparkValidatesRunner1/2, though we can think of an optimization here since we expect most (all?) changes to cross-version - have a "default" runner and the other only runs nightly? I guess we can argue about that 😄

@amitsela thanks for your feedback. I started to refactore to use multi-modules approach. I will update the PR soon, then, we can discuss about the default.

I'm resuming/finalizing the multi-modules split. I will update the PR soon.

I did the split, but actually, I'm wondering if we still need to support Spark 1. Most of the distributions now use Spark 2.x.
I propose to do a complete Spark 2 upgrade, removing the "multi-version" support.
@amitsela @jkff @aviemzur @iemejia thoughts ?

I am in a big +1 for only support Spark 2. The extra complexity of supporting both versions of the runner seems less valuable considering that Spark 1 has not released any fix in more than a year and most of the new things are going into version 2. Also The Big Data/Cloud distributions are already on Spark 2. So legacy Spark 1 users can just stay with Beam 2.2.0 and we can maybe add some fixes if needed there, and support the rest in the newer versions.

+1 for Spark2 only, although that might be better to do in beam-dev?

I prepared two branches:

A branch with both Spark 1.x and Spark 2.x support with three artifacts as we discussed (common, spark1, spark2)
A complete clean branch upgrading to Spark 2.x and dropping Spark 1.x support.
I will move the discussion & vote from the dev mailing list to the user mailing list, and, depending of the result, I will merge one of the two branches.

+1 on only supporting Spark 2.x

Following the vote on the mailing lists, I updated the PR with Spark 2.x update only (no support of Spark 1 anymore).

Please ping the thread when the PR is ready for review. I think you said on the mailing list that some tests are still pending? And, since this is a major change, please also run the ValidatesRunner tests and I think it'd be prudent to perform the typical manual release validation steps (quickstart, game examples) against a real Spark cluster.

Retest this please

Run Spark ValidatesRunner

Potential issue found:
17/11/27 23:02:23 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, sandbox-hdp.hortonworks.com, executor 1, partition 1, PROCESS_LOCAL, 4837 bytes)
17/11/27 23:02:23 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, sandbox-hdp.hortonworks.com, executor 1): java.lang.AbstractMethodError: org.apache.beam.runners.spark.translation.MultiDoFnFunction.call(Ljava/lang/Object;)Ljava/util/Iterator;
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336)
    at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1038)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1029)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:969)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1029)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:760)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I'm checking ...

@jbonofre maybe a new PR for this? this has too much noise already, and the Pr should be something like "upgrade Spark to Spark 2.1.0" or something, no ?

@amitsela fair enough, let me create a new PR replacing this one.

This PR is superseded by #4208


clean up "oper:" from oper capab names

https://github.com/oragono/oragono/blob/80bee85c454fd4c0c9fc71d05f054fbca3d64e7d/oragono.yaml#L516
We can just normalize this out during config parsing --- strings.TrimPrefix(capab, "oper:") on everything --- and then change all the references in the code to remove oper:.
What was the original purpose of this?

Original purpose of this was to match the charybdis style of oper capab names, e.g. https://github.com/charybdis-ircd/charybdis/blob/master/doc/reference.conf#L425
I'd be totally fine with this. Let's keep the strings.TrimPrefix(capab, "oper:") as a permanent thing with a comment in the code mentioning the original reasoning for the oper: prefixes on these capabs.


Build Postman Collection

This is a quick PR to add Postman Collection which will be available on the following URL - api.codex.storage/postman.json.
We also will add a link to it in the documentation - Codex API.
And some cosmetic changes

Remove unnecessary quotes
Default shell on Ubuntu is bash

LGTM


Error importing WKTPlot

Describe the bug
When attempting to import WKTPlot, I get the error shown in the screenshots section. It appears that I'm missing wktplot.common. I included a screenshot of the instillation directory to show that the file is missing the common folder.
To Reproduce
Steps to reproduce the behavior:

Open a new terminal
pip install WKTPlot
python3
from wktplot import WKTPlot
See the error below

Screenshots
Traceback (most recent call last):
  File "/Users/thomas/sdss-demo/roads.py", line 4, in <module>
    from wktplot import WKTPlot
  File "/Users/thomas/sdss-demo/venv/lib/python3.9/site-packages/wktplot/__init__.py", line 1, in <module>
    from .plots.standard import WKTPlot  # noqa: F401
  File "/Users/thomas/sdss-demo/venv/lib/python3.9/site-packages/wktplot/plots/standard.py", line 5, in <module>
    from wktplot.common.file_utils import get_random_string, sanitize_text
ModuleNotFoundError: No module named 'wktplot.common'

Desktop (please complete the following information):

OS: macOS
Version 11.6.2

Additional context
Add any other context about the problem here.

Thanks for creating this issue. Looks like the folder got skipped over when the package was build for some reason. I will look into this tonight.

This should be fixed from this latest PR - https://github.com/FuzzFoundation/WKTPlot/pull/30

Release v2.3.1 is now published with this fix: https://github.com/FuzzFoundation/WKTPlot/releases/tag/v2.3.1


Add new machine-learning free color correction

It turns out, the color correction detection from colour science sometimes has troubles with detecting half colorcheckers (although these are the only ones used in the calibration). To mitigate this a new color correction routine has been added, in which the ROI corresponding to the color checker has to be specified by providing the pixel coordinates of the white L marks on the color checker. The remaining routine follows the same idea, however is much simpler, as it does not utilize a EOTF transformation, but only works in linear RGB space.
The example has been extended to showcase the use of the new color correction routine. The results after all are slightly different. There are several reasons to this: The lack of use of EOTF. And in fact, the machine learning based algorithms detect colours with RGB values outside the valid range of values in [0,1]. So there is quite some discrepancy, which is also visible in the final result. My current feeling is therefore, one may have to be a bit careful with the use of the automatic color correction. Instead one should possibly use the manual one. Setting it up is not much more tedious then defining a coarse ROI continaing the colorchecker.

I think it looks good, but it seems slightly strange to have two colorcorrections. Is your sense that the manual one simply is better than the old one? It looks like they both need the same type of input to work properly? Is it situational which one works better? We should probably give some indication as to which one should be used when.
I would always use the non-manual one just from the name of them, because it seems more likely that I would make a mistake myself in using a manual one than an automatic one.

I agree with you. Two do not look to good. I think experience will have to show. This is the thing with images. Automatic things may not be robust on all inputs. And the currently automatic one fails. It throws an error that it does not even find the color checker sometimes. And I am actually also not too happy with the performance of the color matching routine. It goes via EOTF (nonlinear RGB space), and through these mappings often colors are outside the valid range [0, 255].
Setup for the manual is slightly dfferent. It requires four coordinates instead of a ROI provided as tuple of slices. You choose what is more straightforward for non-experts ;)

Let's have a short discussion on naming etc. before merging.


[Button] add support for moving between icon button to end/start icon

Duplicates

[X] I have searched the existing issues

Latest version

[X] I have tested the latest version

Summary 💡
I'd love to get some help with this problem. when i building a responsive ui, and want the icon to be only icon without text on small screen, i need to build my own component. i think it will be usefull for lot of pepole. thank you!
Examples 🌈
<IconButton>
   <Search />
</IconButton>
...
<Button endIcon={<Search />}>
   Search
<Button />

my solution:
const matches = useMediaQuery("...")
...
<Button endIcon={<Search/>} showText={matches}>
    Search
</Button>

Motivation 🔦
No response

You can render either Button or IconButton conditionally, based on the useMediaQuery result:
const isWideScreen = useMediaQuery("(min-width:600px)");

return isWideScreen ? (
  <Button endIcon={<Search />} color="primary">
    Search
  </Button>
) : (
  <IconButton color="primary">
    <Search />
  </IconButton>
);


Plugin 121 BusSharingMode. Fixed bug where it would mess up plugins that came after it

This plugin changes the $VM variable, thereby rendering it unusable to plugins that are executed after it. I've simply renamed $VM to $Object

This is a good catch; recommend you open an issue and link your PR to it to get more visibility.
Also, PRs should be submitted against the "dev" branch not master. They won't accept against master so you'll have to re-submit.


Onchain permissioning deprecation warnings on CLI

PR description

Add deprecation messages to help descriptions
Add warning log if permissioning-*-contract-enabled option is specified

Note I did not add any @Deprecation annotations to types or methods - feedback welcome on whether you think this is needed.
Thanks for sending a pull request! Have you done the following?

[ ] Checked out our contribution guidelines?
[ ] Considered documentation and added the doc-change-required label to this PR if updates are required.
[ ] Considered the changelog and included an update if required.
[ ] For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

[ ] unit tests: ./gradlew build
[ ] acceptance tests: ./gradlew acceptanceTest
[ ] integration tests: ./gradlew integrationTest
[ ] reference tests: ./gradlew ethereum:referenceTests:referenceTests

I think the @Deprecation is useful if the type or method is public and can be used by third party like in the plugin API

I think the @Deprecation is useful if the type or method is public and can be used by third party like in the plugin API

yeah I went with adding it to API methods and plugins


Multiple issues with $validate() and $reset()

I noticed multiple issues with $validate() and $reset() method. I will attach a screencast so it is easy to see.
I have a form with both client and server validations.

When I first call $validate() then I see that all rules are triggered properly and all error messages appear ✅
Then I call $reset() and I expected all validations to reset and all error messages to disappear. But something strange happened. Only the server error disappeared. All other error messages stayed.
Reproduction:
stackblitz.com/~/github.com/martinszeltins/regle-test-9
Screencast:
https://github.com/user-attachments/assets/713e5af0-e72f-45c1-a7b9-1c9fa12a1862

That's indeed weird! I'll look

There was indeed a bug on reset with externalErrors, fixing and adding the case to the tests

So I followed this behaviour: by default, touching a field clears it's external errors (at the source).

If you have external errors from the start, they will be cleared in the first $validate(), because it also touch the fields.
But that's not a problem, because server errors should be set after a async call, after the $validate
Then errors are shown
$reset is called, external errors are cleared

I fixed the edge case that you had, and added tests for it. Try 0.2.9 👍

@victorgarciaesgi oh yes, I should set server errors after client-side validation passes. Thanks for the release! 🎉


Update outdated matlab-scripts.hml

The page still indicates to use mlabwrap
The project has not been updated since 2011
Matlab now supports binding to python

see https://github.com/ome/ome-documentation/pull/2013


Fix: Fixed issue with resolving paths containing ".."

Resolved / Related Issues
Items resolved / related issues by this PR.

Closes #9808

Details

First commit is the actual fix, others are a refactoring
I removed .Contains() from GetPathWithoutEnvironmentVariable() since they are not really needed: .Replace() won't do anything if the string isn't matched
I can't test if this still works for ftp

Validation
How did you test these changes?

[x] Built and ran the app
[ ] Tested the changes for accessibility

I found a better way to achieve the same thing, I'll open a different PR


UAR-101: who is filing update

JIRA link
https://companieshouse.atlassian.net/browse/UAR-101
Change description
New page to capture details of who is completing the update.
New controller/view used for new page. Existing whoIsRegistering model is used.
Existing who-is-filing view content put into a template so both existing and new view can use the same content without duplication.
Work checklist

[ ] Tests added where applicable
[ ] UI changes meet accessibility criteria

Merge instructions
We are committed to keeping commit history clean, consistent and linear.  A BREAKING CHANGE can be part of commits of any type,
types other than fix: and feat: are allowed, for example build:, chore:, ci:, docs:, style:, refactor:, perf:, test:, and others,
footers other than BREAKING CHANGE: <description> may be provided.

looks ok but will need to resolve conflicts and get sonar report before approving


Dew Point

Does the BOM API expose Dew Point?

If you see dew point in the bom app itself or on their website it should be exposed by the API. I remember last year for a few weeks some on the BOM sensors were actually broken.


Disappearing header with scroll

Describe the bug
When the list of logs is long and you scroll to the bottom of the list, the top header with the clear icon gets scrolled out of view along with it. The user needs to scroll all the way to the top to clear the list.
To Reproduce
Steps to reproduce the behavior:

Execute  a bunch of console actions.
Scroll down to the bottom of the list

Expected behavior
Possible fixes

Fixed header in the floating window.
A dedicated floating clear button, which is always accessible.

Screenshots

I'll make the header stick to top. - Thanks Pratyush!

@pratyushok Fixed with the current release (v1.1.2).


remove obsolete qemu patch

since qemu version 7.1.1 is used in vertical builds, this patch became obsolete
Tracked-On: OAM-105834
Signed-off-by: Chenthati, Pradeep<EMAIL_ADDRESS>

reviewers: @basilchew @bnkoppad Please review


source_id registration of NorESM2-LM

"NorESM2-LM":{
"aerosol":"OsloAero",
"atmosphere":"CAM-OSLO (2 degree resolution)",
"atmospheric_chemistry":"OsloChemSimp",
"cohort":[
"CMIP6"
],
"institution_id":[
"NCC"
],
"label":"NorESM2-LM",
"label_extended":"NorESM2-LM (low atmosphere-medium ocean resolution, GHG concentration driven)", (This can be expanded - no char limitation)
"land_ice":"CISM",
"land_surface":"CLM",
"ocean":"MICOM (1 degree resolution)",
"ocean_biogeochemistry":"HAMOCC",
"release_year":"2018",
"sea_ice":"CICE",
"source_id":"NorESM2-LM"
}

@IngoBethke take a peek at CMIP6_source_id.html and let me know if there are any further tweaks.. You'll note that all the atmosphere components have the suffix ; XX x XX; ? levels; top level ?) and the ocean components have ; XX x XX; ? levels; top grid cell ?) - when you finalise the component version info you can also update these entries..
The reason we are catching all this information in the CMIP6_CVs repo, is that CMOR (if you are using it to write your data) with automagically include this information in your files without you having to declare this info.. We are trying to automate and minimize many of the config steps for users

Hello Paul
Thank you for adding the information to the table.
I have a few additions/corrections:
Release year of NorESM2-LM should be corrected to 2017.
Atmosphere entry for configurations LM, LME and LMEC should be set to “CAM-OSLO (2 degree resolution; 144 x 96; 32 levels; top level at 3 mb)”.
Atmosphere entry for configurations MM and MH should be set to “CAM-OSLO (1 degree resolution; 288 x 192; 32 levels; top level at 3 mb)”.
Atmosphere entry for configuration HH should be set to “CAM-OSLO (1 degree resolution; 1152 x 768; 32 levels; top level at 3 mb)”.
Ocean entry for configurations LM, LME, LMEC and MM should be set to “MICOM (1 degree resolution; 360 x 384; 53 levels; top grid cell minimum 10 m)”.
Ocean entry for configurations MH and HH should be set to “MICOM (0.25 degree resolution; 1440 x 1152; 53 levels; top grid cell minimum 10 m)”.
Cheers,
Ingo

On 29 Nov 2016, at 21:56, Paul J. Durack<EMAIL_ADDRESS>wrote:
@IngoBethke https://github.com/IngoBethke take a peek at CMIP6_source_id.html http://rawgit.com/WCRP-CMIP/CMIP6_CVs/master/src/CMIP6_source_id.html and let me know if there are any further tweaks.. You'll note that all the atmosphere components have the suffix ; XX x XX; ? levels; top level ?) and the ocean components have ; XX x XX; ? levels; top grid cell ?) - when you finalise the component version info you can also update these entries..
The reason we are catching all this information in the CMIP6_CVs repo, is that CMOR (if you are using it to write your data) with automagically include this information in your files without you having to declare this info.. We are trying to automate and minimize many of the config steps for users
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/WCRP-CMIP/CMIP6_CVs/issues/156#issuecomment-263696149, or mute the thread https://github.com/notifications/unsubscribe-auth/AOP92Dq_pXp4vLuZotZr0I7jMEPWZwf3ks5rDJF9gaJpZM4K8oI_.

@IngoBethke for the top grid cell minimum 10 m does that mean the top grid cell is 0-10 m with a centered value at 5 m? Or 0-20 m with a centered value at 10 m?

It is more complicated than that. Our ocean model uses a bulk mixed layer formulation with variable mixed layer depth, varying in time and space. The layer thickness of the top layer can exceed several 100 m but it cannot become smaller than 10 m (in which case the centered value would be 5 m).

On 30 Nov 2016, at 14:55, Paul J. Durack<EMAIL_ADDRESS>wrote:
@IngoBethke https://github.com/IngoBethke for the top grid cell minimum 10 m does that mean the top grid cell is 0-10 m with a centered value at 5 m? Or 0-20 m with a centered value at 10 m?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/WCRP-CMIP/CMIP6_CVs/issues/156#issuecomment-263878801, or mute the thread https://github.com/notifications/unsubscribe-auth/AOP92PcBRnyBC7lcvtDUShAzIG9Ob5oeks5rDYA9gaJpZM4K8oI_.

@IngoBethke ok interesting, it's a z-level model? So what values will be reported on the data that you will publish on ESGF, will the z-coordinate varying in time, or will these be interpolated back to standard 53 levels before publishing - if it's the latter then let's document the final format (with additional information in parentheses if you would like to include it, I'd recommend you do)

Our ocean component uses a vertical density coordinate. The density of the layers can, however, depart slightly from their pre-defined reference densities. The top layers, representing the bulk mixed layer, are not density constrained. So, the most correct is to say that our model uses a generic layer coordinate with layer thicknesses varying in space and time.
For CMIP5, groups were advised to online-interpolate to z-level in addition to writing output on native levels, which we did. We then cmor-ized both outputs, native on density and interpolated to z. CMIP5 only supported publication of one product, either on native levels or interpolated z-level. For the ease of the users we chose to publish the interpolated z-level output. We are probably going to do the same for CMIP6.

Ingo

On 30 Nov 2016, at 15:17, Paul J. Durack<EMAIL_ADDRESS>wrote:
@IngoBethke https://github.com/IngoBethke ok interesting, it's a z-level model? So what values will be reported on the data that you will publish on ESGF, will the z-coordinate varying in time, or will these be interpolated back to standard 53 levels before publishing - if it's the latter then let's document the final format (with additional information in parentheses if you would like to include it, I'd recommend you do)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/WCRP-CMIP/CMIP6_CVs/issues/156#issuecomment-263883841, or mute the thread https://github.com/notifications/unsubscribe-auth/AOP92EloXvhsXCdRIP_5KBIKSob6CAHJks5rDYVpgaJpZM4K8oI_.

@IngoBethke ok this provides more context. The interpolated data that folks will get from the archive will be top grid cell 0-2.5 m so let's go with the text below, or tweak the below to something that you're happy with for the 2 model versions, or for each model version if you want that detail included (note I have included your 70 levels you mentioned above, rather than the 53 levels in the current text):
MICOM (0.25 degree resolution; 1440 x 1152; 70 levels; top grid cell minimum 0-2.5 m [native
model uses hybrid density and generic upper-layer coordinate interpolated to z-level for
contributed data])

Your suggestion reads very well. Thanks again for putting time and thought into it.

On 30 Nov 2016, at 16:00, Paul J. Durack<EMAIL_ADDRESS>wrote:
@IngoBethke https://github.com/IngoBethke ok this provides more context. The interpolated data that folks will get from the archive will be top grid cell 0-2.5 m so let's go with the text below, or tweak the below to something that you're happy with for the 2 model versions, or for each model version if you want that detail included (note I have included your 70 levels you mentioned above, rather than the 53 levels in the current text):
MICOM (0.25 degree resolution; 1440 x 1152; 70 levels; top grid cell minimum 0-2.5 m [native
model uses hybrid density and generic upper-layer coordinate interpolated to z-level for
contributed data])
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/WCRP-CMIP/CMIP6_CVs/issues/156#issuecomment-263894608, or mute the thread https://github.com/notifications/unsubscribe-auth/AOP92O9ioDsKkt_urXsZ9ii0ETfqaJboks5rDY-EgaJpZM4K8oI_.

For CMIP6 we allow multiple horizontal grids, but I don't think we can easily accommodate multiple vertical grids, so it would be good to do as in CMIP5 and decide on one vertical grid.  [It would be possible, but possibly confusing, to report some variables on one vertical grid and others on another vertical grid.]

Thanks for advise. We will proceed as in CMIP5 then wrt the vertical coordinate. Concerning the horizontal grid, is the recommendation to provide both, cmor-ized output on the native grid (e.g. tripolar) and interpolated to regular grid?

On 30 Nov 2016, at 17:57, taylor13<EMAIL_ADDRESS>wrote:
For CMIP6 we allow multiple horizontal grids, but I don't think we can easily accommodate multiple vertical grids, so it would be good to do as in CMIP5 and decide on one vertical grid. [It would be possible, but possibly confusing, to report some variables on one vertical grid and others on another vertical grid.]
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/WCRP-CMIP/CMIP6_CVs/issues/156#issuecomment-263929171, or mute the thread https://github.com/notifications/unsubscribe-auth/AOP92GdL-e4fQ72GQh53waysvr-4n4b-ks5rDar1gaJpZM4K8oI_.

@IngoBethke just saw another typo:
Atmosphere entry for configuration HH should be set to
“CAM-OSLO (1 degree resolution; 1152 x 768; 32 levels; top level at 3 mb)”.
Should actually be:
“CAM-OSLO (0.25 degree resolution; 1152 x 768; 32 levels; top level at 3 mb)”.
Please peek CMIP6_source_id.html just to confirm all final changes (at this stage)

You are right, it should be 0.25 degree atmospheric resolution for HH. I've check the table and couldn’t spot further mistakes.

On 30 Nov 2016, at 19:33, Paul J. Durack<EMAIL_ADDRESS>wrote:
@IngoBethke https://github.com/IngoBethke just saw another typo:
Atmosphere entry for configuration HH should be set to
“CAM-OSLO (1 degree resolution; 1152 x 768; 32 levels; top level at 3 mb)”.
Should actually be:
“CAM-OSLO (0.25 degree resolution; 1152 x 768; 32 levels; top level at 3 mb)”.
Please peek CMIP6_source_id.html http://rawgit.com/WCRP-CMIP/CMIP6_CVs/master/src/CMIP6_source_id.html just to confirm all final changes (at this stage)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/WCRP-CMIP/CMIP6_CVs/issues/156#issuecomment-263955009, or mute the thread https://github.com/notifications/unsubscribe-auth/AOP92JSEn-SNKFBZYw2OT1qbpWcWs7f8ks5rDcFegaJpZM4K8oI_.

@IngoBethke ok great, this issue is closed.. As those version numbers start to lock in, please just open up a new issue so we can finalise all the model config details and get this all done and dusted


missing semi-colon is handled differently in version 5.0.1

Environment
System:

OS: macOS 10.15.3
CPU: (4) x64 Intel(R) Core(TM) i7-7660U CPU @ 2.50GHz
Memory: 403.66 MB / 16.00 GB
Shell: 5.7.1 - /bin/zsh

Binaries:

Node: 12.14.1 - /usr/local/bin/node
Yarn: 1.10.1 - /usr/local/bin/yarn
npm: 6.13.4 - /usr/local/bin/npm

npmPackages:

styled-components: 5.0.1 => 5.0.1

Reproduction
v5.0.1: https://codesandbox.io/s/upbeat-lederberg-7l9yz ❌
v4.4.1: https://codesandbox.io/s/jovial-newton-rutj6 ✅
Steps to reproduce

style a component and omit the semi-colon in the template string eg:

styled.div`
  min-width: ${({ minWidth }) => (minWidth ? `${minWidth}px` : "")} // missing semi-colon here
`;

Expected Behavior
It can still parse the styles
Actual Behavior
It doesn't parse the styles correctly:
    min-width: 80px max-width:500px;

We've never supported omitting semicolons, if it worked before it was a fluke. styled-components only supports valid CSS, style objects, and interpolations.

v4 use stylis directly, and set semicolon: false, this means "autocomplete missing semicolons". But v5 use @emotion/stylis and set options.semicolon: true, this is a breaking change. It is recommended to write to ChangeLog.

Is there an eslint or ts rule that we can apply to find these automatically?

We've never supported omitting semicolons; if it worked before it was a fluke. styled-components only supports valid CSS, style objects, and interpolations.

How do you explain https://github.com/styled-components/styled-components/pull/1532 ?

@probablyup please reconsider this issue.

Is there an eslint or ts rule that we can apply to find these automatically?

If we use PostCSS and interpret it as a CSS AST, we would be able to statically parse it as incorrect CSS syntax, as like  this.
There is also postcss-styled-syntax, which handles css-in-js syntax like styled-components in PostCSS.
I feel this is the best way to detect the lack of semicolon, but I'd like to know if I'm missing a consideration 🤔


[WIP] openblas: fix aarch64 build

Motivation for this change
Attempt to fix the openblas breakage on aarch64
https://github.com/NixOS/nixpkgs/pull/56216
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

@GrahamcOfBorg build openblas

This fix is really ugly but it works now on arm and x86. Any suggestions?


Update controls-exposer.js

Add try catch.
This prevents zigbee2mqtt from crashing when pairing another new device

Hey! I added the extension to my z2mqtt but I dont see brightness in HA yet.. What am I missing?


Get Junctions or Bifurcation from 3D skeletons

Hello,
Is is possible to extract degrees of the junctions or bifurcation of 3D skeleton of the vessels?
I saw that this output has been deprecated in the recent releases on Skan.
So what should I do?
Many thanks !

Hi @dzeego!
Yes I'm in the process of updating the docs and also the API to make these things more obvious. But, specifically, class skan.Skeleton contains the degree information for every point of the skeleton. To find the coordinates of bifurcations, do skeleton.coordinates[skeleton.degrees > 2].
... And, I just noticed that there's no such thing as degrees in the documentation, even though it's in the object. 🤦 I'll leave this open until I update that.

Hi @jni !
Thank you very much for your reply ! Really helpful :)
I make use of the fact that this issue remains open to ask another question please:
Is it possible to extract the pre-skeletonized diameters (from the original image) directly from the skeleton.
To be more clear, I would like to directly find the different diameters along the skeleton. Is that possible?
Thanks again !

@dzeego you'll need to do some custom graph work for this. By default, skan does not have any notion of a skeleton "root" node, or bifurcations along a path — skeletons can contain cycles. For example in the original skan paper the skeletons measured are a mesh on the plane.
I think the new skan API I'm working on will let you do this more easily, but basically:

construct a networkx.Graph or networkx.MultiGraph from skan.summarize() node ID pairs
Define a node in the graph as the root
use networkx traversal algorithms to define an edge's distance (in number of hops) from the root
add that distance to the table
use the 2D display (skan.draw) or 3D display (see the documentation for displaying skeletons in 3D in napari) functions to display according to that distance.

I hope to update both the API and the docs shortly to make this custom analysis easier. Thanks for your patience!


build_essential should work without a resource name

We should setup a default name here like we did with apt_update.

      # allow bare apt_update with no name
      property :name, String, default: ""

Fixed by #7197


scrape: fix prometheus_target_scrape_pool_target_limit metric not set on creating scrape pool

fix issue #12001

Thank you. It is interesting as I was thinking we would run an ApplyConfig at startup.

Thank you. It is interesting as I was thinking we would run an ApplyConfig at startup.

I think so and very confuse me, util  i add debug log, then found ApplyConfig only invoked at config reload, not startup.

hi  @roidelapluie , please take a look, Thanks.


Scene.GetAttributeAtPath Causes a lot of GC allocation.

I have a scene with 21 prims, each with 2 attributes and approximately 600 time samples.
Calling Scene.GetAttributeAtPath($fullAttributePathAsString) seems to be allocating a lot of memory, causing up to 248B of GC Alloc each time it's called.
Here's a screenshot from the profiler

And here's the code snippet
string tag = "/Prim/Path";
pxr.UsdAttribute attr = scene.GetAttributeAtPath(tag + ".Data");

I Also noticed that replacing the Scene.Stage.GetAttributeAtPath($pathToAttributeAsSdfPath) results in much smaller allocation size (~72B in this case)

Adding @jcowles and @laurakasian, just FYI.

Yeah, I agree. We need to be careful about introducing a mix of string & SdfPath APIs into this class and try to keep it consistent.
If you REALLY want to be efficient here, I think the right solution is for the caller to cache the UsdAttribute and only call this method once, which will avoid all subsequent garbage generation.

In hindsight, I think exposing paths as a string was a bad idea -- all path arguments in USD.NET.Scene probably should have been SdfPath-typed.
It is convenient to use strings, but there are many ways to query SdfPaths from USD.NET.Scene and almost zero ways to actually use them (as method arguments).


Add mpi_mode in json

To add some plotting fixes for benchmark.py

I think you messed up with your branches, I see some skewing stuff here now

I think you messed up with your branches, I see some skewing stuff here now

Checkout out and pushed accidentally, fixed.

Now you have to do mpirun -n 2 but two threads plot together the same pic.

I don't think there's any point in plotting a roofline when using MPI.
One plots a scalability graph when using MPI, typically.

let me ask again: does this patch really work if you run not just without mpirun ..., but rather *without DEVITO_MPI= ... * ?
also: I see you've rebased the commit history, that's good. However, the commit message "Revert ..." doesn't make much sense

Now you have to do mpirun -n 2 but two threads plot together the same pic.

I don't think there's any point in plotting a roofline when using MPI.
One plots a scalability graph when using MPI, typically.

Yes, but we now aim to LinePlotter for scalability, not Roofline.

let me ask again: does this patch really work if you run not just without mpirun ..., but rather *without DEVITO_MPI= ... * ?
also: I see you've rebased the commit history, that's good. However, the commit message "Revert ..." doesn't make much sense

If you dont touch DEVITO_MPI and run python benchmark.py bench ....
we produce:
acoustic_arch[unknown]_shape[50,50,50]_nbpml[10]_tn[250]_so[2]_to[2]_dse[advanced]_dle[advanced]_at[aggressive]_nt[1]_np[1]_rank[0]
If you do mpirun -n 2 python benchmark.py bench -P  you get the message It seems that you're running over MPI with 2 processes, but DEVITO_MPI is unset. Setting DEVITO_MPI=basic...
you get 2 files:  acoustic_arch[unknown]_shape[50,50,50]_nbpml[10]_tn[250]_so[2]_to[2]_dse[advanced]_dle[advanced]_at[aggressive]_nt[1]_np[2]_rank[0]_mpi[basic]
and rank[1] as well

What I think I'm asking is whether the plotter is capable of parsing two different filenames -- one with _mpi[...] (longer) and the other without ?

yeah probably it's all good as long as one runs the plotter the same way bench was run (ie, both w/ or w/o DEVITO_MPI...). RIght?

SO just to sum up, what is needed, that can be probably be done in other PR is opescibench tweaking to parse according to the filename. This need emerges for one reason. RooflinePlotter and LinePlotter inherit from Plotter in general, and the one is needed for weak, while the other for strong scaling. SO the arguments of Roofline should be irrelevant to MPI, wihle this is not true for LinePlotter.

@FabioLuporini I was thinking of even removing np and rank from no-mpi-produced jsons. As long as we are going to use them form Rooflines we do not need this info.

I think perhaps the filename template could be fixed once and for all -- ie always with mpimode, mpirank, etc. Then it's up to whatever uses it decide what to do with the extrapolated information (so the RooflinePlotter could/will drop mpimode, mpirank etc on the floor ?)
what would you say?

I think perhaps the filename template could be fixed once and for all -- ie always with mpimode, mpirank, etc. Then it's up to whatever uses it decide what to do with the extrapolated information (so the RooflinePlotter could/will drop mpimode, mpirank etc on the floor ?)
what would you say?

Well, it could be. I will make these changes and then it could be merged, based on the hypothesis that Roofline does not care for nothing that has (rank>0 && np>1).

based on the hypothesis that Roofline does not care for nothing that has (rank>0 && np>1).

wdym by "doesn't care" exactly?
anyway, is this now good to go, basically?

@btrb has offered to double-check it but I think it is GTG now.

so, if i'm properly reproducing what's expected, i'm doing (for the MPI case, let's say) is:
DEVITO_MPI=1 DEVITO_OPENMP=1 mpirun -n 2 python benchmarks/user/benchmark.py bench -P acoustic -x 1
DEVITO_MPI=1 DEVITO_OPENMP=1 mpirun -n 2 python benchmarks/user/benchmark.py plot -P acoustic --max-bw 12.8 --flop-ceil 80 linpack

which returns the following img in devito/results:

i'm not sure if that makes any sense.. please let me know if i should apply any corrections

merged.


feat(travis): display emoji for Pending status

Encountered No emoji specified for status 'Pending' when using on_start: always:

notifications:
  webhooks:
    ...
    on_start: always    # default: never

Testing Plan:
GIFs or Screenshots:

A tangential issue: would it be OK to include the repository_slug in the MESSAGE_TEMPLATE?  Our organisation has almost 340 repos so it would be really helpful to be able to see which repo is involved, at a glance (without hovering over the URLs).
https://docs.travis-ci.com/user/notifications/#configuring-irc-notifications

repository_slug: your GitHub repo identifier (like svenfuchs/minimal)

Merged, since this change is great.
Re: repository_slug: Yes, definitely.  We might want to just use that in the topic, e.g. zulip/zulip builds, rather than putting it into the body, so that each repository gets its own thread naturally?

@timabbott Thanks for the merge.

Re: repository_slug: Yes, definitely. We might want to just use that in the topic, e.g. zulip/zulip builds, rather than putting it into the body, so that each repository gets its own thread naturally?

This is an interesting one because there are different use cases.  From our circumstances (still in progress, only started using Zulip in the last few days):

Our regular notifications will follow the pattern you've suggested (use it as the topic) and we're already achieving that by setting the topic in the URL:

e.g. https://xyz.zulipchat.com/api/v1/external/travis?api_key=key&stream=Travis&topic=zulip%2Fzulipbuilds

However, our second use case is that we have a weekly run across ~100 repos which would need to be grouped under one topic, such as &topic=2020-W21 -- this is where the repository_slug being immediately visible would be extremely helpful.

@timabbott Using ZT, was nice to see the changes come through... however, the emoji is not being rendered!

https://saltstack-formulas.zulipchat.com/#narrow/stream/239693-CI/topic/saltstack-formulas.2Fsalt-formula/near/198718260

This is the how I found the emoji:

https://pypi.org/project/emoji/, which linked to
https://www.webfx.com/tools/emoji-cheat-sheet/

It appears to be available here:

https://github.com/zulip/zulip/search?q=arrows_counterclockwise&unscoped_q=arrows_counterclockwise

Is there some additional step required to link this up properly?  Or is a collectstatic needed at your end?

Ahh, that emoji codepoint is named just counterclockwise in Zulip; can you submit a quick PR renaming it?

Ahh, that emoji codepoint is named just counterclockwise in Zulip; can you submit a quick PR renaming it?

@timabbott In case you haven't seen it, the fix is at #15110.


ci: change to use GitHub actions

Which issue(s) this PR fixes:

https://github.com/longhorn/longhorn/issues/8039
What this PR does / why we need it:
Special notes for your reviewer:
Additional documentation or context

@merigfy backport v1.6.x

@mergify backport v1.5.x


feat: Polyfill Promise.all()

Why
As reported by @vvo, AlgoliaSearchCore.prototype.searchForFacetValues does not work in IE11 because we rely on the native Promise.all() since #677.
Solution

Polyfill the all() method in each client._promise build
Use client._promise.all() instead of Promise.all()

I only trust Google and my intuition here, it's not enough.
Integration tests still work so there's for sure no regression. I was hoping we run the browser tests on develop. Do we have a way to run them on this branch though?

You can comment out the bits that check for node in scripts/test and scripts/test-noe

Yay, all browser tests pass \o/

Perfect, great work!


use GVK to get Kind of consumer obj

Instead of using hardcoded kind use  GVK to get the consumer Kind and use it for comparision.
Signed-off-by: Madhu Rajanna<EMAIL_ADDRESS>/assign @nb-ohad

/lgtm

/cherry-pick release-4.11

/cherry-pick release-4.11


Broken link -  shared Windows 10 PC

Current link is broken

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c696c9ed-b095-68ff-d50d-df345daf8a4a
Version Independent ID: 50ad3b26-8e9b-6a56-7b3c-56c1374c4747
Content: What's new in Windows 10 Enterprise 2019 LTSC
Content Source: windows/whats-new/ltsc/whats-new-windows-10-2019.md
Product: w10
GitHub Login: @greg-lindsay
Microsoft Alias: trudyha

@officedocsbot assign @mypil

@jandreacola - Thank you for pointing this out.
I will get this issue over to the Win10 ITPro writing team for investigation.
Thank you for reporting and making the docs better. Much appreciated.
I made a note to request the team to update this when the work is complete.

Created PR https://github.com/MicrosoftDocs/windows-itpro-docs/pull/3132

Broken link: https://docs.microsoft.com/en-us/windows/configuration/set-up-shared-or-guest-pc.md
Working link: https://docs.microsoft.com/en-us/windows/configuration/set-up-shared-or-guest-pc
This needs to be fixed internally by Microsoft technicians with access to the public pages.


Broken link "Data Orchestration using Azure Data Factory for Azure Data Lake Store"

The "Data Orchestration using Azure Data Factory for Azure Data Lake Store" link under "Next steps" points to "https://support.office.com/en-us/article/important-information-about-office-mix-preview-end-of-service-c1c04f84-a7bb-4602-9645-258017155258?ui=en-US&rs=en-US&ad=US" which doesn't seem correct.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 493e3542-cd60-9c10-41c2-f68a38528384
Version Independent ID: 0c081260-2bf3-6ae4-0c52-365def2357b5
Content: Best practices for using Azure Data Lake Store
Content Source: articles/data-lake-store/data-lake-store-best-practices.md
Service: data-lake-store
Product: unspecified
GitHub Login: @sachinsbigdata
Microsoft Alias: sachins

@gumaflux Thanks for the feedback. We are actively investigating and will get back to you soon.

@gumaflux Thanks for the feedback! I have assigned the issue to the content author to evaluate and update as appropriate.

Hi,
Happy I could help
Regards, Gustav
On 21 May 2018 at 23:45, VairavanS-MSFT<EMAIL_ADDRESS>wrote:

@gumaflux https://github.com/gumaflux Thanks for the feedback! I have
assigned the issue to the content author to evaluate and update as
appropriate.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/azure-docs/issues/8924#issuecomment-390805236,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABWTi8ZsqZSBhXcHuAjkMGQNT9Wur93Aks5t00NvgaJpZM4UHBFI
.

Is the broken link fixed yet?

@gumaflux Great catch! Thank you for the feedback.
Fix should be published in 72 hours.

#in-progress
#please-close


fixed @ CVE-2022-2900

Description:
Affected of this package are vulnerable to Server-side Request Forgery (SSRF) due to improper detection of protocol, resource, and pathname fields. Exploiting this vulnerability results in bypassing protocol verification. parse-url inproperly detecting protocol,resource and Pathname . This allow to bypass protocol check . Also this bug make ssrf check bypass.
CVE-2022-2900
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N
GHSA-j9fq-vwqv-2fm2

Related issue(s):
Fixes #
Notes for reviewer:

Checklist

[x] Documented (Code comments, README, etc.)
[x] Tested (unit, integration, etc.)

Thanks @mik-patient
We have this onboard to address as part of https://github.com/hashgraph/hedera-json-rpc-relay/pull/442 but appreciate this submission.
Please review the Contributing Guide noted on the README to ensure submission jobs pass on your PR.
DCO for example if failing in your submission.

@mik-patient
Taking this prompt i rebased my own stale PR https://github.com/hashgraph/hedera-json-rpc-relay/pull/442 and went to v6.0.0
Happy to take your contribution PR if you're able to fix the DCO issues.
The goal would be to get this in before EOD 10/14/22

Hi! @Nana-EC Thank you for your respond. please approved the pull-request as merged for fixed this issue-vulnerabilities. I hope you will be happyness days.
Regards, @mik-patient

HI @mik-patient , please see my previous response regarding following the contribution guidelines.

I won't be able to approve the PR until all items are passing including tests.
Our sprint ends at the end of the week (Sunday) and I'll be cutting a new release on Monday.
Hopefully you can get this PR fixed by then, if not due to timing I'll have to go with my PR and close this one without merging.

Hi! @Nana-EC I cannot for change this file because this file is larged. could you please change the file for fixing the issue?

HI @mik-patient ,
From the repo simply running npm i in your console, this will install packages and update the package-lock.json
Following this git commit the modified file and push to update this PR.
If you're unable to do this then we should close this  PR as that would be required to address the issue and pass the build/test gates.
As mentioned I already have a PR that fixes this ready to check in but I'm happy to accept your PR once you've addressed the PR comments and concerns within the current sprint timeline I noted.

Closing due to sprint end.
As noted this was already captured in https://github.com/hashgraph/hedera-json-rpc-relay/pull/442
Thank you for your attempt and we welcome any future contributions


Any chance of updating kernel version to 5.6+?

Hi, thank you for this amazing project, really appreciate it.
My quick question was about having a chance to upgrade the kernel to 5.6+, I'm trying to use wireguard, and seems like it's not working as intended in kernels sub 5.6, and in this case it's 4.9.
Any info with a workaround would be appreciated, thank you.

The Jetson Nano only supports kernels < 5, due to its limited CUDA support. Kernels above 4 require CUDA 11, which can't run on the Nano due to low-level driver dependencies. Sorry.

I see, okay, thank you for the background.


Index addition/exclusion

I'd love to be able to exclude a specific index from the maintenance, not just an entire table.

You can do that today.
https://ola.hallengren.com/sql-server-index-and-statistics-maintenance.html#Indexes


The Latest Version Can't Access Files Even if I Manually Grant "Manage All Files"

Build version: 0.82.1
**Game version:**Any
Game resolution: Any
Device model: OPPO K9S
Android version: 11
Description of the issue: App can't browse for game files even after I give the app "manage all files permission" manually in the popup window upon opening the app. The file browser is empty with the error message "Permission to access filesystem denied". It seems that the app does not ask for file permission in a particular way: It does not have the manage files permission in app settings. You can only give the permission through the special permission page (the popup window upon opening the app).

In latest version of JL-mod, file browser error/bug appear for me! (Previous build work! fine)
https://user-images.githubusercontent.com/81209454/171529510-d8819b1f-802c-4eef-a960-43f91bd2106e.mp4

Fixed


Order flags alphabetically

Ordering flags by flag name, removing unused parameter in FlagTreeStructure constructor.

Thanks for doing this @carmenquan ! I wonder though, should we perform the sort at the network fetch level so all of the plugin uses the alphabetically sorted collection? In other words, are there instances in the plugin where we don't want flags to be alphabetically sorted?

that's a great point Yus, I'll move the logic over there so we can make sure we're consistent across the plugin to start


Problem with set_i2c_address

Hi,
First, congrats for your work.
Currently I'm testing with two sensors. My problem is that I'm unable to change the default i2c address.
So, I connect the LPn pin to enable/disable the sensor, as exposed on the datasheet, to program the new address. But, when I call to set_i2c_address it gives me the following error:
Traceback (most recent call last):
File "test_2.py", line 20, in
driver_l.set_i2c_address(62)
File "/home/pi/.local/lib/python3.7/site-packages/vl53l5cx/vl53l5cx.py", line 745, in set_i2c_address
self.wr_byte(0x7fff, 0x02)
File "/home/pi/.local/lib/python3.7/site-packages/vl53l5cx/vl53l5cx.py", line 404, in wr_byte
self._i2c_bus.i2c_rdwr(write_addr_and_value)
File "/home/pi/.local/lib/python3.7/site-packages/smbus2/smbus2.py", line 658, in i2c_rdwr
ioctl(self.fd, I2C_RDWR, ioctl_data)
OSError: [Errno 121] Remote I/O error
This is my code:
import time
import RPi.GPIO as gpio
from vl53l5cx.vl53l5cx import VL53L5CX
gpio.setwarnings(False)
gpio.setmode(gpio.BOARD)
gpio.setup(7, gpio.OUT)
driver_l = VL53L5CX()
gpio.output(7, False)
gpio.setup(11, gpio.OUT)
gpio.output(11, True)
driver_l.set_i2c_address(62)

I'll try to check what is happening with it later on today or early in the morning and return back to you.

It seems that there's some discrepancy between how i2c works for smbus (or such) and what it does in C code. I've fixed it in the source. I'll push new (patch) version in PyPi in a moment... Let me know if it works now.
BTW wasn't sure if I have every even tried to change i2c address with VL53L5CX - I did it for up to 8 sensors for VL53L0X and VL53L1X.
Out of curiosity - what are you planning to do with two sensors on the same bus? If you find bus being swamped it is worth considering using second i2c bus on the RPi (or on Pi4 there are 4 exposed i2c busses on GPIO pins! :D )

Wow, thanks for your quick response and patching.
Now I am out of muy home, I will test next Monday.
For me it is the first time I use the pi's i2c channel. I didn't known that
the pi has more than one i2c ports(I'm testing with a pi 3). I'll try It.
I'll keep you informed of my progress.
Thanks one more time.
El sáb, 7 may 2022 15:49, Daniel Sendula @.***>
escribió:

It seems that there's some discrepancy between how i2c works for smbus (or
such) and what it does in C code. I've fixed it in the source. I'll push
new (patch) version in PyPi in a moment... Let me know if it works now.
BTW wasn't sure if I have every even tried to change i2c address with
VL53L5CX - I did it for up to 8 sensors for VL53L0X and VL53L1X.
Out of curiosity - what are you planning to do with two sensors on the
same bus? If you find bus being swamped it is worth considering using
second i2c bus on the RPi (or on Pi4 there are 4 exposed i2c busses on GPIO
pins! :D )
—
Reply to this email directly, view it on GitHub
https://github.com/Abstract-Horizon/vl53l5cx_python/issues/2#issuecomment-1120213259,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG2OISYGM7BS7PDJ5N7FFJ3VIZYHZANCNFSM5VI2AJRA
.
You are receiving this because you authored the thread.Message ID:
@.***>


Problem (connected) component decomposer

Extend EnergyImpactDecomposer to support a variable subproblem size (related to #206) and return one connected subgraph (component) at the time, or implement a dedicated decomposer for this purpose (e.g. ComponentDecomposer).

ComponentDecomposer implemented in #232.


Implement authentication flow

[x] development environment
[x] authentication enable / disable
[x] integrate AuthServiceClient
[x] handle authenticated flow (authenticated() == true then req.auth = secrets).
[x] handle unauthenticated flow (if authenticated() != true then generateTokenUrl())
[x] send token_url in message to user

Added in #82


Coolwatcher fail to burn A9G

Hi.
Please let me know any advice on this.
I have 5 days stucked on this, trying to set the A9G  flashing enviroment. This has been a real pain because official site is outdated about software.
I downloaded your fies for windows:
[CSDTK42_Update2_18_7_27.7z]
[GPRS_C_SDK_V2129.zip]
I put them in C: root,  using folder names:
CSDTK
GPRS_C_SDK
Later on CSDTK I run   "config_env_admin" as administrator, which finisk OK.
Later  on GPRS_C_SDK  folder I open a POWER SHELL WINDOW and run  "./build.bat app"  which also finish OK.
I compiled the program call from DEMO folder which finished OK also.
Later on coolwatcher I select the flash Programmer as host_8955_flsh_spi32m_ramrun.lod
And finally select the compiled file, CALL (from demo folder):   C:/GPRS_C_SDK/hex/call/call_B2129_debug.lod
But when I press the burn button (launch fastpf), I get several errors and the process stops:

I´m using Windows 11, a PROLIFIC cable and the latest driver taken from PROLIFIC page.  The cable look fine when checked on Windows device manager.
Any idea why this error ?

1. SDK version（SDK 版本）
{
}

2. In what kind of operation problems appear, and how to reproduce the problem ?（什么样的操作步骤问题会出现，是否是稳定复现，如何复现问题？）
{
}

I'm also experiencing the same problem, I've already changed the USB to SERIAL adapter but the problem still persists..


Merge ServiceFabricDeploy task related changes

Looks good to me. However, test the tasks once as this is a bug payload.

Bump up the powershell and compose task versions. Holding sign-off for this.


Provide peekInt8 to reduce allocations

Background:
In a project I use, sarama produces by far the most ongoing allocations (mainly, I think, because have lots of small messages) which negatively impacts our GC stats. I recently went through and tried to reduce them where I could and have been quite successful in that sarama doesn't show up in the pprof top10 by alloc_objects.  This is the simplest change I made so far. The others follow the pattern from #1161 and are more invasive.
I benchmarked this change by consuming 5 million messages and then dumping out a heap profile.
Before the change:
Showing nodes accounting for 30777165, 99.24% of 31012468 total
Dropped 6 nodes (cum <= 155062)
Showing top 10 nodes out of 19
      flat  flat%   sum%        cum   cum%
  10362760 33.41% 33.41%   25777082 83.12%  github.com/Shopify/sarama.(*MessageBlock).decode
   5172719 16.68% 50.09%   25887933 83.48%  github.com/Shopify/sarama.(*MessageSet).decode
   5123816 16.52% 66.62%    5123816 16.52%  github.com/Shopify/sarama.(*partitionConsumer).parseMessages
   5087542 16.40% 83.02%    5087542 16.40%  github.com/Shopify/sarama.(*realDecoder).peek

Here you can see that peek is responsible for 16.4% of the objects allocated, a little over one per message (makes sense, because of the record batching in kafka).
After the change:
Showing nodes accounting for 25577632, 99.73% of 25647802 total

Peek doesn't show up at all and we allocate 25.6 million objects vs 31 million, which is the expected reduction.

Need to work with my company for the CLA


Help

Hello BAByte. I’ve tried your contribution in order to get MacOS working well on a Surface Book. But unfortunately I couldn’t.
I’ve seen that people in the thread you mentioned are successful in that task. But the problem is that  I could read the page translating, but I can’t download the EFI folders they attach because I can’t register. It asks for an invitation code that I have no idea how to get. And since it’s in Chinese, makes my life even harder.
Since you’re an user there, could you provide me an invitation code or upload those files here?

The cover file you need is in the warehouse. Sorry I don’t have an invitation code either. In addition: It is not recommended to use Surface Book to install Mac OS, which will speed up battery consumption.

Hello BAByte. I’ve tried your contribution in order to get MacOS working well on a Surface Book. But unfortunately I couldn’t.
I’ve seen that people in the thread you mentioned are successful in that task. But the problem is that I could read the page translating, but I can’t download the EFI folders they attach because I can’t register. It asks for an invitation code that I have no idea how to get. And since it’s in Chinese, makes my life even harder.
Since you’re an user there, could you provide me an invitation code or upload those files here?

The cover file you need is in the warehouse. Sorry I don’t have an invitation code either. In addition: It is not recommended to use Surface Book to install Mac OS, which will speed up battery consumption.


Subsecond value in Squid Access Log timestamp is incorrect

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[X] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug
In Services>Web Proxy>Access Log, the displayed time in column Date displays an incorrect subsecond value. For example :
2023-05-31T21:40:01.550000
when the value stored in the file /var/log/squid/access.log is
1685562001.553
It should be
2023-05-31T21:40:01.553
To Reproduce

Go to '.Services>Web Proxy>Access Log'
Look at the Date column, see the value.

Expected behavior
Correct time is displayed
Describe alternatives you considered
The issue is located in /usr/local/opnsense/scripts/syslog/logformats/syslog.py , as shown by the backend command queryLog.py :
/usr/local/opnsense/scripts/syslog # ./queryLog.py --filename access --module squid --limit 1
{"filters":"","rows":[{"timestamp":"2023-05-31T21:40:01.550000","parser":"SysLogFormatEpoch","facility":null,"severity":null,"process_name":"","pid":null,"rnum":2,"line":"1186 <IP_ADDRESS> TCP_TUNNEL\/200 6181 CONNECT update.qnap.com:443 - HIER_DIRECT\/<IP_ADDRESS> -"}],"total_rows":2,"origin":"access"}

/usr/local/opnsense/scripts/syslog # tail -n1 /var/log/squid/access.log
1685562001.553   1186 <IP_ADDRESS> TCP_TUNNEL/200 6181 CONNECT update.qnap.com:443 - HIER_DIRECT/<IP_ADDRESS> -

Line 90 should be
        return len(line) > 15 and line[0:10].isdigit() and line[10] == '.' and line[11:14].isdigit()
and line 94 :
        return datetime.datetime.fromtimestamp(float(self._line[0:14])).isoformat(timespec='milliseconds')
After this change,
/usr/local/opnsense/scripts/syslog # ./queryLog.py --filename access --module squid --limit 1
{"filters":"","rows":[{"timestamp":"2023-05-31T21:40:01.553","parser":"SysLogFormatEpoch","facility":null,"severity":null,"process_name":"","pid":null,"rnum":2,"line":"1186 <IP_ADDRESS> TCP_TUNNEL\/200 6181 CONNECT update.qnap.com:443 - HIER_DIRECT\/<IP_ADDRESS> -"}],"total_rows":2,"origin":"access"}

Environment
OPNsense 23.1.9 (amd64, OpenSSL).

@PacoTyson looks reasonable, thanks! fixed in https://github.com/opnsense/core/commit/038d5b9bcf78ca7b97ab80513d9780062ca68374


Bloated declaration files

Bug Report
The way Typescript generate d.ts files results in a lot of duplicated code which probably slow down typescript services.
In our internal project we have many files that contains the same type duplicated more than 600 times, coming from imported inlined types, resulting in ~230kb of .d.ts size increase.
🔎 Search Terms
emit declarations, dts, duplicate types, dedupe dts
🕗 Version & Regression Information

This is the behavior in every version I tried, and I reviewed the FAQ for entries about emit declarations, duplicate types and dedupe dts.

This behavior is different in v3.3.3333

⏯ Playground Link
Check the .D.TS tab in the playground
https://www.typescriptlang.org/play?ts=4.8.2#code/PTAEBkHlIaVBBAKqRAJAoqAdAES4gZVABcBDAI1AGcBTG0ASwFsAHAewCdiAKAIg5qkAxsV4BKUEwCuAG2IMWM+vKY0qAWABQIBDKptGAOxIALBlVBDStU-RqH5AkgE8W9c6AAmUxQyvEaTywtLWZ2LlAAb0sBUgD0JVUHUABfUAAzDjYmUH5BEV4AbhDNGgAPcOJLNkMqKtJQAF4YwXjE+x4Ack6xYtKKziqhGrrQSmahWLaaJK6evvLK6tqhppa4mgSZju5u3q1FweXRzzXJ1s32h135g4GI4ZXQegmpy+3rvYX7oZGq9LOby2sxu+36S0eowA5oCLsCdl87hC-qATLCNvDPrdwUdIVUGOjpiDETiHiiAFaE97E7GHMlPADWVMxczBdN+TxkzKurO+yKeOVecJ5oL5uJRxiFGJFJPZxyqBilRIRtJ+8tALG5H15SPFTwAjlqaWy1XjQBwjSqTfzRhYldSrX0gA
In a real project with long import paths between modules the size increase if more substantial. for example: https://github.com/barak007/ts-dts-dedupe/blob/master/dist/index.d.ts
💻 Code
import { create } from "./a/b/c";
export const a = create();
export const b = create();

// file: a/b/c.ts
export interface Base {}
export interface G1<T, T1> {}

export function create<T>(): G1<G1<G1<T, Base>, Base>, Base> {
  return {};
}

🙁 Actual behavior
The resulting .d.ts file is:
export declare const a: import("./a/b/c").G1<
  import("./a/b/c").G1<
    import("./a/b/c").G1<unknown, import("./a/b/c").Base>,
    import("./a/b/c").Base
  >,
  import("./a/b/c").Base
>;
export declare const b: import("./a/b/c").G1<
  import("./a/b/c").G1<
    import("./a/b/c").G1<unknown, import("./a/b/c").Base>,
    import("./a/b/c").Base
  >,
  import("./a/b/c").Base
>;

🙂 Expected behavior
Whenever the compiler need to use an external type it can create an import * as NAME from "request"; statement, then use NAME.TYPE to access the type instead of inline it with the import() syntax.
It will turn the previous .d.ts into:
import * as ABC from "./a/b/c";
export declare const a: ABC.G1<ABC.G1<ABC.G1<unknown, ABC.Base>, ABC.Base>, ABC.Base>;
export declare const b: ABC.G1<ABC.G1<ABC.G1<unknown, ABC.Base>, ABC.Base>, ABC.Base>;

Typescript can go even further and dedupe identical types:
import * as ABC from "./a/b/c";
type Alias = ABC.G1<ABC.G1<ABC.G1<unknown, ABC.Base>, ABC.Base>, ABC.Base>;
export declare const a: Alias;
export declare const b: Alias;

The compiler will need to be hygienic here and might also generate a readable name since it might appear in user autocomplete.

This sounds like #30258.

See also #44044 and #49730; the former is an open issue so I'd class this a duplicate of that.


Mock-up site wide grouping

First shot:

For now we'd just sort the groups themselves consistently alphabetically or something.

This filter option would not preclude us from having a global search feature. I think this would be a good idea anyway.

I like the sort menu and how it works with groups here.
Question:

how do you select ASC/DESC?

Oh whoops. Forgot that:

Then on mobile these should probably just collapse below the mass select things and filter.

Closing for now.


Propagate click events to original textarea

In the app I am working on, I need to know when the user clicks inside of
TinyMCE. Because TinyMCE is in an iframe, the events do not propagate when the
iframe is the event target.
This change will trigger a click event on the original textarea element.

Closing this as outdated - feel free to file again with Plunker example if it still is relevant.


Make rechunk robust to integer inputs

Fixes #351
cc @shoyer

this was my code. sorry about that guys i should've added a test for integer inputs.

LGTM, thanks!


change batch_size=2

I want to realize the real-time response,so I need to change the batch_size,how can I do?

@ashawkey


GH-44579: [C++] Use array type to compute min/max statistics Arrow type

Rationale for this change
arrow::ArrayStatistics uses raw C++ types such as int64_t and std::string for min/max types. We need to convert raw C++ types to Arrow types when we use arrow::ArrayStatistics for generating statistics array. (GH-45038)
We can't map std::string to an Arrow type. Because it may be arrow::binary(), arrow::utf8() or something.
What changes are included in this PR?
Use arrow::DataType information of associated array when we convert arrow::ArrayStatistics's min/max raw C++ types to Arrow types.
Are these changes tested?
Yes.
Are there any user-facing changes?
No.

GitHub Issue: #44579

I'll merge this in the next week if nobody objects this.


Incompatibility with Magento 2.3.3

Fatal error: Uncaught Error: Call to a member function setBodyAttachment() on null in /home/ccarnell/Sites/xx-uk2/vendor/eadesignro/module-pdfgenerator/Model/Email/TransportBuilder.php:16

This class has been refactored.
https://github.com/magento/magento2/blob/2.3-develop/lib/internal/Magento/Framework/Mail/Template/TransportBuilder.php

Any updates on this?

hi,
I had the same error. The reason is that magento has changed the email classes ... again.
The message in the TransportBuilder class based now on EmailMessageInterface instead MessageInterface.
https://community.magento.com/t5/Magento-DevBlog/Backward-incompatible-Changes-in-the-Mail-Library-for-Magento-2/ba-p/144787
I tried the patch (EmailMessageInterface backward compatibility issue patch for Magento 2.3.3 on https://magento.com/tech-resources/download#download2333) but the error was still the same.
The reason is that the message property of the TransportBuilder object is not initialized than the addAttachment method is called. But also the initialization of that object solved the problem because the new EmailMessage was still incompatible with the use of the zend framework in the Message class of that module.
The way I solved the problem for me was to exchange the complete content of the TransportBuilder class to the version I found there: https://magento.stackexchange.com/questions/292760/unable-to-add-attachment-in-email-after-upgrade-to-magento-2-3-3-version

Thx, @nobodyMO, you just saved my day.

@nobodyMO Thank you so much. I was trying from 6 hours, Implemented all ways to attach image with custom email but failed. Your comment saved me. Magento keep changing files. Anyway, Thanks a lot.

I've got the same problem with Magento 2.3.5-p2. @eadesignro do you plan to fix this soon?

Hi SaturnusDJ,
I only use my fork to keep the extension compatible with the Magento version used for our shop and the PHP version used.
I don't currently have a development environment with PHP 8.2 and therefore can't test whether there are other problems with the changed PHP restrictions in the new version.
If I understand the message correctly, all attributes of classes must be declared explicitly in PHP8.2. This can be fixed relatively easily. Simply insert the following lines between lines 13 and 14:
/**
     * @var UrlInterface
     */
public $_url;

However, it is possible that this is not the only missing declaration and that attributes must also be declared in other php files.


Fixed iOS build on Xcode 9.3

Removed APPLICATION_EXTENSION_API_ONLY = YES for iOS.
It builds successfully as all other platforms.
Fixes https://github.com/krzysztofzablocki/LifetimeTracker/issues/42

covered by #46