[SPARK-7159][ML] Add multiclass logistic regression to Spark ML

What changes were proposed in this pull request?
This patch adds a new estimator/transformer MultinomialLogisticRegression to spark ML.
JIRA: SPARK-7159
How was this patch tested?
Added new test suite MultinomialLogisticRegressionSuite.
Approach
Do not use a "pivot" class in the algorithm formulation
Many implementations of multinomial logistic regression treat the problem as K - 1 independent binary logistic regression models where K is the number of possible outcomes in the output variable. In this case, one outcome is chosen as a "pivot" and the other K - 1 outcomes are regressed against the pivot. This is somewhat undesirable since the coefficients returned will be different for different choices of pivot variables. An alternative approach to the problem models class conditional probabilites using the softmax function and will return uniquely identifiable coefficients (assuming regularization is applied). This second approach is used in R's glmnet and was also recommended by @dbtsai.
Separate multinomial logistic regression and binary logistic regression
The initial design makes multinomial logistic regression a separate estimator/transformer than the existing LogisticRegression estimator/transformer. An alternative design would be to merge them into one.
Arguments for:

The multinomial case without pivot is distinctly different than the current binary case since the binary case uses a pivot class.
The current logistic regression model in ML uses a vector of coefficients and a scalar intercept. In the multinomial case, we require a matrix of coefficients and a vector of intercepts. There are potential workarounds for this issue if we were to merge the two estimators, but none are particularly elegant.

Arguments against:

It may be inconvenient for users to have to switch the estimator class when transitioning between binary and multiclass (although the new multinomial estimator can be used for two class outcomes).
Some portions of the code are repeated.

This is a major design point and warrants more discussion.
Mean centering
When no regularization is applied, the coefficients will not be uniquely identifiable. This is not hard to show and is discussed in further detail here. R's glmnet deals with this by choosing the minimum l2 regularized solution (i.e. mean centering). Additionally, the intercepts are never regularized so they are always mean centered. This is the approach taken in this PR as well.
Feature scaling
In current ML logistic regression, the features are always standardized when running the optimization algorithm. They are always returned to the user in the original feature space, however. This same approach is maintained in this patch as well, but the implementation details are different. In ML logistic regression, the unregularized feature values are divided by the column standard deviation in every gradient update iteration. In contrast, MLlib transforms the entire input dataset to the scaled space before optimizaton. In ML, this means that numFeatures * numClasses extra scalar divisions are required in every iteration. Performance testing shows that this has significant (4x in some cases) slow downs in each iteration. This can be avoided by transforming the input to the scaled space ala MLlib once, before iteration begins. This does add some overhead initially, but can make significant time savings in some cases.
One issue with this approach is that if the input data is already cached, there may not be enough memory to cache the transformed data, which would make the algorithm much slower. The tradeoffs here merit more discussion.
Specifying and inferring the number of outcome classes
The estimator checks the dataframe label column for metadata which specifies the number of values. If they are not specified, the length of the histogram variable is used, which is essentially the maximum value found in the column. The assumption then, is that the labels are zero-indexed when they are provided to the algorithm.
Performance
Below are some performance tests I have run so far. I am happy to add more cases or trials if we deem them necessary.
Test cluster: 4 bare metal nodes, 128 GB RAM each, 48 cores each
Notes:

Time in units of seconds
Metric is classification accuracy

algo
elasticNetParam
fitIntercept
metric
maxIter
numPoints
numClasses
numFeatures
time
standardization
regParam

ml
0
true
0.746415
30
100000
3
100000
327.923
true
0

mllib
0
true
0.743785
30
100000
3
100000
390.217
true
0

algo
elasticNetParam
fitIntercept
metric
maxIter
numPoints
numClasses
numFeatures
time
standardization
regParam

ml
0
true
0.973238
30
2000000
3
10000
385.476
true
0

mllib
0
true
0.949828
30
2000000
3
10000
550.403
true
0

algo
elasticNetParam
fitIntercept
metric
maxIter
numPoints
numClasses
numFeatures
time
standardization
regParam

mllib
0
true
0.864358
30
2000000
3
10000
543.359
true
0.1

ml
0
true
0.867418
30
2000000
3
10000
401.955
true
0.1

algo
elasticNetParam
fitIntercept
metric
maxIter
numPoints
numClasses
numFeatures
time
standardization
regParam

ml
1
true
0.807449
30
2000000
3
10000
334.892
true
0.05

algo
elasticNetParam
fitIntercept
metric
maxIter
numPoints
numClasses
numFeatures
time
standardization
regParam

ml
0
true
0.602006
30
2000000
500
100
112.319
true
0

mllib
0
true
0.567226
30
2000000
500
100
263.768
true
0

References
Friedman, et al. "Regularization Paths for Generalized Linear Models via Coordinate Descent"
http://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html
Follow up items

Consider using level 2 BLAS routines in the gradient computations
Add model summary for MLOR
Add initial model to MLOR and add test for intercept priors
Consider changing the tree aggregation level for MLOR/BLOR or making it user configurable to avoid memory problems with high dimensional data - SPARK-17090
Refactor helper classes out of LogisticRegression.scala
Design optimizer interface for added flexibility in ML algos
Support compressing the coefficients and intercepts for MLOR models

Test build #60900 has started for PR 13796 at commit e4681b0.

Test build #60900 has finished for PR 13796 at commit e4681b0.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60900/
Test FAILed.

Test build #60901 has started for PR 13796 at commit f4c817f.

cc @dbtsai @jkbradley @mengxr If you get a chance to review and/or provide feedback on the approach I'd appreciate it.

Test build #60901 has finished for PR 13796 at commit f4c817f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60901/
Test PASSed.

@sethah Thanks for implementing MultinomialLogisticRegression! This would be a major feature for Spark 2.1. @dbtsai is probably the best people to review this PR. But he is taking a break now. Do you mind waiting him for couple days?

@sethah I apologize for the delay. I just came back to US. Gonna make the first pass. Thanks.

@sethah Please merge master since there is a conflict. Thanks.

Test build #63702 has started for PR 13796 at commit 9d4559e.

Test build #63704 has started for PR 13796 at commit 026e1f6.

Test build #63702 has finished for PR 13796 at commit 9d4559e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63702/
Test PASSed.

Test build #63704 has finished for PR 13796 at commit 026e1f6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63704/
Test PASSed.

Test build #63798 has started for PR 13796 at commit 2f11bd7.

Test build #63799 has started for PR 13796 at commit 0d72c6f.

Test build #63798 has finished for PR 13796 at commit 2f11bd7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63798/
Test PASSed.

Test build #63799 has finished for PR 13796 at commit 0d72c6f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63799/
Test PASSed.

Merged build finished. Test PASSed.

Test build #63804 has started for PR 13796 at commit 51f51e6.

Test build #63804 has finished for PR 13796 at commit 51f51e6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63804/
Test PASSed.

Maybe we could move the shared code MultiClassSummarizer, LogisticAggregator and LogisticCostFun into LogisticRegressionUtils.scala?

@dbtsai Thanks for taking the time to review this! Major items right now:

Adding derivation to the aggregator doc (this is mostly finished, just fighting scala doc with Latex)
Deciding whether to add initial model and tests in this PR or as a follow up
Refactoring logistic regression helper classes to a separate file

Let me know if you see anything else.

@sethah Thank you for great work. I'll make another pass tomorrow.

Test build #63849 has started for PR 13796 at commit 96c52a5.

I added the derivation and other details to the scala doc for LogisticAggregator. I have not had a chance to run the doc build yet, but the content is there now for review. I'll build the docs today and make sure it looks alright.

Test build #63849 has finished for PR 13796 at commit 96c52a5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63849/
Test PASSed.

Test build #63878 has started for PR 13796 at commit b4203a5.

I checked that the derivations in the LogisticAggregator doc render nicely with scala doc, even though it's a private class and so will not show up in the API. There was a slight complication with latex and scaladoc which is described here.

Test build #63878 has finished for PR 13796 at commit b4203a5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63878/
Test PASSed.

Merged build finished. Test PASSed.

Test build #63938 has started for PR 13796 at commit 61521e6.

Test build #63938 has finished for PR 13796 at commit 61521e6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63938/
Test PASSed.

Test build #63951 has started for PR 13796 at commit f02b226.

Test build #63951 has finished for PR 13796 at commit f02b226.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63951/
Test PASSed.

I go through the PR again, and it's in a very good shape. Only couple minor issues needed to be addressed. Thank you @sethah for the great work. This will be a big feature in Spark 2.1

Test build #63996 has started for PR 13796 at commit 0c851d7.

Thanks @dbtsai for the detailed review! I addressed most comments. We still need to:

Decide whether to handle numClasses being specified in metadata
Decide what happens when numClasses == 1 (only label 0.0 is encountered)

Also, one thing I'm concerned about is having separate MultinomialLogisticRegression and LogisticRegression. Of course, we do this mainly because we cannot change the LR API to support a matrix of coefficients very easily. Still, I think it's quite annoying to have to switch to a different estimator for multiclass. The multinomial estimator more or less supercedes the functionality of BLOR, but LogisticRegression is a canonical name and users may gravitate to it. Further, even when/if people realize that you can use MLOR for both binary and multiclass, it may be confusing what LogisticRegression is used for. I just want to discuss it before we make it public.

Test build #63996 has finished for PR 13796 at commit 0c851d7.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
          logWarning(s\"All labels belong to a single class and fitIntercept=false. It's a \" +

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63996/
Test PASSed.

I also created an umbrella JIRA for tracking follup items - SPARK-17133

Test build #64007 has started for PR 13796 at commit ffc64d4.

Test build #64007 has finished for PR 13796 at commit ffc64d4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64007/
Test PASSed.

Test build #64033 has started for PR 13796 at commit fc2aa95.

Test build #64033 has finished for PR 13796 at commit fc2aa95.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64033/
Test PASSed.

@sethah Thank you for this great weighted MLOR work in Spark 2.1. I merged this PR into master, and let's discuss/work on the followups in separate JIRAs. Thanks.

@dbtsai Thanks for all of your meticulous review. Very much appreciated! Glad we can have MLOR in Spark ML now.

@sethah I also have a concern to have a separate MLOR class, and I prefer consolidate them into one so we can maintain them easier. This has to be done before the release of 2.1 otherwise, we can not change the interface anymore. Can you create a JIRA discuss this issue? Thanks.

I found a problem in the merged code:  when reg == 0 the minimizer of softmax cost is not unique.
In such case, it will cause Hessian matrix non-invertible, and I thinks it may cause quasi-newton's methods such as LBFGS run into numerical problems.
so, is it better to forbid the reg==0 case for softmax parameters ?
cc @sethah @dbtsai

@WeichenXu123 Do you run into this potential issue with any dataset? If so, we may need to consider optimize softmax with pivoting when reg == 0.

@dbtsai
simply say, when reg==0, the minimizer is not a point but becomes a plane. so the second derivative on the plane will turn into zero. so ...
and I give a reference here:
http://ufldl.stanford.edu/wiki/index.php/Softmax_Regression
it mentions this problem.

The solution of this overparameterized problem in the link is just adding the regularization, and users may not want it. I think we need to optimize it on (k-1) parameters, and then put the final or the first one back by centering the intercepts and weights. @sethah could you have a JIRA to track this issue? Thanks.

@dbtsai
in this jira
https://issues.apache.org/jira/browse/SPARK-17163
it consider to unify interface for binary logistic regression and softmax.
but I think they are not equivalent in fact (when numClass == 2).
I think the better way is:

extend the binary logistic regression to numClass > 2 cases, but optimize it on (numClass-1) parameters.
modify the softmax reg==0 case computation, use the way described in 1

SPARK-17201


Better DI support

At the moment DI support is limited to creation of root validators. The container is abstracted behind IValidatorFactory
We should allow resolution of child validators and other components during
Validation. To do this a resolver should be exposed on ValidationContext that calls into the container. New overloads of SetValidator should make use of this.

Naively support .net core DI to handle this.
Optionally, create a package that integrates the native DI into your stuff with a configure call, etc.

@waynebrantley  Supporting native .net core DI directly isn't a good idea as it's not supported on all platforms that FluentValidation supports.
The plan is to use the existing IValidatorFactory to do this, which can then wrap a container of your choosing. For FluentValidation.AspNetCore this will be provided by the asp.net core service provider. For other platforms will provide a simplistic implementation by default which can then be swapped out.

I'd also love to see a separate package for Microsoft.Extensions.DependencyInjection that allows us to use it outside of ASP.NET Core.
Great project! 👍

I've written a set of IoC Extensions here:
https://github.com/nsarris/FluentValidation.IoC
It's an abstract solution and I have implemented an IoC provider with Unity. It's very easy to add one for Microsoft.Extensions.DependencyInjection. The solution supports net 46/47/standard/core
Please check the tests for usage. Unfortunately I didn't have time for more tests or proper documentation
FluentValidation's architecture has Validators building their logic upon construction, so I tried to build the DI extensions around the idea of them being singletons, to avoid the build up overhead.
The validation state is held within the Context classes so the whole idea is to attach an IValidatorFactory and a custom IDependencyResolver in the context. For this I used the Dictionary for custom data. It would be really easy to fuse this with the FluentValidation if the context class would add those two properties.
The other thing I tried to tackle was Literal resolution. The current version of FluentValidation treats string literals in a static way, either at build up or with the LanguageManager which is static. So I introduced an ILiteralService which can be injected as well. As there is no context available in the WithMessage delegates, I had to use a ServiceLocator for the LiteralService.
For IValidatorProvider/IDependencyResolver there is the option of injection by constructor, explicit construction or serviceLocation (if not provider)
I decided to keep IValidatorProvider and IDependncyResolver separate, although my Unity implementation provides them in the same container instance.
I can provide more info if you are interested.

@nsarris thanks for the suggestion. This seems far more complex than what I'm aiming for with the DI support here, at least for the initial version.
The plan for the initial version of the DI support is to provide a way to resolve child validators only specifying the type, eg SetValidator<MyChildValidator>() rather than having to instantiate the child validator, or pass it via the constructor.
This would be done by passing an IValidatorFactory into the validator via the ValidationContext. This would be passed in via the integration with ASP.NET Core's validation provider. In time, this may be expanded to other platforms, but for now the focus on this is only for ASP.NET Core via the service provider mechanism. We already have an implemention of IValidatorFactory that works with the service provider which can be used for this. For integration with other DI containers, it'll be up to the user to provide the appropriat adapter. I'm not at all keen on providing integration with the various different containers out there, I've tried this in the past and it's a maintenance nightmare.
For future versions, we can look at providing hooks for doing this outside of web projects, but that's not the focus for now.
That being said, I'd be happy to review your solution to see if any of this could be used for what we're aiming for. Unfortunately I don't have much free time at the moment either, and as this is a fairly sizable piece this will take me some time to review. I know you said you don't have time for documentation, but please could you at least provide a high-level overview of the architecture and usage. That'll give me a good starting point to be able to do a proper review.

so I tried to build the DI extensions around the idea of them being singletons, to avoid the build up overhead.

Since v6.4 the validator cache means that validators do not need to be singletons, and the asp.net cor integration intentially registers them as transient to avoid the confusion and complexities that come from singleton-scoped dependencies, which can cause a lot of gotchas especially for new devs who aren't familiar with the implications. It's important that any solution can work with transient as the default scope.

For this I used the Dictionary for custom data. It would be really easy to fuse this with the FluentValidation if the context class would add those two properties.

I am not keen on adding additional properties to the context. As this is going to be a separate package, using the RootContextData dictionary to extend the context, and then wrapping this with extension method is preferable.

The other thing I tried to tackle was Literal resolution

Could you provide some details of what your use case is for this? This sounds like it'd be out of scope for what we're aiming for here, at least for the initial version, or am I missing something?

That being said, I'd be happy to review your solution to see if any of this could be used for what we're aiming for. Unfortunately I don't have much free time at the moment either, and as this is a fairly sizable piece this will take me some time to review. I know you said you don't have time for documentation, but please could you at least provide a high-level overview of the architecture and usage. That'll give me a good starting point to be able to do a proper review.

Sure, I will try to put something together this weekend for you to get a better idea.

For integration with other DI containers, it'll be up to the user to provide the appropriat adapter. I'm not at all keen on providing integration with the various different containers out there, I've tried this in the past and it's a maintenance nightmare.

I built everything around the existing IValidatorFactory and two interfaces (IDependencyResolver and ILiteralService). The unity container project is just a proof of concept implementation. You can use any container you want by inheriting those simple interfaces so I think we're on the same page here.
There's three interfaces because I tried to tackle three DI related issues.

The resolution of Validators by validated type or validator type at run time, through IValidatorFactory, which is what you are looking for.
The resolution of dependencies within the validator's payload. This requires a proper IoC Container/Resolver and a disposable wrapper around an IoC context. More on this on a later post .
The resolution of string literals at run time using proper DI which I prtially cover later on in this post.

Since v6.4 the validator cache means that validators do not need to be singletons, and the asp.net cor integration intentially registers them as transient to avoid the confusion and complexities that come from singleton-scoped dependencies, which can cause a lot of gotchas especially for new devs who aren't familiar with the implications. It's important that any solution can work with transient as the default scope.

The buildup in constructor kind of contradicts this. In order for the validators to really be transient you could refactor things in the future to use rule builder classes to seperatre the rule building (singleton) from the actual validation execution (transient) . This would really help with DI as well as you could add dependencies in the executor (via constructor or otherwise).
In the current state of the framework you can emulate this partially by using a validator for the executor (transient) with an included validator as the rule builder (singleton). Alas, this doesn't help with DI because the rule builder needs the dependencies.

I am not keen on adding additional properties to the context. As this is going to be a separate package, using the RootContextData dictionary to extend the context, and then wrapping this with extension method is preferable.

This is exactly how I approached it and I have no problem keeping it this way.

Could you provide some details of what your use case is for this? This sounds like it'd be out of scope for what we're aiming for here, at least for the initial version, or am I missing something?

In FluentValidation you provide string literals in three ways:
WithName, WithMessage and WithLocalizedMessage, ValidatorOptions.PropertyNameResolver,  ValidatorOptions.DisplayNameResolver (and the ValidatorOptions.Language translations)
None of these ways are really DI friendly in theory. The WithName and WithMessage don't inject the ValidationContext and thus it's not possible to resolve anything within the delegate. The WithLocalizedMessage is based on static properties on a resource type.  Finally the PropertyNameResolver/DisplayNameResolver are statics which is ServiceLocation not proper DI (plus there is no ErrorMessageResolver). I implemented ILiteralService to move towards a proper DI solution for string literals. I did have to implement it with Service Location for now but I was hoping to talk you into adding an overload to WithName/WithMessage that injects the context into the delegate.
The service looks like this:
public interface ILiteralService
{
    string GetPropertyName(Type entityType, string propertyName);
    string GetPropertyName<T>(string propertyName);
    string GetValidationErrorMessage(string code);
}

This is used like this:
RuleFor(x => x.Id)
.NotEmpty().ResolveName()
.WithErrorCode("VatValidationServiceFailure")
.ResolveMessage();

ResolveName and ResolveMessage are two of the extension methods in the IoC package. The encapsulate the LiteralService (currently retrieved by a static ServiceLocator) to provide literal resolution at runtime (for property names and error messages respectively).
On a sidenote, a minor issue I faced here was that I had to use reflection to the get the error code from the ErrorCodeSource built from WithErrorCode. (RuleBuilder.Rule.CurrentValidator.Options.ErrorCodeSource._message)

Sure, I will try to put something together this weekend for you to get a better idea.

Thanks.

The buildup in constructor kind of contradicts this. In order for the validators to really be transient you could refactor things in the future to use rule builder classes to seperatre the rule building (singleton) from the actual validation execution (transient)

Agreed, I’ve looked at doing this before but this is such a massive breaking change that really has very little benefit for the vast majority of users, and introduces several other complexities regarding how the rules interact with each other. I spent quite a while prototyping the feasibility of this and in the end decided to keep with the current approach.
Some thoughts regarding ILiteralService:

I think this would be better split into different resolver classes for each job rather than a single service. This keeps each class focused on a single task.
no need for both generic and non generic versions of GetPropertyName. The usual approach with FluentValidation is the public api uses generics and internal api (which this would be) doesn’t.
GetPropertyName should probably be GetDisplayName, and there should be a separate GetPropertyName (to correspond to WithName and OverridePropertyName respectively)
I think we can get rid of the ResolveName method. I’d imagine that this would be the default behaviour, falling back to the default message if a custom resolver is not specified.
I don’t really like GetValidationErrorMessage being tied to the error code. Error codes and messages are intentionally separate, and I don’t like the idea of having the set an error code to be able to find the default message for the validator. Again I think this should be a separate resolver class that allows you to provide the message for a specific validator, or it would fall back to use the LanguageManager. If you do opt to provide a custom resolver, then I could imagine that you could use this to implement the error code-based behaviour if you wanted to, but I wouldn’t want to provide this as a default api method.
if these services are going to be used internally by FV core then there will also be a need to specify them if the user isn’t using a container. Probably via a global option, maybe replacing the current global DisplayNameResolver/PropertyNameResolver methods with instances that can be swapped out. Not ideal but we do need to support that scenario. Another idea maybe to provide a default, transparent ioc implementation for FV’s own services (maybe by embedding TinyIoc, or something), which automatically gets swapped out with the ServiceProvider when running under AspNetCore.

Thanks again for this, you’ve given me a lot to think about.

Hi, I'm afraid there are no current plans to support open generics in this way, although if you or someone else wanted to submit a pull request to add it I'd be happy to give it a review.

There's three interfaces because I tried to tackle three DI related issues.

The resolution of Validators by validated type or validator type at run time, through IValidatorFactory, which is what you are looking for.
The resolution of dependencies within the validator's payload. This requires a proper IoC Container/Resolver and a disposable wrapper around an IoC context. More on this on a later post .
The resolution of string literals at run time using proper DI which I prtially cover later on in this post.

Just to wrap No.2 up as I promised. I think we're on the same page with No.1 and No.3 revolves around injecting the context in the validation rule delegate.
No.2 is about addressing the need to resolve dependencies within the validation delegates. Suppose that to validate a VAT number you have a dependency on a service with side effects that calls a goverment API to fetch data. How would you inject that dependency into the validator. One option is to have the validator created by a DI container and inject the dependency in the validators constructor. The problem with this approach is that the validator needs to be instantiated each time and that will cost you the rule buildup. We discussed this above and you mentioned this is by design. However I tried to provide an option for the validators to be singletons (so no buildup cost) and the dependency resolver to be injected in the context, which in turn injects it into the validator delegate.
To implement this i used a simple IDependencyResolver interface that you can use to create a proxy to any container and inject it into the context. For the implementation of the context buildup I used an IoCValidationContext class that uses a IValidatorFactory and IDependencyResolver as dependencies and goes on to create the ValidationContext and provide an interface for Validation execution.
The IoCValidationContext also handles lifecylcle by IDisposable, by simple propagating Dispose to the IDependencyResolver implementation.
This is the basic idea around it. If you want me to further document my project I think I can find some time to do it soon.

FYI, I took another look and I found out that proper dependency injection for name and message resolution can be achieved using the Configure method and DisplayName/PropertyName/MessageBuilderContext (which gets rid of the ServiceLocation approach)
For the first two I created a custom ILiteralSource class from which you can extract the ILiteralService dependency as the GetString function is called with a ValidationContext.
For the later, the MessageBuilderContext contains a reference to the parent ValidationContext, from where you can get validation time dependencies.

Yes, that definitely helps to simplify it a bit.
The only thing to keep in mind is that a MessageBuilder is associated with the rule, not with the individual validator, so this will not do what you'd expect it to:
// Only the `bar` message will ever be used.
RuleFor(x => x.Foo).NotNull().ResolveMessage("foo").Must(Whatever).ResolveMessage("bar");

It'd be better to set the ErrorMessageSource, as this is per-validator. You can again use a StringSource for this:
return rule.Configure(config => {
	config.CurrentValidator.Options.ErrorMessageSource = new MySource(code, ...);
});

Somewhat related to this, I recently committed a change that allows you to retrive the message using an error code in a very smiliar way to this, but the message is retrieved from the LanguageManager.
For injecting validators, rather than using a new method that takes a Type, I was thinking instead of a static method that allows you to specify the validator type:
RuleFor(x => x.Address).SetValidator(Resolve.Validator<Address, AddressValidator>());

Resolve.Validator would just return a delegate, meaning it can be passed to the existing WithMessage(Func<T,TValidator>), and is strongly typed. What do you think of that syntax?

It'd be better to set the ErrorMessageSource, as this is per-validator. You can again use a StringSource for this:

Worked like a charm, thanks this is much cleaner.

Somewhat related to this, I recently committed a change that allows you to retrive the message using an error code in a very smiliar way to this, but the message is retrieved from the LanguageManager.

Yes, I noticed but that still requires setting up a static resource class. This is acceptable in certain cases but it in general it violates most of the principles of DI/IoC

For injecting validators, rather than using a new method that takes a Type, I was thinking instead of a static method that allows you to specify the validator type:
Resolve.Validator would just return a delegate, meaning it can be passed to the existing WithMessage(Func<T,TValidator>), and is strongly typed. What do you think of that syntax?

Like I said above statics and DI are contradicting patterns. In my humble opinion statics should be considered an anti-pattern nowdays, except for pure functions in the sense of helpers and extensions methods. Statics create problems with unit testing, break inversion of control and do not allow any other lifecycle except singleton and do not support DI scoping. I sencerily advise against it. I wanted to suggest the same for the validator options but I was happy to see you already decided to change it.
In the suggestion above the static "Resolve" class would essentially be a staic ServiceLocator for the IoC container/ Dependency resolver. I believe that instead of setting up the resolver statically you should inject it in the ValidationContext (which is what I did). Your ValidationContext carries the state of the validation process. If you inject your resolver in the context(or a leaner wrapper around it using an interface like I did) you have access to it in all steps of the process, including Validator resolution.
So then you can resolve either by spefic validator (AddressValidator) or generic type (Address) or with any kind of inteface mapping provider by the framework choice.

In the suggestion above the static "Resolve" class would essentially be a staic ServiceLocator for the IoC container/ Dependency resolver.

No, it wouldn’t....sorry I don’t think I explained it clearly. The method would not act as a service locator - it would simply return a delegate that would use the validatoncontext for injection - no service location required. I’ll try and throw up an example at some point.

Ok I see what you mean know and it sounds promising

I've settled on the following syntax. InjectValidator() will infer the child validator type to inject:
RuleFor(x => x.Address).InjectValidator();

Assuming the Address property is of type Address, InjectValidator will resolve an IValidator<Address> from the service provider.
For more explicit control over the type to resolve, the method can optionally take a lambda which receives the service provider and the context:
RuleFor(x => x.Address).InjectValidator((s, ctx) => s.Resolve<MyCustomAddressValidator>());

The service provider will be of type IServiceProvider (no need for a custom IDependencyResolver interface here), as this is part of the framework already, and is implemented by the .NET Core built-in container (meaning no additional dependency or package required for the common interface).
For 8.2 these new methods will be part of the FluentValidation.AspNetCore package, and can potentially be split out into a separate package in the future.
Resolving string literals for name/message as well as other services (like LanguageManager) won't be in this release as this still needs further planning, but will look at this for the future.
I won't be including an alternative of the IoCValidationContext for scoping/disposal. I strongly feel that scoping and disposal should be a feature the container should handle and FluentValidation itself should be completely agnostic of this. In a web scenario, the per-http-request lifetime would be handled transparently and this will all work correctly provided validators and their dependencies have their scopes set correctly when registered.

@JeremySkinner if Address implements IAddressProvider, potentially that is what I want resolved.
Perhaps an overload of RuleFor(x => x.Address).InjectValidator<IAddressProvider>();

@waynebrantley You can't specify type parameters like that, as the method will already have type parameters for T and TProperty. In this case you'd use the second overload.
RuleFor(x => x.Address).InjectValidator((s, ctx) => s.GetService<IValidator<IAddressProvider>>());

The service provider will be of type IServiceProvider (no need for a custom IDependencyResolver interface here), as this is part of the framework already, and is implemented by the .NET Core built-in container (meaning no additional dependency or package required for the common interface).

Agreed, I was going to do this myself and add the GetService with an extension method for .net only.

For 8.2 these new methods will be part of the FluentValidation.AspNetCore package, and can potentially be split out into a separate package in the future.

Do you mean you'll have seperate .net and core implementations or remove it for .net alltogether?

Resolving string literals for name/message as well as other services (like LanguageManager) won't be in this release as this still needs further planning, but will look at this for the future.

I've covered our needs with the IoC extensions project. I'd gladly help if needed. I would only like to point out that InjectMessage(Func<IServiceProvider, IValidationContext, string>) and InjectPropertyName(Func<IServiceProvider, IValidationContext, string>) similar to InjectValidator would cover most cases I can think of

I won't be including an alternative of the IoCValidationContext for scoping/disposal. I strongly feel that scoping and disposal should be a feature the container should handle and FluentValidation itself should be completely agnostic of this. In a web scenario, the per-http-request lifetime would be handled transparently and this will all work correctly provided validators and their dependencies have their scopes set correctly when registered.

I used IoCValidationContext for three reasons:

To remove all transient dependencies from the validators and move them to the IoCValidationContext instead, thus the ServiceProviders (DependencyResolver and IValidatorFactory)  are present in the IoCValidationContext ctor and not the validator ctor. (I understand you designed the validators as transient but I tried an approach that allows them to be used as singletons too.)

To abstract the building of an actual ValidationContext and the injection of the ServiceProviders into its RootContext. I guess this is simplified by the DependencyResolver removed (and IValidatorFactory replaced by IServiceProvider) but I'm missing how the IServiceProvider is acually injected into the flow.

To allow for the encapsulated ServiceProviders to be disposed. This was only to cover the case when the injected service providers scope was for the validation only. I agree this is an edge case and it could be handled manyally outside the scope of the validation process.

Do you mean you'll have seperate .net and core implementations or remove it for .net alltogether?

Not sure I understand the question. This will be part of the FluentValidation.AspNetCore package, which is a netstandard2.0 library. ASP.NET Core runs on both .NET Core and the older .NET Framework. This initial implementation is for web projects only.

I would only like to point out that InjectMessage(Func<IServiceProvider, IValidationContext, string>) and InjectPropertyName(Func<IServiceProvider, IValidationContext, string>) similar to InjectValidator would cover most cases I can think of

I feel that injecting string literals is not going to be a common scenario, so I'd like to be able to provide hooks for this so you (or others) can do this, without it needing to be part of the public API. I feel a custom IStringSource is the best way to do this, which you can make use of if you want to.

I'm missing how the IServiceProvider is acually injected into the flow.

For this initial (web-only) implementation, it'll automatically be injected into the context by the ModelValidatorProvider which is invoked transparently by MVC's validation infrastructure.
Initial code is here: https://github.com/JeremySkinner/FluentValidation/commit/fcff176f171f1d8bbd85e716a5bdf3699ca44bdb

I have rivisted and completely refactored https://github.com/nsarris/FluentValidation.IoC based on your 8.2 implementation. At it's current form it's basicly an in place replacement of your FluentValidation.DepedencyInjection project (minus the ASP.NET stuff)
Per your suggestion I've moved to Microsoft.Extensions.DependencyInjetion.Abstractions for DI and removed the DependencyResolver and the notions of scoping/lifecycle. and kept all your semantics in (e.g. InjectValidator). So it's basicly an augmented version of your implementation. I can send a PR that merges into FluentValidation.DepedencyInjection if you want.
If you want the check the source first here are the key points:

Syntax
There are a lot of extensions to make life easier, a lot of "Custom" overloads to reduce boilerplate and a UseDependecies() builder extension to make injecting syntax cleaner (check the Test.Common validators)

Validator factory. I had to create an augmented IValidatorFactory to add support for specific validator resolution (e.g. not IValidator but TValidator). I know it's a breaking change but I'm not aware of how many people acutaly use this interface, but it would be usefull to augment it. I also provide a default implementation which is a wrapper around the service provider (to facilitate injection automations).

DI tools: Added extensions for IServiceCollection and reflection based auto discovery of validators and scrutor based auto registration of validators.

ValidationContextProvider (previously IoCValidationContext). The previous version was disposable to handle scoping and lifecycle and thus a context. Since this was stripped out, this is now more clean as it's only a ValidationContextProvider/Builder/Factory. This also wraps around the service provider and is DI friendly (only needs an IServiceProvider).  It also comes with a lot of syntactic goodies to make life easier.

Literals. This is now a seperate set of extensions and can be removed, moved to a different namespace or be a seperate addon on its own.

Let me know what you think!

@nsarris I can't tell what your tests are testing / trying to test.  I suggest you break each assert into its own test case, and improve the naming of your tests so its not simply called Test.
    [TestFixture]
    class Tests
    {
        [Test]
        public void Test()
        {

@jzabroski
This is mostly proof of concept, it doesn't have proper testing and no documentation at all. I will try to put something together if it makes that much difference. I merely posted this early version here to see what Jeremy thinks so I can decide how to go through with it. Even so these very basic tests (I put them together in like 10 mins) have around 60% code coverage (it's all extensions, there is no actual execution logic, just wiring)

Hi @nsarris thanks for the update. It'll take me a while to get through all of this, but here's some initial thoughts.
I think we can merge some of this into the core with changes, but a lot of it I feel is over and above what I'd be prepared to support so would be better off as either something you continue to use in your own project, or a separate library if you feel like providing support for that.

DI tools: Added extensions for IServiceCollection and reflection based auto discovery of validators and scrutor based auto registration of validators.

I'd like to get extension methods on IServiceCollection for registering validators into the core, but it'll need some changes. At the moment we provide methods for registering validators as part of the call to AddFluentValidation:
services.AddMvc()..AddFluentValidation(fv => {
   fv.RegisterValidatorsFromAssemblyContaining<MyValidator>();
   // or:
   fv.RegisterValidatorsFromAssemblyContaining(typeof(MyValidator));
   // or:
   fv.RegisterValidatorsFromAssembly(typeof(MyValidator).Assembly);
});

Let's make the IServiceCollection extension methods follow the same convention.  (3 methods, RegisterValidatorsFromAssemblyContaining<T>, RegisterValidatorsFromAssemblyContaining(Type), RegisterValidatorsFromAssembly(assembly).).
Internally, these would use the existing FluentValidation.AssemblyScanner to do the work of locating the validators instead of ReflectionHelper.AutoDiscoverValidatorTypes.
I'm not keen on merging in the behaviour of ReflectionHelper.AutoDiscoverValidatorTypes - I feel that this level of mapping is beyond what FluentValidation should support out the box, and is very much heading into the territory of DI containers that perform automapping. This really is out of scope of what I want to support - at the end of the day, FluentValidation is a validation library not a DI library and I don't want to start having to support a whole load of code that isn't related to the core domain of validation. I'm also not keen on bringing across the ServiceLocator class for the same reasons. However I would be open to adding in extensibility points to AssemblyScanner to allow the logic to be pluggable, so you can swap out the default search behaviour if you want to.
Also, the default service lifetime for validators should always be Transient. Although singleton is better for performance, it adds a huge amount of confusion and complexity with ensuring that singleton validators don't depend on non-singleton (or web-scoped) services. This is a common pitfall for many developers. Making the default lifetime transient avoids this (and those that really know what they're doing can of course use singleton instead).

Validator factory. I had to create an augmented IValidatorFactory to add support for specific validator resolution

I don't think we'll bring this into the core as 1) it's a breaking change and 2) it'd only be necessary if we're for more complex auto registrations. If you need this behaviour, I'd suggest resolving the validators from the container directly and bypass the factory.

Literals. This is now a seperate set of extensions and can be removed, moved to a different namespace or be a seperate addon on its own.

Looks good, although as I've mentioned in previous posts I don't think the literal injection is something we want to support in the core project as this isn't a common scenario, and we provide the necessary extension points to do this.
So if you're interested in putting together a PR for the service collection extensions, then the first step would be the 3 Register* methods that delegate to AssemblyScanner to do the search and register the results with the container as transient by default.
A second step as a separate PR would be to add extension points to allow the AssemblyScanner's logic to be pluggable.

Thanks for the swift reply

I'm not keen on merging in the behaviour of ReflectionHelper.AutoDiscoverValidatorTypes - I feel that this level of mapping is beyond what FluentValidation should support out the box, and is very much heading into the territory of DI containers that perform automapping. This really is out of scope of what I want to support - at the end of the day, FluentValidation is a validation library not a DI library and I don't want to start having to support a whole load of code that isn't related to the core domain of validation. I'm also not keen on bringing across the ServiceLocator class for the same reasons. However I would be open to adding in extensibility points to AssemblyScanner to allow the logic to be pluggable, so you can swap out the default search behaviour if you want to.

I completely missed the existence of AssemblyScanner in the core. In fact I was really reluctant to implement any reflection code in my code as well for the same reasons you describe. I'll revert to AssemblyScanner.
The ServiceLocator I forgot to mention was supposed to be plugged into the static options if you'd want to merge it, to support non DI driven environments. I'm not really keen on service location either so I'm ok with ridding it.

Also, the default service lifetime for validators should always be Transient. Although singleton is better for performance, it adds a huge amount of confusion and complexity with ensuring that singleton validators don't depend on non-singleton (or web-scoped) services. This is a common pitfall for many developers. Making the default lifetime transient avoids this (and those that really know what they're doing can of course use singleton instead).

You're right about the default lifetime, I was thinking that with everything being injected through the context the validators wouldn't have any dependencies, but yes it's a pitfall. I'll revert to Transient.

I don't think we'll bring this into the core as 1) it's a breaking change and 2) it'd only be necessary if we're for more complex auto registrations. If you need this behaviour, I'd suggest resolving the validators from the container directly and bypass the factory.

Here's what got me buffled during my design. Is there a point to IValidatorFactory when you have DI? If yes then we should only register the IValidatorFactory. If not not then we should register validators so what's the point of IValidatorFactory? My current implementation registers validators and simply implements IValidatorFactory as a wrapper around IServiceProvider. I chose to get resolution through the factory partly because that's what I saw in your 8.2 code and partly because I'd hoped you'd agree to augment IValidatorFactory. So my suggestion is this: Use the service provider under all the InjectValidator methods and leave IValidatorFactory for the SetValidator methods. It's clear semanticaly and provides a clean layer on top of the ServiceProvider.

Looks good, although as I've mentioned in previous posts I don't think the literal injection is something we want to support in the core project as this isn't a common scenario, and we provide the necessary extension points to do this.

Like I said this can work as a separate library. If you want to endorse it, add a seperate repo in the FluentValidator family and I could PR it there.

So if you're interested in putting together a PR for the service collection extensions, then the first step would be the 3 Register* methods that delegate to AssemblyScanner to do the search and register the results with the container as transient by default. A second step as a separate PR would be to add extension points to allow the AssemblyScanner's logic to be pluggable.

Sure I can do that. For the AssemblyScanner to be pluggable there are three options. Inject it (fall back to default), service locate it (through static options) and provide it in the IServiceCollection extension (e.g. services.AddValidators(assemblyScanner)). All of the above can coexist. What's your pick?
All of the above cover evertything except all the rulebuilder extension methods I built for ease of use. What's your thoughts on them? Do you want some examples/documentation? Do you prefer to place them in seperate repo or maybe namespace?

Is there a point to IValidatorFactory when you have DI?

Not as much these days - it predates the built in ASP.NET ServiceCollection. But it still has its uses. For example, some users still rely on the [Validator(typeof(MyValidator))] attributes to wire up validators to models, rather than using the container, which is legacy but still supported (via the AttributedValidatorFactory). It allows instantiation of validators to be external to the regular DI process. Personally I wouldn't recommend using anything other than the container to instantiate validators, but the old approach is still supported (at least until FluentValidation 9 anyway). The default implementation of IValidatorFactory now just requests the container to resolve the service.
Starting with FluentValidation 9, I'll probably drop IValidatorFactory from the validation pipeline for ASP.NET Core, but the legacy ASP.NET support (MVC5/WebApi2) will continue to use IValidatorFactory as there's no native DI there (I may also just drop MVC5/WebApi2 support completely, but not decided yet)

So my suggestion is this: Use the service provider under all the InjectValidator methods and leave IValidatorFactory for the SetValidator methods. It's clear semanticaly and provides a clean layer on top of the ServiceProvider.

I'll have a think about that and get back to you, but if IValidatorFactory is going away with 9.x anyway, it may make sense to just leave it as-is for 8.x

All of the above can coexist. What's your pick?

I think I'd like to keep it as simple as possible. As the scanner is only used during startup, I want to avoid having to deal with injecting anything here. My initial thought would be to just allow passing a delegate to the AssemblyScanner. When specifed, this delegate would be executed to perform the scan rather than using the built-in logic. The service collection extension methods, would then allow this callback as an optional argument, passing it to the scanner.

All of the above cover evertything except all the rulebuilder extension methods I built for ease of use. What's your thoughts on them

Haven't had a chance to look at that yet, will try early next week.

I've released 8.2 with the dependency injection extensions & service collection extensions. It should show up in nuget shortly. I'm going to close this issue - please open new issues for future enhancements to this.


Adding libcurl as a dependency.

This is copied from #5387 and will be used by OpenCensus Zipkin
exporter and AWS credentials puller (#5215).
Signed-off-by: Ivan Vitjuk<EMAIL_ADDRESS>Description: Adding libcurl as a dependency.
Risk Level: Low
Testing: ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.dev'
Docs Changes: N/A
Release Notes: N/A

/retest

Closing as #5387 is already merged.


Geopackage support

add support for geopackage storage formats
http://www.geopackage.org/spec/

found an android friendly provider that provides tiles via google maps tile provider
https://github.com/ngageoint/geopackage-android

quick update, i was able to get this to work, however the performance was terrible when using it synchronously. when loading the tiles asynchronously, tiles just don't show up and i'm not entirely sure why.

I wrote a tool which enables you to preconvert geopack files into osmdroid sqlite databases. This more or less meets the objective. It can be found here: https://github.com/spyhunter99/geopkg4osmdroid

It would be cool to support geopackage directly in osmdroid: it is a standard and is supported by gdal, which permits conversion from a lot of formats. Maybe the performance of geopackage-android has improved during the last year?

Looking back, I think the issue was related to database indexing. There is
a project which converts raster tiles in geopackage into osmdroid format
(see the wiki/related projects). Including the GRL package with an example
in osmdroid is feasible, however vector data probably wont' be supportable
(at least easily)
On Wed, Jan 4, 2017 at 1:56 PM, legrostdg<EMAIL_ADDRESS>wrote:

It would be cool to support geopackage directly in osmdroid: it is a
standard and is supported by gdal, which permits conversion from a lot of
formats. Maybe the performance of geopackage-android has improved during
the last year?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/osmdroid/osmdroid/issues/180#issuecomment-270454783,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB4kGY8fbjJk8HqQf_l8dFvkJxX7LAgjks5rO-t7gaJpZM4GWhey
.

@legrostdg you sir, are in luck. I still have the on device code. I'll see if it can be updated and the performance issues fixed. Unless you want to give it a whirl? I'm a bit busy at the moment on other tasks

i think the reason why it wasn't originally merged has the version at the time had a lot of dependencies on google libraries, which i don't like and it sort makes understanding the licensing, complicated.
i also wanted to just included it with the samples osmdroid app, however the minsdk is 8 for osmdroid. the NGA stuff enforces a minsdk of 14. Enforcing that for the sample app would significantly reduce our test coverage unfortunately but it may be worth it. Maybe we can split up the sample app somehow to have a version for travis with a low minsdk but publish to the play store a version with everything in it.
https://github.com/ngageoint/geopackage-android/issues/18

@spyhunter99 unfortunately I reported this as the user, not as a coder, as I don't speak java... I would be happy to assist if it could be coded in haskell or python!
I think this worth increasing the minsdk but you are the one who decides :-).
Thanks a lot for taking time on this issue!


Unable to run on Safari due to normalize-url bug

Current version of normalize-url (6.1.0) has a regex bug that affects Safari users.
This is resolved in<EMAIL_ADDRESS>but the major version bump is due to converting the library from cjs to esm. I attempted using esm on this fork, but wasn't able to get the app running; I did however get the tests to pass.

I also tried updating normalize-url, however it seemed as if it would be too much hassles and I don't have the time. So, for now, I am sticking with version 6.1.0. Sorry!


Fixed compiler issues with Swift 5.8 and macCatalyst.

Removed xrOS references, and added checks that enable this package to be included in macCatalyst apps without compiler errors.  Also fixes so that there are no compiler errors when using Swift 5.8 or earlier.  Added an orientation property if on iOS that wraps the UIDevice current orientation if UIDevice is available.

@kudit I'm sorry but your PR contains a lot of different changes. I can't accept your PR unless split it up into separate PRs with smaller changes. I closed your other PR for the same reason:

https://github.com/devicekit/DeviceKit/pull/369

They're all related.  It's all checks to fix compiler issues when building for macOS (which doesn't support UIKit) and Swift 5.8 (which doesn't recognize xrOS or visionOS os( tags).  Only thing that might be able to be separated is the orientation code but other than that, all those checks are needed to remove compiler warnings.  It's a lot of code because there are so many places where UIKit is assumed and the alternate code path wasn't provided/implemented.  If you review, you'll see that it's all pretty simple and straight forward.   Is there a way to not include the generated file so that the code only needs to be updated in one place rather than two nearly identical files?


Adding a MUI Switch to the InputTypes

Hi,
Happy new year!
Do you think it'll be possible to have a Switch InputTypes?
Today, I'm using a select with true or false values.
            <FormControlLabel               labelPlacement="end"               control={                 <Switch                   checked={published}                   onChange={togglePublished}                 />               }               label="Publish"             />
It'd be like a Xmas present ;)
Thanks for all that scheduler module, it's awesome !
Best,

Hello
I think in this case its better you setup up your own editor, check customEditor in the readme.
you can setup your own custom fields/validations etc
Feel free to re-open if you face any issues


rewrite with egoscale v3

Description
We rewrite snap-o-matic with egoscale v3.
We disable the CI. It fails due to missing credentials which we don't intend to provide.
Backwards incompatible changes

logging to a file or syslog is no longer supported (the flags -l --log have been removed)
Previously the program checked whether a snapshot was created by snap-o-matic before deleting a snapshot. Now it just sorts the snapshots by created-at and deletes the oldest ones.

Testing
I tested running it locally and on an Exoscale compute instance(to make sure that getting the instance ID from the metadata server works.)
# on a compute instance:
> export EXOSCALE_API_KEY=...
> export EXOSCALE_API_SECRET=...
> export EXOSCALE_API_ENDPOINT=https://api-ch-dk-2.exoscale.com/v2

> ./snap-o-matic
2024/04/29 10:38:55 INFO looking up instance ID via metadata service
2024/04/29 10:38:55 INFO defaulting to host instance instance-id={UUID:228a4dc0-e2c9-480c-a37f-71370360af6d}
2024/04/29 10:38:55 INFO settings config="{APIEndpoint:https://api-ch-dk-2.exoscale.com/v2 SnapshotsRetention:7 DryRun:false InstanceID:{UUID:228a4dc0-e2c9-480c-a37f-71370360af6d} CredentialsFile: LogLevel:info}"
2024/04/29 10:38:55 INFO found snapshot id=8e80d3b6-d5c4-40e9-bf67-ddbd509bfba8 created-at=2024-04-29T10:32:03.000Z
2024/04/29 10:38:55 INFO found snapshot id=7fc55314-ca87-45a8-ad6a-c863a916a77b created-at=2024-04-29T10:30:06.000Z
2024/04/29 10:38:55 INFO found snapshot id=413164aa-13c0-4b07-aa00-c74f14e83263 created-at=2024-04-29T10:07:15.000Z
2024/04/29 10:38:55 INFO found snapshot id=a2d57b5f-ff67-46e0-b01f-9f7b87d264ed created-at=2024-04-29T09:59:24.000Z
2024/04/29 10:38:55 INFO found snapshot id=056270fe-3a95-4536-8aea-d9a8e9c1dbd2 created-at=2024-04-29T09:54:21.000Z
2024/04/29 10:38:55 INFO creating snapshot
2024/04/29 10:38:55 INFO waiting for operation to succeed operation-id=d625f0a3-739e-43a0-b8d5-d027843e773d command=get-snapshot
2024/04/29 10:39:28 INFO snapshot created successfully snapshot-id=d639e428-5b88-4757-a918-693e492984ee

make build and docker build . both work.

For me, you can go, just tried to run it multiple time from a newly created instance:
ubuntu@test:~/snap-o-matic$ ./bin/snap-o-matic
2024/04/29 15:11:57 INFO looking up instance ID via metadata service
2024/04/29 15:11:57 INFO defaulting to host instance instance-id={UUID:2efdbd5c-5ba3-4481-abfc-080fcb8acfc5}
2024/04/29 15:11:57 INFO settings config="{APIEndpoint:https://api-ch-gva-2.exoscale.com/v2 SnapshotsRetention:7 DryRun:false InstanceID:{UUID:2efdbd5c-5ba3-4481-abfc-080fcb8acfc5} CredentialsFile: LogLevel:info}"
2024/04/29 15:11:57 INFO found snapshot id=55edaee4-5d7a-4bfc-b198-20f9056e0355 created-at=2024-04-29T15:10:17.000Z
2024/04/29 15:11:57 INFO creating snapshot
2024/04/29 15:11:57 INFO waiting for operation to succeed operation-id=b65988b5-a90c-48b2-a54c-9e469fb0da7c command=get-snapshot
2024/04/29 15:12:27 INFO snapshot created successfully snapshot-id=0aefbe5e-5eae-44b7-a853-f3abd00cc959
ubuntu@test:~/snap-o-matic$

You can ship for me


IPP-USB Gadget Support

Develop a Linux USB gadget and corresponding support code in PAPPL to allow communication with the default printer in PAPPL using the IPP-USB protocol. References:

Kernel Gadget API
IPP-USB specification

There are two parts to this:

[ ] A kernel driver, either an updated version of the USB printer gadget or a new g_ippusb gadget. The gadget needs to expose at least three pairs of endpoints to support printing and scanning simultaneously. Descriptor limits may be an issue.
[ ] Support code in PAPPL to basically map the endpoints to local sockets, either localhost or (better) a domain socket listener.

This code should coexist with the USB printer gadget - the client device can only select the legacy printer class or the new IPP-USB class (not both), but the PAPPL code needs to properly handle that selection.

OK, so I've done some prototyping using the FunctionFS gadget driver which allows me to export N instances of IPP-USB descriptors and endpoints (tested with N=3 which is the typical configuration for printers), and then no need to do a separate kernel driver!
 But it looks like it works well.

Also need the following changes for macOS:

[ ] Support for Set-Printer-Attributes over localhost
[ ] Support for printer-wifi-ssid, printer-wifi-password (write only), and printer-wifi-state attributes
[ ] Callbacks to get current Wi-Fi state and to try connecting to a specified Wi-Fi network.

Additional issues found during testing:

[ ] Wi-Fi test harness in testpappl
[ ] When connected over localhost, only advertise http/ipp URLs
[ ] Should use domain socket when available (but need to store the domain socket path in the server object)

This work is now done.


API changes

There are three types of constructor now, I do think it's a good idea to combine those three overloading into one single constructor, include the metadata file path in the object instead of an argument. e.g.
var IdP = new IdentityProvider({
      privateKeyFile: './privateKey.pem',
      privateKeyFilePass: 'myPassword',
      metadata: './metadata_idp.xml'
});

Remarks: According to the semantic versioning 2, it changes the API and not plan to be backward compatibility, that's why it's bumped to v2.0.

It will be the breaking change in v2.0, and it is  confirmed to included in the first release of v2.0.


[express-apollo-prisma] Add database migration

Description

It should provide seeding and migrations as an example of how to best utilize the tooling provided to do repeatable data tasks.

updated Acceptance Criteria:
having changes from PR: https://github.com/thisdot/starter.dev/pull/1081

add the Seeding migration script (Docs: https://www.prisma.io/docs/guides/database/seed-database)

https://www.prisma.io/docs/guides/database/seed-database


How to design a "Pandora Schema" with Collection2?

I have a small personal works.
One logic of this project is a main stream. Everythings like a new posts / events or somethings will generate a snapshot and contained in this stream (Like notifications).
I guess this snapshot is not a generic data type but help to up query performance.
So how i design this schema?

I see the blackbox


Implemented Indexer for Search transaction

Description
Features

Implemented Indexers to query transactions.
Updated .env with analog urls' i.e networks and indexers
Issues.
network options are null and are unused in code. Commented for now.
Removed node version 12.x workflow config from build.yaml file. i.e it's not supported now.

looking good. however in Search.tsx there is a problem. It uses isTransaction and isAddress which only work on ethereum. These need to be generalized somehow. One way to do it would be to try the query as a transaction, and if it doesn't find anything try the query as an address


Generate type definitions for Client Libraries

We already have defined types that are potentially going to be shared across the client and server libraries in the types crate.
This issue will try to create underlying type definitions to be used by client libraries.
Serde Reflection
Serde Generate
Acceptance Criteria
Test that the client(mini implementation) and server are both able to communicate using the types generated.
i:e: generated types in a language are send and properly interpreted by the rust server.

APPROACH.

[ ] Using Serde reflection generate a spec document(yaml file) to be used by other languages
[ ] Write tests to verify that the yaml spec document is up to date with the types crate.
[ ] Generate types implementation in other language(python for now)
[ ]


Command Payload and interaction not resolving properly

Following your examples and docs, I tried to implement a command with the newly supported slash commands. However, the arguments passed to the handler, doesn't seem to resolve properly.
After running !deploy once the bot has started, the command below also doesn't display/require the song param in Discord itself.
The following example will log (interaction, undefined) from the handler call.
import { TransformPipe } from '@discord-nestjs/common';
import {
  Command,
  DiscordTransformedCommand,
  Payload,
  UsePipes,
  Param,
} from '@discord-nestjs/core';
import { Logger } from '@nestjs/common';
import { CommandInteraction } from 'discord.js';

// Dto Definition
class PlayDto {
  @Param({
    name: 'song',
    description:
      'Name or URL of song/playlist. Could be from (Youtube, Spotify, SoundCloud)',
    required: true,
  })
  song: string;
}

// Command
@Command({
  name: 'play',
  description: 'Plays a song',
})
@UsePipes(TransformPipe)
export class PlayCommand implements DiscordTransformedCommand<PlayDto> {
  private readonly logger = new Logger(PlayCommand.name);

  handler(@Payload() dto: PlayDto, interaction: CommandInteraction) {
    console.log({
      dto,
      interaction,
    });

    return 'Nice.';
  }
}

Please give me a shout if you need additional information!

I know that this issue has been closed and solved. I had the same issue and discovered that I was missing "useDefineForClassFields": true in my tsconfig.json

I know that this issue has been closed and solved. I had the same issue and discovered that I was missing "useDefineForClassFields": true in my tsconfig.json

Yes, this solution has been added to the readme back then : https://github.com/fjodor-rybakov/discord-nestjs#i-created-dto-and-added-transformpipe-but-when-i-receive-response-to-the-command-the-dto-fields-are-missing
But having it surfaced here too isn't bad either :)


[QUESTION] what is the benefit of the Ref type

@szokodiakos
Hi! Could you please explain the benefit of the Ref type as it is defined by Typegoose as T or string ? Maybe you have code examples where it is used to process an unpopulated/ populated reference property?
As far as I can see, Mongoose returns something like ObjectID {_bsontype: "ObjectID", id: Buffer(12)} for an unpopulated reference property - so not a string.
Therefore I don't see why Typegoose defines Ref as T | string ...
Thanks a lot & best regards,
Marcel

I think you're right, it should be
type Ref<T> = T | ObjectID;

Fixed in v5.2.1


(BUG) The screen is absent after scanning unsuccessfully three times a row

Precondition:

The face is scanned unsuccessfully three times a row
Steps to reproduced:

Scan face one more time (unsuccessfully again)
Pay attention to the absence of the screen for those who can't pass verification for 3 times
Click the "Contact support" button and then the back arrow button
Pay attention to the screen for those who can't pass verification for 3 times (it is displayed now)

Actual result: The screen is absent after scanning unsuccessfully three times a row (Look at the video)
Expected result:  The screen should be displayed after scanning unsuccessfully three times a row
Screen

Checked on QA V0.22.1-2   DEV 0.22. 1-1
LG V30+ // Android 8.0 // Google Chrome 81.0.4044.138
Desktop: Windows 10 // Google Chrome 83.0.4103.61
Video: https://www.screencast.com/t/jcwmkZ45o81

The user gets the "Sorry about that..." screen on the third unsuccessful attempt
Checked on DEV/QA V0.22.1-2
LG V30+ // Android 8.0 // Google Chrome 81.0.4044.138
Desktop: Windows 10 // Google Chrome 83.0.4103.61


Juniper Switches Syslog Parser

What is the sc4s version?
3.27.0
Is there a pcap available? If so, would you prefer to attach it to this issue or send it to Splunk support?
I can provide this through a support case.
What the vendor name?
Juniper
What's the product name?
Switch
If you're requesting support for a new vendor, do you have any preferences regarding the default index and sourcetype for their events?
Existing vendor
Do you have syslog documentation or a manual for that device??
Feature Request description:
our customer have a large fleet of Juniper Switches, we've got SC4S configured and we're getting the logs into Splunk, but 90% of the logs are coming as nix:syslog. Digging into the "Splunk Add-on for Juniper", it doesn't really do much for switches, it focuses more on firewalls. Please assist.
Do you want to have it for local usage or prepare a github PR?
github PR

Hi @marakji-splunk
Could you please create a support case and provide the pcap file?
Thanks.

Apologies for the delay, please give me some time as I'm still figuring out
a way to do the PCAP, i'm running my SC4S on microk8s.
Thanks,
On Mon, Jul 29, 2024 at 7:37 PM cwadhwani-splunk @.***>
wrote:

Hi @marakji-splunk https://github.com/marakji-splunk
Could you please create a support case and provide the pcap file?
Thanks.
—
Reply to this email directly, view it on GitHub
https://github.com/splunk/splunk-connect-for-syslog/issues/2535#issuecomment-2255460685,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A4HS7RC3OCBCLZ7HWAKEHLDZOYEN7AVCNFSM6AAAAABLJMZOOGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJVGQ3DANRYGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Mustapha Arakji
Senior Professional Services Consultant
Splunk Inc.
+61 421 821 634  |  @.***
SYDNEY | AMER | APAC | EMEA
[image: Splunk: Resilience. Let's build it together.]
https://www.splunk.com/
This message is intended only for the personal, confidential, and
authorized use of the recipient(s) named above. If you are not that person,
you are not authorized to review, use, copy, forward, distribute or
otherwise disclose the information contained in the message.

Closing this GitHub issue, due to unavailability of the PCAP file. If it is still not resolved, please feel free to reopen this case once a support ticket is created with the PCAP file attached. Thanks!


Just now added corresponding registry key  and  download link for tem…

…plates
I taken example from windows 10  v1903, i found  the corresponding registry keys  for   hide all  notifications  and  hide  non-critical notifications
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender Security Center\Notifications]
"DisableNotifications"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender Security Center\Notifications]
"DisableEnhancedNotifications"=dword:00000001
Also  i added  the download link for latest administrative templates for Windows 10 v1809.
So please add these registry keys and download link as my contribution in this document.
Thanking you

Just a small note and advice regarding PR titles (for future use): skip references to when the PR was created (now, just now, right now, and so on) as well as skipping personal pronouns (I, me, myself). PR and issue titles should be short, concise and technical, not personal notes.

@nenonix @JohanFreelancer9 Copy editing is needed for this PR. Thanks.

@dansimp A community contributed PR that has passed copy review is available for technical review. Thanks.

@dansimp A community contributed PR that has passed copy review is available for technical review. Thanks.

@Dansimp Following up for updats.

@Dansimp Following up for updates.


How data age is calculated for retention purposes

Documentation does not mention how data age is calculated. Is TimeGenerated field being used or some other internal field?
Data collecter API contains time-generated-field header to set TimeGenerated. If user is setting this value older than configured retention limit will the data be pruned as soon as pruning process is started after data import?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a73a8a62-95c6-9ec9-9199-695ad1e5e8a9
Version Independent ID: 6089f013-8b5b-661c-9ee4-d6b267737357
Content: Manage usage and costs for Azure Monitor Logs - Azure Monitor
Content Source: articles/azure-monitor/platform/manage-cost-storage.md
Service: azure-monitor
Sub-service: ****
GitHub Login: @bwren
Microsoft Alias: bwren

@jikuja Thanks for your comment. We will review the issue and get back to you shortly.

@jikuja Log Analytics data retention can be configured between 30 and 730 days in all current pricing tiers. Data which is older than the retention setting will be removed (based on the TimeGenerated field).

@jikuja As there are no documentation changes, we will close the issue for now. If there are further questions , please revert and we will be glad to assist you.


Docs/links

What does this PR do?
update test grid and Docs
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

Codecov Report

Merging #106 into master will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #106      +/-   ##
==========================================
- Coverage   92.52%   92.50%   -0.03%
==========================================
  Files          75       75
  Lines        3827     3816      -11
==========================================
- Hits         3541     3530      -11
  Misses        286      286

Flag
Coverage Δ

#unittests
92.50% <ø> (-0.03%)
:arrow_down:

Impacted Files
Coverage Δ

...bolts/models/self_supervised/amdim/ssl_datasets.py
71.64% <0.00%> (-1.22%)
:arrow_down:

..._bolts/models/self_supervised/moco/moco2_module.py
84.34% <0.00%> (-0.40%)
:arrow_down:

pl_bolts/models/rl/common/wrappers.py
92.52% <0.00%> (-0.21%)
:arrow_down:

pl_bolts/datamodules/mnist_datamodule.py
97.77% <0.00%> (-0.05%)
:arrow_down:

pl_bolts/losses/self_supervised_learning.py
97.43% <0.00%> (-0.02%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1cbe585...63b3c64. Read the comment docs.


replace content-type-parser with whatwg-mimetype

Any idea why this causes so many test failures?

Done in https://github.com/jsdom/jsdom/commit/d5639655d32d45b6690c9ac30901136e650eb99a


Feat/lmstudio integration

Description
Add Support for LM Studio
Resolves #13556
New Package?
Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

[x] Yes
[ ] No

Version Bump?
Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

[x] Yes
[ ] No

Type of Change
Please delete options that are not relevant.

[x] New feature (non-breaking change which adds functionality)

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[x] Added new notebook (that tests end-to-end)

Suggested Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] I have added Google Colab support for the newly added notebooks.
[x] My changes generate no new warnings
[x] New and existing unit tests pass locally with my changes
[x] I ran make format; make lint to appease the lint gods

@logan-markewich Let me know if we have additional requirements before the merge into main.


gx_save + gx_restore to save and restore .RHistory and .RData

Make the convience gx_save more convenient by also storing the .RHistory file and allow .RData and .RHistory to be restored by gx_restore.

LGTM. Thanks  @scholtalbers.

Thanks @scholtalbers I'd been meaning to get around to this for a long time now.

Thanks to @erasche comments I understood what you meant @bgruening - thought somehow i forgot to wrap something in the function :)

:+1:


Unable to load admin login

Error message
ErrorException: Trying to get property 'user_id' of non-object
in /.../public_html/v3/app/admin/classes/User.php:58
As the user variable has no object yet, it cannot get the property, catching scenario with a Null coalescing operator and I will investigate further at a later date.
This has now allowed me to log in to the admin panel

You got this problem after update from .19 to .20?

You got this problem after update from .19 to .20?

Yes

Thank you, already fixed here tastyigniter/ti-ext-broadcast@4b9ea73f667bc5be543bce77ee3488fba6dde9b8


Implement /object/patch/add-link

Hi, thank you for this wonderful library! For a project I'm working on I need to use "/object/patch/add-link" which still had a TODO, so I implemented it. The groundwork you've already laid made that very easy and straightforward, so thank you, and good job.
If this needs any fix ups to meet your standards, let me know. Thanks!

Looks good to me! Glad it was easy to contribute!
One small thing, do you mind just running cargo fmt so the checks pass. Ty again!

Looks good to me! Glad it was easy to contribute!
One small thing, do you mind just running cargo fmt so the checks pass. Ty again!

Sorry about that, I thought I had already! Done now.


Prevent deadlock if working on non-main thread

Fixes #2213.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Fixes #4 and #5

Implements -v/--version flags for showing the version number
Implements -h/--help flags for showing helptext

:+1: Beautiful.


Bad Closure Assumption

The current closure implementation assumes the first nested algorithm/lambda can promote a value from the stack to the heap. This works in trivial cases, but fails if the first nested algorithm is further nested in a conditional statement. In this case the value is never promoted, eventually resulting in a crash due to invalid access on read.
closure_crash.txt
One approach is to better understand Bob's solution and if it is applicable to a tree-walk interpreter using a stack for variables. http://www.craftinginterpreters.com/closures.html
Another approach is to use two-pass symbol resolution.

I think the two-pass approach is preferable. The first pass will gather information for the document map, the closure depth and lexical depth of declaration, and other properties about the variable, including eventually the scope for scope-based autocompletion. It should also make a list of all accesses.
The second pass will handle the details of stack offset or closure index depending on the variable type. This step will potentially come much later in the process, as not every nested read needs to be closed over, and making this distinction depends on optimization passes.

This bug has finally been addressed. There are still some questions about the behaviour of closures, but they are much less dire.


proposal: pkg/log: add flag to show relative filename paths.

many terminals allow clickable paths to open the file in the default editor and all recognise relative paths.
I propose to add a flag to show a relative path : Lrelativefile in
https://golang.org/pkg/log/#pkg-constants
Using Llongfile renders the log lines too long and in most cases wraps the text in more than a single line.

good point.
I just checked and vscode would make it clickable with : The working directory of the terminal.

What if either change during the execution of the program?

In reality when you run the binary in foreground you can't change the terminal directory anyway and when you run it in background this is even less relevant, no?

I understand the rationale but it seems a bit surprising and special purpose.
The go command does this, but the go command spends almost all its time printing these kinds of messages. It's not clear that every binary should do it.
In general I don't think most commands should change their output based on the directory where they are run.
/cc @robpike

got it thanks!


module stopped my running application

browser freezes on the page that uses this module, when I remove it,  everything works


First-run "Thank you for choosing GitLens! [...]" info message shown on every start up

GitLens Version: v4.3.3
VSCode Version: 1.15 (July 2017)
OS Version: macOS Sierra 10.12.6

Steps to Reproduce:

Start VSCode
Observe the first-run "Thank you for choosing GitLens! [...]" info message pop down

Not experiencing the issue with VSCode-insiders 1.16 (August 2017) on the same machine.
Reinstalling GitLens does nothing.

@damsleth do you have any other issues with that instance of vscode?

@damsleth Can you open up the devtools in vscode (Help -> Toggle Developer Tools), and then select Application tab at the top, and then expand Local Storage and select file://. Using the filter bar at the top type global/eamodio.gitlens and then please send me the value of the only key.
It should look like this:

storage://global/eamodio.gitlens : {"gitlensVersion":"4.3.3"}
i.e. no "suppressWelcomeNotice":true.

@damsleth this is so strange -- since the version number is in there and that is what I key off. See:
https://github.com/eamodio/vscode-gitlens/blob/d3013a03d471a1ff5cc14853a2899924f0f195d7/src/extension.ts#L273-L278
Using the devtools again can you delete the key (press the del key) and then restart vscode and see if the welcome message goes away on the 2nd run?

Deleted key, closed and restarted VSCode twice, key value still the same {"gitlensVersion":"4.4.2"}

@damsleth I'm so sorry this is still dragging on. I'm really baffled. I've added more logging in v4.4.3 so hopefully that will help point us in the right direction. Thank you for your patience.
Before you upgrade to 4.4.3 -- can you set "gitlens.outputLevel": "verbose" and then after you upgrade please post the output from the GitLens output channel?

Don't worry about it, it's not exactly a blocker ;)
Here's the verbose GitLens output (same for first upgrade-run and all subsequent ones).
GitLens(v4.4.3) active: /Users/damsleth/Code/ Git found: 2.8.4 (Apple Git-73) @ /usr/bin/git git -c core.quotepath=false rev-parse --show-toplevel   cwd='/Users/damsleth/Code' Git version: 2.8.4 (Apple Git-73) GitLens first-time install CodeLens config changed; resetting CodeLens provider getRemotes('/Users/damsleth/Code') getRemotes('/Users/damsleth/Code') git -c core.quotepath=false remote -v   cwd='/Users/damsleth/Code' git -c core.quotepath=false remote -v   cwd='/Users/damsleth/Code' getStash('/Users/damsleth/Code') git -c core.quotepath=false stash list --name-status --full-history -M --format=%H -%nauthor-date %ai%nreflog-selector %gd%nsummary %B%nfilename ?   cwd='/Users/damsleth/Code'

@damsleth I've just published GitLens 5.0.0-alpha.2. I've added a secondary check to suppress the welcome message -- hopefully this will resolve the issue (🤞 as I have no idea why reading from the globalState doesn't seem to be working in this case).
Please make sure to completely remove any prior version before installing a vsix -- instructions here: https://github.com/eamodio/vscode-gitlens/wiki/Installing-Pre-release-Versions


FOSDEM 2024 Confidential Computing Devroom - Feb 4, 2024

Not sure how active Enarx is at the moment, but dropping this here anyway. It may be interesting for the community to have an update on the current state and future of the project!
Info
https://falder.org/fosdem24-cfp
Dates

CFP closes:    2024-12-01

Location
Brussels, Belgium
CFP
URL to the Call for Proposals / Papers: https://falder.org/fosdem24-cfp

This looks fantastic, tagging @nickvidal; I probably cannot travel.


[SPARK-34796][SQL][3.1] Initialize counter variable for LIMIT code-gen in doProduce()

What changes were proposed in this pull request?

This PR is to fix the LIMIT code-gen bug in https://issues.apache.org/jira/browse/SPARK-34796, where the counter variable from BaseLimitExec is not initialized but used in code-gen. This is because the limit counter variable will be used in upstream operators (LIMIT's child plan, e.g. ColumnarToRowExec operator for early termination), but in the same stage, there can be some operators doing the shortcut and not calling BaseLimitExec's doConsume(), e.g. HashJoin.codegenInner. So if we have query that LocalLimit - BroadcastHashJoin - FileScan in the same stage, the whole stage code-gen compilation will be failed.
Here is an example:
  test("failed limit query") {
    withTable("left_table", "empty_right_table", "output_table") {
      spark.range(5).toDF("k").write.saveAsTable("left_table")
      spark.range(0).toDF("k").write.saveAsTable("empty_right_table")

      withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
        spark.sql("CREATE TABLE output_table (k INT) USING parquet")
        spark.sql(
          s"""
             |INSERT INTO TABLE output_table
             |SELECT t1.k FROM left_table t1
             |JOIN empty_right_table t2
             |ON t1.k = t2.k
             |LIMIT 3
             |""".stripMargin)
      }
    }
  }

Query plan:
Execute InsertIntoHadoopFsRelationCommand file:/Users/chengsu/spark/sql/core/spark-warehouse/org.apache.spark.sql.SQLQuerySuite/output_table, false, Parquet, Map(path -> file:/Users/chengsu/spark/sql/core/spark-warehouse/org.apache.spark.sql.SQLQuerySuite/output_table), Append, CatalogTable(
Database: default
Table: output_table
Created Time: Thu Mar 18 21:46:26 PDT 2021
Last Access: UNKNOWN
Created By: Spark 3.2.0-SNAPSHOT
Type: MANAGED
Provider: parquet
Location: file:/Users/chengsu/spark/sql/core/spark-warehouse/org.apache.spark.sql.SQLQuerySuite/output_table
Schema: root
 |-- k: integer (nullable = true)
), org.apache.spark.sql.execution.datasources.InMemoryFileIndex@b25d08b, [k]
+- *(3) Project [ansi_cast(k#228L as int) AS k#231]
   +- *(3) GlobalLimit 3
      +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [id=#179]
         +- *(2) LocalLimit 3
            +- *(2) Project [k#228L]
               +- *(2) BroadcastHashJoin [k#228L], [k#229L], Inner, BuildRight, false
                  :- *(2) Filter isnotnull(k#228L)
                  :  +- *(2) ColumnarToRow
                  :     +- FileScan parquet default.left_table[k#228L] Batched: true, DataFilters: [isnotnull(k#228L)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/chengsu/spark/sql/core/spark-warehouse/org.apache.spark.sq..., PartitionFilters: [], PushedFilters: [IsNotNull(k)], ReadSchema: struct<k:bigint>
                  +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, false]),false), [id=#173]
                     +- *(1) Filter isnotnull(k#229L)
                        +- *(1) ColumnarToRow
                           +- FileScan parquet default.empty_right_table[k#229L] Batched: true, DataFilters: [isnotnull(k#229L)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/chengsu/spark/sql/core/spark-warehouse/org.apache.spark.sq..., PartitionFilters: [], PushedFilters: [IsNotNull(k)], ReadSchema: struct<k:bigint>

Codegen failure - https://gist.github.com/c21/ea760c75b546d903247582be656d9d66 .
The uninitialized variable _limit_counter_1 from LocalLimitExec is referenced in ColumnarToRowExec, but BroadcastHashJoinExec does not call LocalLimitExec.doConsume() to initialize the counter variable.
The fix is to move the counter variable initialization to doProduce(), as in whole stage code-gen framework, doProduce() will definitely be called if upstream operators doProduce()/doConsume() is called.
Note: this only happens in AQE disabled case, because we have an AQE optimization rule EliminateUnnecessaryJoin to change the whole query to an empty LocalRelation if inner join broadcast side is empty with AQE enabled.
Why are the changes needed?

Fix query failure.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Added unit test in SQLQuerySuite.scala.

cc @maropu could you help take look when you have time, thanks.
The PR has exactly same change with https://github.com/apache/spark/pull/31892.

Test build #136298 has started for PR 31911 at commit 519ac59.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40880/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40880/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40880/

Test build #136298 has finished for PR 31911 at commit 519ac59.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136298/

retest this please

Test build #136302 has started for PR 31911 at commit 519ac59.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40884/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40884/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40884/

Test build #136302 has finished for PR 31911 at commit 519ac59.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Thanks! Merged to branch-3.1.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136302/

Thank you @maropu for review!


Feature request: Open file at location

Many text editors include the option to open a file at a give line and column number.  This is useful with SyncTeX-aware PDF viewer, for example -- the viewer can open the source TeX file at the location corresponding to a given place in the PDF file.  Light Table should have a command line option to specify the line and column number, and either open the file at that location or move to that location in the already-open file.

Opening files to a line is available with 0.8.0-alpha - https://groups.google.com/forum/#!topic/light-table-discussion/rkfHTplo4ss. Happy to take a PR for columns e.g. FILE:LINE:COL


Use variables for CSS colors

Preferably CSS variables (at root) and not Sass variables, so that the user can change colors in their client.

this is a good idea and something i should have done in the first place T_T


Merge upstream Prometheus at f9d9b9d

This PR merges the changes below from upstream.
There were merge conflicts in tsdb/db_test.go and tsdb/head.go - please review these carefully.
Performance and resource utilisation improvements

https://github.com/prometheus/prometheus/pull/14525
https://github.com/prometheus/prometheus/pull/14470

Minor changes and refactoring

https://github.com/prometheus/prometheus/pull/13503

Bug fixes

https://github.com/prometheus/prometheus/pull/14538
https://github.com/prometheus/prometheus/pull/14514
https://github.com/prometheus/prometheus/pull/14515
https://github.com/prometheus/prometheus/pull/14517
https://github.com/prometheus/prometheus/pull/14496

Non-production changes

https://github.com/prometheus/prometheus/pull/14537

Not relevant to Mimir

https://github.com/prometheus/prometheus/pull/13458
https://github.com/prometheus/prometheus/pull/14450
https://github.com/prometheus/prometheus/pull/14346
https://github.com/prometheus/prometheus/pull/14382
https://github.com/prometheus/prometheus/pull/14506

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.6 out of 12 committers have signed the CLA.:white_check_mark: bboreham:white_check_mark: pracucci:white_check_mark: colega:white_check_mark: krajorama:white_check_mark: cstyan:white_check_mark: charleskorn:x: tylitianrui:x: kushalShukla-web:x: maxamins:x: bwplotka:x: mmorel-35:x: fpetkovskiYou have signed the CLA already but the status is still pending? Let us recheck it.

LGTM, thanks.
I don't see where the conflicts in head.go and db_test.go could come from 🤔 , the changes there are straightforward. I guess it's "git stuff"


method refactoring

isset is a bit faster than array_key_exists.
the difference is what you get when a key exists and the corresponding value is null.
//if ($this->attributes['somekey'] === null)
// $defaultValue = 'something';
// 1) array_key_exists => getAttribute('somekey', $defaultValue) => null
// 2) isset            => getAttribute('somekey', $defaultValue) => 'something'

when i use this pattern i always assume i want to get some not null default value if the real value is null.

It's a micro-optimization at best, and if the associated value is falsy, I want to return it verbatim, as such, the current iteration is correct for my intentions.

good! I wasn't sure if it was intentionally implemented to return element values set to null. regards.