[SPARK-10780][ML] Add an initial model to kmeans

https://issues.apache.org/jira/browse/SPARK-10780
This PR aims to add warm-start to KMeans algorithm.

Test build #50935 has started for PR 11119 at commit 36b1729.

Test build #50935 has finished for PR 11119 at commit 36b1729.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50935/
Test PASSed.

re the first question - I don't think this necessarily needs to be a code generated param (although if we do end up having more shared params with templated types we should definitely do codegen). For now maybe just a hand written HasInitialModel seems fine (although I'd put it in a separate file rather than tacking it on the end of the generated code) - but thats just my personal thoughts. Maybe @dbtsai can chime in too?

Agree, for code-gen, if we want to do it in this way, we would rather put them in a separate place. But will be nice to extend the code-gen framework so it can use one codebase to handle generic type.
+@jkbradley @mengxr  BTW, we still need to run the separate sbt code to do code-gen, and why don't we do it in the compile time using quasiquote? This will not hurt the performance since it's compile time.

Test build #51090 has started for PR 11119 at commit 166a6ff.

Test build #51090 has finished for PR 11119 at commit 166a6ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51090/
Test PASSed.

@yinxusen I'll be away for Spark summit east. Gonna work on this again when I'm back. Thanks.

Never mind, take your time.
2016年2月12日星期五，DB Tsai<EMAIL_ADDRESS>写道：

@yinxusen https://github.com/yinxusen I'll be away for Spark summit
east. Gonna work on this again when I'm back. Thanks.
—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/11119#issuecomment-183248930.

--
Cheers
Xusen Yin    (尹绪森)
LinkedIn: https://cn.linkedin.com/in/xusenyin

Ping @dbtsai Coming back? :)

Yes, but busy on work. :( Will soon start it in couple days.

Ping @dbtsai

Test build #52578 has started for PR 11119 at commit f56e443.

Test build #52578 has finished for PR 11119 at commit f56e443.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52578/
Test FAILed.

Test build #52591 has started for PR 11119 at commit f3f9226.

Test build #52591 has finished for PR 11119 at commit f3f9226.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52591/
Test PASSed.

test it please

Test build #52711 has started for PR 11119 at commit 08afa4c.

Test build #52711 has finished for PR 11119 at commit 08afa4c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52711/
Test PASSed.

Test build #52816 has started for PR 11119 at commit 31f7b94.

test it please

Test build #52816 has finished for PR 11119 at commit 31f7b94.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52816/
Test PASSed.

Merged build finished. Test PASSed.

Test build #53096 has started for PR 11119 at commit 7c1c8f7.

@dbtsai And as discussed in the JIRA with @MLnick, I think we need to override the default parameters with the initialModel's.

Test build #53096 has finished for PR 11119 at commit 7c1c8f7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53096/
Test PASSed.

@dbtsai @MLnick
I prepare not to add the parameter overriding feature in this PR since it has other issues related, such as making the fit() modify the class parameters causes side-effects, which looks weird and not function programming style, and makes users confused. Let's add another JIRA issue of it after we solving this one.

I'm also a fan of overriding other Params, but am Ok with it being in another JIRA & PR.

@jkbradley Sure. I'll check them later.

I'll create new JIRA on params overriding.

test it please

Test build #53171 has started for PR 11119 at commit 95627b5.

Test build #53171 has finished for PR 11119 at commit 95627b5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53171/
Test PASSed.

@holdenk @dbtsai Any more thoughts on this?

Sorry for the delayed review.  I'm worried that these PRs to set the initial model may need to change significantly if we merge Estimator & Model: [https://issues.apache.org/jira/browse/SPARK-14033]  After the merge, setting an initial model may make less sense, for it would then be easier to copy the initial model and set some Boolean parameter indicating that fitting should begin from the current model.

Never mind, let's close this if that one merged.
Sent from my iPhone

On Mar 23, 2016, at 18:35, jkbradley<EMAIL_ADDRESS>wrote:
Sorry for the delayed review. I'm worried that these PRs to set the initial model may need to change significantly if we merge Estimator & Model: [https://issues.apache.org/jira/browse/SPARK-14033] After the merge, setting an initial model may make less sense, for it would then be easier to copy the initial model and set some Boolean parameter indicating that fitting should begin from the current model.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

@yinxusen Well, I still think we should support initial models, and hopefully some of the code (like unit tests) would be reusable.  Feel free to leave this open until you can see how much needs to be changed.

@jkbradley Sure, I'll leave it here and check back later. :)

@yinxusen since it was decided not to proceed with SPARK-14033, shall we resurrect this PR? It would be great to get the basic support for initial model in so that it can then be extended and support added in other estimators.

@MLnick Of course. Go ahead please.

@yinxusen sorry, I meant we should aim to continue review of this PR and hopefully merge it. Ping @jkbradley @dbtsai @holdenk

@MLnick Ah.. Sorry, my first understanding is wrong. Yeah, we should resurrect it since SPARK-14033 stopped for now. I'll resolve the conflicts soon.

test it please

Test build #55959 has started for PR 11119 at commit a0629b5.

Test build #55959 has finished for PR 11119 at commit a0629b5.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55959/
Test FAILed.

This will be very useful for many use cases. Nice to have it in 2.0 :)

retest it please

@jkbradley @MLnick It's strange that the PR failed Mima test. Do you have any thoughts about it?

Test build #56734 has started for PR 11119 at commit b0eb111.

Test build #56734 has finished for PR 11119 at commit b0eb111.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56734/
Test FAILed.

Merged build finished. Test FAILed.

Test build #56736 has started for PR 11119 at commit 58bf1cf.

Test build #56736 has finished for PR 11119 at commit 58bf1cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56736/
Test PASSed.

@jkbradley @mengxr I updated the MLReadable/MLWritable to previous DefaultMLReadable/DefaultMLWritable to prevent Mima failure. Can we catch it in 2.0?

Test build #57209 has started for PR 11119 at commit 914d319.

Test build #57210 has started for PR 11119 at commit c40192b.

Test build #57209 has finished for PR 11119 at commit 914d319.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57209/
Test PASSed.

Test build #57210 has finished for PR 11119 at commit c40192b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57210/
Test PASSed.

ping! @yinxusen @dbtsai @MLnick It would be nice to get this in so we can add it to other models.

Test build #64398 has started for PR 11119 at commit c40192b.

Test build #64398 has finished for PR 11119 at commit c40192b.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64398/
Test PASSed.

Test build #64581 has started for PR 11119 at commit 6526c08.

Test build #64581 has finished for PR 11119 at commit 6526c08.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64581/
Test FAILed.

Test build #64705 has started for PR 11119 at commit 23a78d6.

Test build #64705 has finished for PR 11119 at commit 23a78d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64705/
Test PASSed.

@yinxusen Thanks for the continued work on this. I made a pass, looking forward to this in 2.1!

Thanks @sethah for reviewing. @yinxusen once it's ready to merge, please remove WIP and I will be more than happy to do a final pass, and help to merge the code. Thanks.

Thanks @sethah and @dbtsai, I'll fix them soon.

Test build #64877 has started for PR 11119 at commit d4f59d9.

Test build #64878 has started for PR 11119 at commit 47f182b.

Test build #64878 has finished for PR 11119 at commit 47f182b.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64878/
Test FAILed.

Test build #64877 has finished for PR 11119 at commit d4f59d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64877/
Test PASSed.

@dbtsai It's ready for your review.

Test build #65237 has started for PR 11119 at commit 78ed9a1.

Test build #65237 has finished for PR 11119 at commit 78ed9a1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65237/
Test PASSed.

Merged build finished. Test PASSed.

ping @dbtsai @sethah

@yinxusen Would you mind responding to the comments on overriding the equals method above? I think it's a design point worth discussing.

Test build #65478 has started for PR 11119 at commit eb7fbbe.

Test build #65478 has finished for PR 11119 at commit eb7fbbe.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65478/
Test FAILed.

Merged build finished. Test FAILed.

Test build #65507 has started for PR 11119 at commit f6e024a.

Test build #65507 has finished for PR 11119 at commit f6e024a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65507/
Test PASSed.

Merged build finished. Test PASSed.

Ping @yinxusen on update. Would like to have it merged soon so we can work on LiR and LoR parts. Thanks.

Test build #65953 has started for PR 11119 at commit 92cf83d.

Test build #65953 has finished for PR 11119 at commit 92cf83d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65953/
Test PASSed.

Ping @dbtsai @sethah

@yinxusen Looking good. I left a few small comments, and we should take care of the checking of initial model params in the read/write test now. After that, I think it will be ready to merge. Thanks!

Test build #66380 has started for PR 11119 at commit 95bf12f.

Test build #66380 has finished for PR 11119 at commit 95bf12f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66380/
Test PASSed.

One minor comment, otherwise LGTM.
We should still consider overriding equals method as a follow up item. Also, we should discuss changing the behavior so that if initial model is set, then we simply use the implied value of k. e.g. we can train using this:
val kmeansModel = new KMeans().setInitialModel(model).fit(df)

I created SPARK-17789 for this.

ping @dbtsai

Test build #66446 has started for PR 11119 at commit 7fc6918.

Test build #66446 has finished for PR 11119 at commit 7fc6918.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66446/
Test PASSed.

Test build #66524 has started for PR 11119 at commit 5fbb132.

Test build #66524 has finished for PR 11119 at commit 5fbb132.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66524/
Test PASSed.

@dbtsai @sethah I updated the code. Now we check the equivalence of K when setting initialModel if K is set previously. We also check the equivalence when fitting a model.

Isn't the point of an initial model mainly the cluster centers? If you
override k what happens to the cluster centers? Discard them? Why not then
just start again rather than have an initial model
On Sat, 8 Oct 2016 at 09:04, DB Tsai<EMAIL_ADDRESS>wrote:

@dbtsai commented on this pull request.
In mllib/src/test/scala/org/apache/spark/ml/clustering/KMeansSuite.scala
https://github.com/apache/spark/pull/11119:

assert(wrongDimModelThrown.getMessage.contains("mismatched dimension"))

}

test("Infer K from an initial model if K is unset") {
val kmeans = new KMeans()
val testNewK = 10
val randomModel = KMeansSuite.generateRandomKMeansModel(dim, testNewK)
assert(kmeans.setInitialModel(randomModel).getK === testNewK)
}

test("Initialize using a model with wrong K if K is set") {
val kmeans = new KMeans().setK(k).setSeed(1).setMaxIter(1)

val wrongKModel = KMeansSuite.generateRandomKMeansModel(3, k + 1)
val wrongKModelThrown = intercept[IllegalArgumentException] {

 kmeans.setInitialModel(wrongKModel).fit(dataset)

I think the behavior could be that setInitialModel should overwrite the
configuration including # of iterations and k, etc. What do you think?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/11119, or mute the thread
https://github.com/notifications/unsubscribe-auth/AA_SBz3tn7GjUrfqnCmOmrcGbi38ZG2gks5qx0BogaJpZM4HV2Td
.

@MLnick I'm not sure I understand what you're saying. Where are we discarding cluster centers?
Maybe we should say that the initialModel always takes precedence over k. So we can just ignore k when initialModel is set, and log a warning at train time that we are ignoring it. There are going to be tradeoffs either way, and I think that is reasonable behavior. I vote to ignore k when initialModel is set. That also alleviate DB's concern about the following situation (which would fail given the current logic):
val km = new KMeans().setInitialModel(kEquals5Model)
val model1 = km.fit(df)
val model2 = km.setInitialModel(kEquals6Model).fit(df)

Again, I think we can all agree there are tradeoffs. Let's see if we can agree on something for now and go with it. If someone feels really strongly, then maybe we can discuss it in another JIRA.

+1 on what @sethah proposed. We can log with warn when k is modified by setting the initial model. Thanks.

I misread DB's meaning in my previous comment.
I agree that the parameter settings of initialModel, if set, should take precedence. If it conflicts with an existing k then log a warning.

I have a few high level questions on this:
Params
Why are we only setting k based on the initialModel? I had thought from previous discussion above (it was a while ago now) that it would set all parameters? Of course some will be ignored like the init settings, but I think the default expectation of using setInitialModel would be that all params are set.
For example, let's say I train a model with various maxIter and tol params. Then I want to use that model later for warm-starting. I want the same settings, just start from the existing centroids. I have to remember to do new KMeans().setInitialModel(model).setMaxIter(model.getMaxIter())....
If there is a good argument against this I'm happy to hear it, but then we must document this behavior clearly.
Saving initial model on Model
What is the reasoning behind saving the initialModel on the Model? It makes sense for the Estimator - I may want to save my estimator, and when loading it of course I'd need the initial model to be loaded, if it was set, so that I can correctly fit my estimator.
But once I've fit a model, why would I save two models?

We should also just check save / load is backward compatible with older versions. It should be, but subtle things can sneak in so let's be careful about that.

@MLnick Good questions. For the setting of params - I'm sure it varies in some cases, but for example, why would you want to use the same maxIter when training with an initial model vs without? I think you can make the case for some and less so for others. I think it's ok to move forward in this regard and have this discussion in another JIRA, but I'm open to other views.
About saving the initial model, it's a good point that you probably don't need to save that with the model. I think there is probably some case where you'd want it, but I'm not sure what that is. Maybe we can leave the functionality for now, and then think about how to make it optional in the future? This shouldn't be a binary compatibility problem AFAIK.

Test build #66991 has started for PR 11119 at commit 127ca06.

Test build #66994 has started for PR 11119 at commit 0e93fda.

Test build #66991 has finished for PR 11119 at commit 127ca06.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66991/
Test PASSed.

Test build #66994 has finished for PR 11119 at commit 0e93fda.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66994/
Test PASSed.

Merged build finished. Test PASSed.

@sethah New behavior of setK is adapted.

Test build #67050 has started for PR 11119 at commit 261fcfa.

Test build #67050 has finished for PR 11119 at commit 261fcfa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67050/
Test PASSed.

@yinxusen Thanks for the updates. Regarding save/load compatibility, I tested it locally and had no issues.
There are still some issues to address, but I think we can sort them out into other PRs. @MLnick @dbtsai for a final pass?

@sethah Thanks, I change these problems.

Test build #67051 has started for PR 11119 at commit b3ea01a.

Test build #67051 has finished for PR 11119 at commit b3ea01a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67051/
Test PASSed.

Merged build finished. Test PASSed.

Please remove WIP in the description.

One minor comment, otherwise LGTM. Thanks a lot @yinxusen and reviewers.

For the k vs other params - I'm ok with just setting k. I do agree that in most cases the other params may be different (in fact likely different from both the defaults and initial model for warm-start scenarios).
The only thing to consider is we will then need to make a decision for each estimator what the behavior is. This may be fine - even desirable. But it is also an option to default to overriding everything, making it more generic to implement for other estimators.
I don't have a very strong feeling either way though.

For the KMeansModel saving the initialModel - it still seems weird to me. Another factor to consider: the main use case in my view is continually re-training a model with the result from the previous training run. This makes up a model chain. Since each model saves the initial model, you'll end up with a recursive set of models all the way back to the beginning.
In a way this is useful I guess since it would allow tracking the lineage.   If someone actually wants that tracking, I'd imagine they would explicitly save each trained model.
Happy to hear dissenting thoughts on this point.

I was also thinking that most of people will use this for daily retraining by passing in the previous model which will cause the model larger and larger due to the model chain which is unnecessary as  @MLnick's point. Depending on what we decide to override, if we only override k and centroids for k means or coefficients for LoR, should we just keep them in the returning model without having the information as full initialModel? Or we just simply cut the lineage?

@MLnick I tend to agree with your opinion about saving the initialModel with the model.
We can remove saving the initial model entirely from the model. Though, right now users could avoid saving it by calling model.clear(model.initialModel) before saving. That at least provides flexibility to do both, but users may not think of it and get tripped up by the long lineage.
It's probably best to not save it at all in the long run. We can block this PR until this is resolved or discuss it in a follow up issue - I'm ok with either. Thanks for bringing this up.

I agree that saving the initialModel may not be practical - since it can be large. However, not saving that param at all also seems a bit contrary to me. When we produce a model from an estimator, we copy over the params that were used to create the model. These params give an indication to how the model was created. If we completely disregard the initialModel when we save the model, then it will appear as though the model was not created with an initialModel. In fact for kmeans, it would look like the model was created using the k-means|| initialization strategy since that is the default. This is misleading.
It would be nice to have a way to avoid saving the model with the initial model data, but still preserve the information about how the model was initialized. You can argue even, that the initialModel should not be a param at all because of the edge cases it seems to introduce.

Good points. Perhaps a solution - while slightly "verbose", is to introduce another param initialModelWriteMode which governs what is saved - full, params or none. Full is obviously the entire model, params is the metadata only without the data, while none is not saved at all.

Test build #67100 has started for PR 11119 at commit 47de1fe.

I agree that too long lineage would hurt performance and also unnecessary. How about cutting a lineage that is longer than 2? Namely, we only keep direct parent model when saving. Keeping direct parent model is clear enough for users to know from where the current model derived, meanwhile not too verbose.

Test build #67100 has finished for PR 11119 at commit 47de1fe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67100/
Test PASSed.

Merged build finished. Test PASSed.

We should change the initMode doc to indicate that the param is ignored when initialModel is set.

@sethah agree with that yes.

@yinxusen for cutting lineage do you intend to check whether the initialModel itself has an initialModel, and if so clear it?
I think that can be a reasonable solution if we have a clean implementation.

Test build #67156 has started for PR 11119 at commit e529972.

Test build #67156 has finished for PR 11119 at commit e529972.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67156/
Test PASSed.

@MLnick @dbtsai @sethah Any thoughts on the new version?

Related thought: if the model holds a pointer to its initialModel, then it will be serialized and shipped along with the model at prediction time. This will be inefficient for large models and even if we cut the lineage, we needlessly double the size of the closure.
It seems best not to have the initialModel in the model at all, but then it is an edge case for params, and it's still nice to know how the model was initialized at fit time. Thoughts?

How about the following:

Since the new generated model is derived from an estimator, the model should have the same params as its parent estimator. That's why there is no need to consider about params of model. We can only care about params of the estimator, as we have already done in the code - the param k and param initMode.

We can treat initialModel as a "transient" variable so that it must be vanished facing serialization, since there is no need to hold initialModel in a new generated model.

We can introduce a dumb model as a placeholder for initialModel, as a consequence, users know the initialization method of the model.

@jkbradley Thanks for your thoughts. I agree it's a good idea to change the KMeans prediction function to not use the entire model in its closure, but that we need a more thorough solution when we generalize this to predictors.
Would you mind pointing me to an example of an algorithm which only copies some, but not all, of the estimator params?

Users can identify the initialization method of the model by looking at Model.parent.initialModel.

Sure, but will they? How will they know that kMeansModel.getInitialModel is invalid, and that they should instead call kMeansModel.parent.getInitialModel? Also, there is some coupling between initMode and initialModel. It's misleading to have:
val km = new KMeans().setInitialModel(...)
km.getInitMode
> "k-means||"

It's especially misleading to have a model that says initialModel is unset (or that it doesn't even have an initialModel param), when it really was, AND that the init mode is some value as well. Maybe we should automatically set initMode to something like "initialModel" in the setInitialModel method. That would give the following behavior:
val km = new KMeans().setInitialModel(...)
km.getInitMode
> "initialModel"
val model = km.fit(df)
model.getInitMode
> "initialModel"

That solves the problem of users having to know to access the initialization modes via its parent, and having conflicting initMode and initialModel. This makes sense since setting an initial model is really just another option for initMode.

Would you mind pointing me to an example of an algorithm which only copies some, but not all, of the estimator params?

ALS is a good example: [https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala#L98]

[users identifying initialization method]

I agree it's misleading to have mismatched Params initialModel and initMode, especially if Model.initialModel does not exist.  I'd say this is an ideal solution:

(in this PR) Have setInitialModel also set k, initMode, etc. (where we create a new initMode called "initialModel").

Calling setInitMode("initialModel") would probably need to throw an error.  This is a minor issue IMO.

(in a follow-up PR)  The above bullet point has one bigger issue: Setting initialModel via km.set(km.initialModel, initialModel) would bypass the setter method and therefore not set k, initMode, etc. appropriately.  This issue with tied Params has appeared elsewhere in MLlib as well.  We could implement a fix by having the Params.set method use Scala reflection to call the corresponding setter method.  We'd just have to take extra care to test this well.

There are some Params in Models without matching setter methods.  Those were added with the intention of having Estimator Params easily accessible from Models.  We'll just have to keep these in mind when writing unit tests.

Ok, unless anyone has strong objections, it seems our plan moving forward with this PR should be:

Change the setInitialModel method to also set initMode to "initialModel"
Change the initMode param to support additional value "initialModel"
setInitMode throws an error when called with setInitMode("initialModel") and instructs user to use setInitialModel instead
Separate KMeansParams to extend KMeansModelParams and have an additional param initialModel
Update the read/write logic accordingly
Update tests
Create a follow up JIRA to address the case of calling set and bypassing the specific setInitialModel method

Let me know if I have missed something. @yinxusen Does this seam reasonable?

setInitMode throws an error when called with setInitMode("initialModel") and instructs user to use setInitialModel instead

On second thought, for this one, it could be good to have it work as long as initialModel is already set.
Otherwise, that plan matches what I have in mind.  Thanks!

@yinxusen Status update?

@sethah Sorry, I got stuck in other things. I'll update this PR tonight.

Test build #68325 has started for PR 11119 at commit 8516a2c.

Test build #68325 has finished for PR 11119 at commit 8516a2c.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68325/
Test FAILed.

This is probably going to miss 2.1 since we are officially in QA now, just as an fyi.

Test build #68368 has started for PR 11119 at commit 6f169eb.

Test build #68368 has finished for PR 11119 at commit 6f169eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68368/
Test PASSed.

Merged build finished. Test PASSed.

@yinxusen I took a look at the updates. Will you be able to create the design doc that Joseph mentioned?

ping?

@yinxusen Do you think you'll have time to work on this?

ping! I could take this over if needed :)

Test build #75004 has started for PR 11119 at commit 6f169eb.

Test build #75004 has finished for PR 11119 at commit 6f169eb.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75004/
Test PASSed.

Do you guys mind if I propose to close this PR?


Scripts not moved to header, but stay in footer

I need to move some scripts to the header.
So in Advanced Options I click Select next to the script, then click move to header. The script appears in the list of files to move to header in the right. I click Save Changes.
The script I moved to header, still says "minified in footer" even though its in the list of scripts to move to the header, and it does not move it to the header on the site.
Also a minor issue, but if I click select and move to header again it duplicates this script in the move to header box.
I am using 1.3.3

You have the wrong software repo. This is a library with nothing to click :)

Oh sorry, not sure I how I got here. I remember following several links to "support" but must have been something else! Ok thanks for letting me know.


keepassxc: Segmentation fault

Describe the bug
program crash when clicking button
https://gist.github.com/davidak/399bf0129ecaa1d6da422bbb3be2f17b
To Reproduce
Steps to reproduce the behavior:

open a database in keepassxc
click settings, secret service integration
enable freedesktop integration
click edit database settings

Expected behavior
dialog open or something, no crash
Screenshots

Notify maintainers
@jonafato @turion
Metadata

system: "x86_64-linux"
host os: Linux 5.6.13, NixOS, 20.03.1950.48723f48ab9 (Markhor)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3.4
channels(root): "home-manager-20.03, nixos-20.03.1950.48723f48ab9, nixos-hardware, nixos-unstable-20.09pre226148.0f5ce2fac0c"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute: keepassxc
# a list of nixos modules affected by the problem
module:

This is an upstream issue.
Workaround: Between steps 3 and 4, click "OK" and re-enter settings. You should then be able to edit the database settings.


Replace some copies and refs with constant refs

To:

avoid unnecessary copying;
avoid unintentional change of value by reference;
give compiler more context for optimizations.

Superseded by https://github.com/godotengine/godot/pull/59626.


Hud on Windows 10

Is it possible to show the hud when using dxvk on windows 10. i've add DXVK_HUD to the system variables but that doesnt work. any idea?

You might need to reboot.

It does work if you set the environment variable correctly, but as @K0bin said you might have to reboot.
Also, Windows is not the target platform of DXVK, it can be useful for debugging purposes, but I'm not going to support setup issues on Windows.

You don't need to reboot. You can add it like this: https://superuser.com/a/989665
Or use setx DXVK_HUD 1 before launching the game from the command line.

i've tried all suggestions. is the creation of the log files a real indication that the game is using dxvk? but i suspect there is something wrong with my w10 install because simple commands like "ping" cant be found in non admin powershell. but thanks for the help anyway.


[BUG] Race condition with global state in process pool

Describe the bug
Prompt flow appears to modify the global working directory and workers in the line execution process pool can get spawned with different working directories.
How To Reproduce the bug
I created an example project to reproduce the bug here: https://github.com/bhanson-techempower/promptflow-concurrency-bug
Some runs will succeed and others will randomly fail with:

Flow path .../promptflow-concurrency-bug/sub_flow2/sub_flow3 does not exist.

Due to the worker for that particular node being spawned after the working directory has been changed.
With the example project the bug is reproduced for me almost every time.
In our production application we're seeing it about half the time when running a batch of 20 runs.
Expected behavior
The flow executes successfully every time because prompt flow does not share extra global state between processes.
Running Information:

Promptflow Package Version using pf -v:

{
  "promptflow": "1.12.0",
  "promptflow-core": "1.12.0",
  "promptflow-devkit": "1.12.0",
  "promptflow-tracing": "1.12.0"
}

Executable '.../promptflow-concurrency-bug/venv/bin/python'
Python (Darwin) 3.11.3 (main, May 25 2023, 12:42:30) [Clang 11.1.0 ]

Operating System: macOS Sonoma 14.5

Additional context

References of _change_working_dir
References of getcwd
This comment also stuck out as suspicious

We've worked around the issue by modifying the way we invoke the sub flows:
from pathlib import Path
flow_path = Path(__file__).parent / "sub_flow1"
flow = load_flow(flow_path)

Also not sure if I should open up another issue, but the example project also shows a bug with the trace viewer. When I open the link provided in the console only the errored cases show up. This is happening every time for me on that project, although I haven't noticed it in our actual applications.
There should be 10 rows here:

We have noticed other subtle bugs with the trace viewer. Sometimes the information displayed in the table format doesn't match the underlying case when you click on it. The same line run might be duplicated a few times and then we need to click into each row to find the actual case we're looking for.
My suspicion is both of these are related to other concurrency issues and not actually a problem with the trace viewer.

Hi @bhanson-techempower , thanks for reaching us and the detail reproduce function. We just investigated this problem, I'll explain more about this.
Conclusion at first

Move the 'load_flow' out of @tool function / 2. Use absolute path / 3. Set concurrency to 1, all of them can resolve the problem.

Root cause

When running the flow, promptflow will change working directory to the flow directory to ensure flow can run successfully with correct imports.
Nodes without dependencies are executed concurrently.
In your flow, the 3 nodes didn't depends on each other so they got executed concurrently in one process, which caused the problem when executing to the load_flow line, as flow path got wrongly resolved due to cwd changed and effect each other.

Detail about the workaround

Move  the 'load_flow' out of @tool function -- the flow will be loaded when importing file, the cwd not changed at this time so flow can be loaded successfully.

Use absolute path -- load flow can work well with abs path because cwd will not effect the path anymore.
Set concurrency to 1 -- the cwd got changed and revert back correctly step by step when concurrency is 1, so load flow can work successfully with correct cwd.

To resolve this problem
To fully resolve this problem, we have to make each node run independently, for example in separate process, I'm afraid it will be a long-term work, I've add the 'long-term' tag and we'll keep this item open for anyone who meet the same problem, this item will be updated if we made changes to related part.


Convert code base from boost::string_view to std::string_view

Issue Description

The codebase currently does not make a lot of use of string_view. Mostly it shows up in the asio facing parts of the code. I tried converting those uses of boost::string_view to std::string_view a few months ago. It almost worked.
There’s a problem in boost 1.70 where in Windows the flag that tells beast to use std::string_view (not boost::string_view) does not work correctly. I understand that particular problem is fixed in Boost 1.71. So once we get to a place where the code base requires boost 1.71 or greater, then I think we can take on this task.
Once we have std::string_view in the code base there are a number of optimizations that can be performed. But I believe it should all start by moving to std::string_view first.
Exported from RIPD-1789

@ckeshava could you think about this and let us know your recommendation?
Feel free to consult with @scottschurr as needed (he opened the original issue)

Hello,
As per my understanding, we are using Boost v1.77. So we should not run into the issue pointed out earlier. I will test for compatibility on Visual Studio 2019 after making these changes.
I believe std::string_view is almost always better than std::string const&. Do I need to think about other alternatives for boost::string_view? What do you think @scottschurr  ?

Hi @ckeshava.  Thanks for looking into this.
In terms of changes, my feeling is that, wherever we can, we should be replacing boost::string_view with std::string_view.  The places that had problems before were boost::asio and boost::beast.  Hopefully fixing those is now possible.  I also had problems with soci wanting to use boost::string_view instead of std::string_view.  It would be interesting to learn the state of the world on that front as well.
In terms of replacing std::string with std::string_view, that needs to be done very carefully with an eye toward lifetimes.  So if we want to address that (and there are definitely places where that replacement is a win), that kind of change would want to be in one or more separate commits from the one described in the previous paragraph (just changing string_view types).
In case it's helpful, here is an outstanding review of what should (and should not) be done with string_view: https://www.youtube.com/watch?v=H9gAaNRoon4
I hope that helps.  Let me know if you have further questions.

We will need to finish eliminating string_views other than std::string_view if we want to upgrade to Boost 1.83 or later. The BEAST_USE_STD_STRING_VIEW macro that we have been relying on was removed in 1.83. We already removed boost::string_view in #4509, but 15 occurrences of boost::beast::string_view remain.


Is there a reason for redirecting hook output to /dev/null?

What's the issue?
Running hook script is redirected to /dev/null.
Why is this important?
Hook script could provide some usefull info, e.g. print error messages, etc. It could be neede to show this info to users.
What are the steps to reproduce?

Add hook script which print out something, e.g.

#!/bin/sh
# hooks/pre-commit/01-example_debug_msg
set -e
echo "My custom debug mesage"
exit 1

Try to commit. Example of result

[Autohook] Looking for pre-commit scripts to run...found 1!
[Autohook] BEGIN 01-example_debug_msg
[Autohook] FINISH 01-example_debug_msg
[Autohook] A pre-commit script yielded negative exit code 1

Expected:
[Autohook] Looking for pre-commit scripts to run...found 1!
[Autohook] BEGIN 01-example_debug_msg
My custom debug mesage
[Autohook] FINISH 01-example_debug_msg
[Autohook] A pre-commit script yielded negative exit code 1

Any additional info?
No
Any screenshots or other media?
No

The reason it does this is because it seemed like a good idea to me at the time! I don't think so at all any more.
Does #9 look like it addresses this adequately?

@nkantar yes, it seems to be OK. I just tested it, and it works nicely. I also added some sytax sugar as o comment to that PR.
Please merge that PR.

Sorry for the delay, I haven't really been actively involved, but just merged this.


Failed to load Sensapi.dll

Always when i try opening roblox, be it the roblox app, or thru the roblox website it gives me an error message, saying that it failed to load sensapi.dll. Sometimes it works normally, sometimes it doesnt. I have no idea what dll is that, or where its located, but i downloaded it and put it on the installation folders, but it still gives me errors

That's a Roblox problem. Does it still happen when using Roblox's original bootstrapper?

That's a Roblox problem. Does it still happen when using Roblox's original bootstrapper? Also, don't download DLL files off the internet.

oh- sure, and no, it doesnt happens when using the original bootstrapper

I don't actually know the fix to this. I've known of one other person who's had this problem, and I'm unable to figure out any correlation between this and Bloxstrap. Especially because people who don't use Bloxstrap sometimes get this error too. One of my friends had some success by running sfc /scannow in the Command Prompt as an Administrator, to do a check on all the WIndows DLLs. It's probably just a placebo, but you can try it out and see if it changes anything.
If that doesn't work and you're sure that this only ever happens with Bloxstrap installed, then I'm sorry, but you just won't be able to use Bloxstrap. Not until Roblox cracks down on what causes the problem at least.

I don't actually know the fix to this. I've known of one other person who's had this problem, and I'm unable to figure out any correlation between this and Bloxstrap. Especially because people who don't use Bloxstrap sometimes get this error too. One of my friends had some success by running sfc /scannow in the Command Prompt as an Administrator, to do a check on all the WIndows DLLs. It's probably just a placebo, but you can try it out and see if it changes anything.
If that doesn't work and you're sure that this only ever happens with Bloxstrap installed, then I'm sorry, but you just won't be able to use Bloxstrap. Not until Roblox cracks down on what causes the problem at least.
*Just saw your edit, yeah that sounds about right. Are you sure that doesn't happen without Bloxstrap?

well shucks, this is still a big help anyways! I will do the scan too anyways, and also, no. Atleast not that i remember, i never saw it happening with the normal roblox bootstrapper. But man, i might be just unlucky. I will keep trying to find a fix for it, but thanks for that already!


Remove unused property

Motivation:
property DEFAULT_MAX_CACHED_BYTEBUFFERS_PER_CHUNK is no longer used
Modifications:
removed unused property
Result:
clean up

please rebase once.

I tried to rebase it but it fails since the base branch is up-to-date. could you check again?

Thanks!


feat(BorderRadius token)!: Adding a border radius scale

Overview

Adding a border radius token (borderRadii) into gamut. There will now be standardized border radii - where none is 0px, sm is 2px, md is 4px, lg is 8px, xl is 16px, and full is 50%.

This also addresses some components which had outdated borderRadii.

In doing so, this PR also adds the Video component's stories from brand to gamut.

There are also other test branches:

For Atoms: https://github.com/Codecademy/gamut/pull/2921
For Molecules + Organisms: https://github.com/Codecademy/gamut/pull/2922

PR Checklist

[ ] Related to designs:
[x] Related to a mixture of JIRA tickets: GM-592, GM-892, GM-828, GM-830
[x] I have run this code to verify it works
[ ] This PR includes unit tests for the code change
[ ] This PR includes testing instructions tests for the code change
[ ] The alpha package of this PR is passing end-to-end tests in all relevant Codecademy repositories

Testing Instructions

Go to the following components and check that the components have been updated:
Alert 3 -> 2 (small)
Button, CTA 2 -> 4 (medium)
Card 0 -> 4 (medium) *
Disclosure 0 -> 4 (medium)
Drawer 0 -> 2 (small)
Floating Card 2 -> 0 (none)
FormInputs:
Checkbox 0 -> 4 (medium)
Input 2 -> 4 (medium)
Select Dropdown 2 -> 4 (medium)**
Text Area 2 -> 4 (medium)**
Modal/dialog 0 -> 2 (small)
Tip 3 -> 2 (small)

Notes:

in changing card from 0 => 4, I also applied these changes to the shadow variants as well (that relied on DynamicCardWrapper )
** the selectdropdown and textarea (from what I can tell, was 0 in prod, prior to change), though selectdropdown did have a 2px border radius for formatGroupLabel (so I added change it from 2px => 4 in this case too, since it seemed like all input-related elements were being updated to 4)

If inclined, pulling the branch locally, try adding a non-standardized borderRadius to an element (e.g. '4px') — this should show a TS error
Check out the test branches! And notion doc
...
Profit

PR Links and Envs

Repository
PR Link
PR Env

Monolith
Monolith PR
Monolith Env

Portal
Portal Link
Portal Env

Another Repo
Another Link
Another Env

📬Published Alpha Packages:@codecademy/gamut@57.0.0-alpha.976eda.0@codecademy/gamut-icons@9.29.2-alpha.976eda.0@codecademy/gamut-illustrations@0.46.1-alpha.976eda.0@codecademy/gamut-kit@0.6.434-alpha.976eda.0@codecademy/gamut-patterns@0.9.14-alpha.976eda.0@codecademy/gamut-styles@16.3.1-alpha.976eda.0@codecademy/gamut-tests@5.1.1-alpha.976eda.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://66b28eb99bd1ab6b4ff66db5--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut@57.0.0-alpha.2e8e59.0@codecademy/gamut-icons@9.29.2-alpha.2e8e59.0@codecademy/gamut-illustrations@0.46.1-alpha.2e8e59.0@codecademy/gamut-kit@0.6.434-alpha.2e8e59.0@codecademy/gamut-patterns@0.9.14-alpha.2e8e59.0@codecademy/gamut-styles@16.3.1-alpha.2e8e59.0@codecademy/gamut-tests@5.1.1-alpha.2e8e59.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://66b292254ce5c56d79c97537--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut@57.0.0-alpha.353249.0@codecademy/gamut-icons@9.29.2-alpha.353249.0@codecademy/gamut-illustrations@0.46.1-alpha.353249.0@codecademy/gamut-kit@0.6.434-alpha.353249.0@codecademy/gamut-patterns@0.9.14-alpha.353249.0@codecademy/gamut-styles@16.3.1-alpha.353249.0@codecademy/gamut-tests@5.1.1-alpha.353249.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://66b29c5ad354ea06a33f34e2--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut@57.0.0-alpha.286032.0@codecademy/gamut-icons@9.29.2-alpha.286032.0@codecademy/gamut-illustrations@0.46.1-alpha.286032.0@codecademy/gamut-kit@0.6.434-alpha.286032.0@codecademy/gamut-patterns@0.9.14-alpha.286032.0@codecademy/gamut-styles@16.3.1-alpha.286032.0@codecademy/gamut-tests@5.1.1-alpha.286032.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://66b2b17ef99cc20d6dc76a7c--gamut-preview.netlify.appDeploy Logs

🚀 Styleguide deploy preview ready!https://66b389b399645d008c3ffad1--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut@57.0.0-alpha.d1fa10.0@codecademy/gamut-icons@9.29.2-alpha.d1fa10.0@codecademy/gamut-illustrations@0.46.1-alpha.d1fa10.0@codecademy/gamut-kit@0.6.434-alpha.d1fa10.0@codecademy/gamut-patterns@0.9.14-alpha.d1fa10.0@codecademy/gamut-styles@16.3.1-alpha.d1fa10.0@codecademy/gamut-tests@5.1.1-alpha.d1fa10.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://66b634dda3dcf120f7b48441--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut@57.0.0-alpha.47b6ec.0@codecademy/gamut-icons@9.30.1-alpha.47b6ec.0@codecademy/gamut-illustrations@0.46.1-alpha.47b6ec.0@codecademy/gamut-kit@0.6.435-alpha.47b6ec.0@codecademy/gamut-patterns@0.9.14-alpha.47b6ec.0@codecademy/gamut-styles@16.3.1-alpha.47b6ec.0@codecademy/gamut-tests@5.1.1-alpha.47b6ec.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://66b638aba5564d138d26762f--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut@57.0.0-alpha.004197.0@codecademy/gamut-icons@9.30.1-alpha.004197.0@codecademy/gamut-illustrations@0.46.1-alpha.004197.0@codecademy/gamut-kit@0.6.435-alpha.004197.0@codecademy/gamut-patterns@0.9.14-alpha.004197.0@codecademy/gamut-styles@16.3.1-alpha.004197.0@codecademy/gamut-tests@5.1.1-alpha.004197.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://66bfa3941f5823929fd6f5c3--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut@57.0.0-alpha.75e367.0@codecademy/gamut-icons@9.30.1-alpha.75e367.0@codecademy/gamut-illustrations@0.46.1-alpha.75e367.0@codecademy/gamut-kit@0.6.437-alpha.75e367.0@codecademy/gamut-patterns@0.9.14-alpha.75e367.0@codecademy/gamut-styles@16.3.1-alpha.75e367.0@codecademy/gamut-tests@5.1.1-alpha.75e367.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://66bfa78e50dbc68a161e1e45--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut@57.0.0-alpha.65eb46.0@codecademy/gamut-icons@9.30.1-alpha.65eb46.0@codecademy/gamut-illustrations@0.46.1-alpha.65eb46.0@codecademy/gamut-kit@0.6.437-alpha.65eb46.0@codecademy/gamut-patterns@0.9.14-alpha.65eb46.0@codecademy/gamut-styles@16.3.1-alpha.65eb46.0@codecademy/gamut-tests@5.1.1-alpha.65eb46.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://66bfac85f0169f956e26608d--gamut-preview.netlify.appDeploy Logs

🚀 Styleguide deploy preview ready!https://66c60452cde1b81c7d0e0b4d--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut@57.0.0-alpha.a9f74b.0@codecademy/gamut-icons@9.30.1-alpha.a9f74b.0@codecademy/gamut-illustrations@0.46.1-alpha.a9f74b.0@codecademy/gamut-kit@0.6.437-alpha.a9f74b.0@codecademy/gamut-patterns@0.9.14-alpha.a9f74b.0@codecademy/gamut-styles@16.3.1-alpha.a9f74b.0@codecademy/gamut-tests@5.1.1-alpha.a9f74b.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://66c650c116ff688f5b779787--gamut-preview.netlify.appDeploy Logs

📬Published Alpha Packages:@codecademy/gamut@57.0.0-alpha.8bfb44.0@codecademy/gamut-icons@9.30.1-alpha.8bfb44.0@codecademy/gamut-illustrations@0.47.1-alpha.8bfb44.0@codecademy/gamut-kit@0.6.439-alpha.8bfb44.0@codecademy/gamut-patterns@0.9.14-alpha.8bfb44.0@codecademy/gamut-styles@16.3.1-alpha.8bfb44.0@codecademy/gamut-tests@5.1.1-alpha.8bfb44.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://66cde19921541118a9de0bc4--gamut-preview.netlify.appDeploy Logs


Close unfinished file and set max file size

If you get binary data by a stream and you wouldn't know the total stream size but you just able to understand this is the end of file when stream is done. You need to finalize the file.
This kind of IoT issues can be solved with this commit
first set file-length as "max" to maximize the file length
then upload stream bytes
finally send "Finish-Incomplate-File" header as "?1" by a head request

Thank you for the PR, but the tus protocol and tusd already support a very similar approach for handling unknown file sizes. I described this in my comment in #1017


techinals`

Dear Sir,
As you have mentioned that we have 300+ technical tools but i was unable to find them. kindly guide me.
kindly reply on this email id -<EMAIL_ADDRESS>

#21


FormBodyWrapperFilter removes Content-Type of part of multipart request

I have some multipart request with JSON part:
16-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "POST /someResource HTTP/1.1[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "Content-Length: 351[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "Content-Type: multipart/form-data; boundary=1-SWNrCOxUj9LMvkn8uVwPrHXJjWOlV4Fj0QDmz[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "Host: localhost:8080[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "User-Agent: Apache-HttpClient/1.0.0 (java 1.5)[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "--1-SWNrCOxUj9LMvkn8uVwPrHXJjWOlV4Fj0QDmz[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "Content-Disposition: form-data; name="json_request_part"[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "Content-Transfer-Encoding: 8bit[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:87 - http-outgoing-0 >> "[{"item_id":"574c478af190a54f83c0f209"]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "[\r][\n]"
2016-05-30 17:59:05 DEBUG wire:73 - http-outgoing-0 >> "--1-SWNrCOxUj9LMvkn8uVwPrHXJjWOlV4Fj0QDmz--[\r][\n]"

As you can see here is part with name 'json_request_part' with JSON content and appropriate content type header.
FormBodyWrapperFilter collects request parts in form partName->partContent without accessing headers (FormBodyWrapperFilter:167):
MultiValueMap<String, Object> builder = new LinkedMultiValueMap<String, Object>();
				for (Entry<String, String[]> entry : this.request.getParameterMap()
						.entrySet()) {
					for (String value : entry.getValue()) {
						builder.add(entry.getKey(), value);
					}
}

So, the headers are not present in the request to downstream service and just are missed here.

You need to send a filename in the content-disposition. I think we can treat this as an enhancement though, and be tolerant of this kind of input.

This fix broke mulitpart requests with additional form params (like metadata). So when I send request like
curl 'http://localhost:19801/api/attachments' -i -X POST -H 'Content-Type: multipart/form-data'  -F 'documents=@first-document-to-upload.txt;type=text/plain' -F 'metaData=8d0da6dc-100a-44d1-bdfe-9e498c5eca51'
I have two parts in request – one for file and second for form-data.
We're iterating over only MultiFileMap but then we're iterating over all parts which is incorrect because I'm getting my form-data added twice.
Here's some code:
MultipartRequest multi = (MultipartRequest) this.request;
// iterate over only file parts – correct
for (Entry<String, List<MultipartFile>> parts : multi
		.getMultiFileMap().entrySet()) {
        // iterate over all parts – incorrect.
	for (Part file : this.request.getParts()) {
		HttpHeaders headers = new HttpHeaders();
		headers.setContentDispositionFormData(file.getName(),
				file.getSubmittedFileName());
		if (file.getContentType() != null) {
			headers.setContentType(
					MediaType.valueOf(file.getContentType()));
		}
		HttpEntity<Resource> entity = new HttpEntity<Resource>(
				new InputStreamResource(file.getInputStream()),
				headers);
		builder.add(parts.getKey(), entity);
	}
}

And then I'm getting body like
--37F9X_t4KxBre7fXs9UrmuMbcRpHoXEOi
Content-Disposition: form-data; name="tempDir"
Content-Type: text/plain;charset=ISO-8859-1
Content-Length: 50

1467891109770_8d0da6dc-100a-44d1-bdfe-9e498c5eca51
--37F9X_t4KxBre7fXs9UrmuMbcRpHoXEOi
Content-Disposition: form-data; name="documents"; filename="first-document-to-upload.txt"
Content-Type: text/plain

This document is intended to be uploaded and attached to thread.
--37F9X_t4KxBre7fXs9UrmuMbcRpHoXEOi
Content-Disposition: form-data; name="tempDir"

1467891109770_8d0da6dc-100a-44d1-bdfe-9e498c5eca51
--37F9X_t4KxBre7fXs9UrmuMbcRpHoXEOi--

Please open a new issue if there's a problem. This was an enhancement and it is closed.


How to Install on Linux, Ubuntu?

It seems it just works on Windowsmachine, couldn't find the window "AI Command"by my Unity Version 2022.2.13f on ubuntu

I developed this project on macOS. So, it's not Windows-only. I haven't tested it on Ubuntu but sure that it works on it.
Is there the "AI Command" item in the "Window" menu?

The "AI Command" item in the "Window" menu is not available. Didn't know what I have to do.

I am speechless,
Meanwhile I installed an different AI-tool "https://github.com/srcnalt/OpenAI-Unity" after that I looked again on the "Window" item and for unexplained reasons I could finaly find it. It works well.


Set a default project in space-cli

The problem faced currently?
most of the space-CLI commands require --project flag for proper functioning. This is really inconvenient sometimes. It would be great if we can set a default project.
How can we solve it?
Add a command for setting default project to use if a project is not provided through --project flag.
Command : space-cli set default project "project-name-here"

If you want this feature to be implemented, give it a thumbs up reaction, so that we can determine which features are important to you.
👍

We can store this in the same accounts.yaml file which we use to store credentials

Do you want to store this in accounts or config.yaml? Seems more like a config parameter
I can take this up

Hey @thejasbabu. I think we should put it in the accounts.yaml file. @sharadregoti should we configure this on an account level or on global level?

We should store this parameter at the account level. As space-CLI supports multiple clusters in v19. Each cluster is associated with an account. Switching account through space-CLI should also change the default project for that cluster/account.

@thejasbabu should I assign this issue to you?

Done by @sharadregoti


[Experimental] Float8 support in AQT

Add float8 support to current Affine Quantized Tensor.

I think a good next step would be to add numerical tests, and ensure that this new object matches the numerical behavior of Float8Tensor.

can i ask whether we're sure we should include float8 tensors in AQT instead of another paradigm?

can i ask whether we're sure we should include float8 tensors in AQT instead of another paradigm?

AQT conceptually aligns a lot with fp8/fpx. Instead of writing a separate tensor subclass, it's more efficient to add float support to AQT. The concept of AQT is shared, the major difference is for dtype. This is an experimental PR, to test the feasibility. The design will be modified.

(sorry didn't mean to close this, missclick)

@jainapurva can you add this to https://github.com/pytorch/ao/blob/main/torchao/_models/llama/generate.py and https://github.com/pytorch/ao/blob/main/torchao/_models/llama/eval.py as well and get some e2e numbers for them

@jainapurva can you add this to https://github.com/pytorch/ao/blob/main/torchao/_models/llama/generate.py and https://github.com/pytorch/ao/blob/main/torchao/_models/llama/eval.py as well and get some e2e accuracy and perf numbers for them

Testing perf in this PR: https://github.com/pytorch/ao/pull/732


Add input sanitization checks

In general, we should audit and ensure user inputs are properly sanitized to prevent injection attacks, XSS, etc.

[x] XSS checks on URL creation, and URL reading, including all params #294
[x] input sanitization on all input types (partially covered by #294 )

Resolved by #294


[Bug] The priority of options in table should be higher than the default table options in catalog

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
Currently the default table options in catalog will override the options in table, the priorities for them are wrong
Solution
No response
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

hi, @FangYongs , I implemented the previous related logic, would you assign this to me?
Thanks.

After reviewing the code, the table options have higher priority than default options.
Let's close the issue. cc @FangYongs


Test for Bool/Int vars failed on Anaconda 5.0.1, x86_64 CentOS 6.9

Hi,
With cvxpy branch 1.0 installed on Anaconda 5.0.1 on x86_64 CentOS release 6.9, running 'nosetests cvxpy' produced the failures shown below.
These appear to be integer optimization tests. Do these test require some integer solvers? If so and the solvers are not installed, should cvxpy return an error when user attempts to solve a problem with integer/bool variables? Should the test then detect such an error and call it a PASS?
Best regards
.....
FAIL: test_bool_prob (cvxpy.tests.test_mip_vars.TestMIPVariable)
Traceback (most recent call last):
File "/home/zxz/projects/cvxpy/cvxpy/tests/test_mip_vars.py", line 68, in test_bool_prob
self.assertAlmostEqual(result, 0.04)
File "/home/zxz/projects/cvxpy/cvxpy/tests/base_test.py", line 41, in assertAlmostEqual
super(BaseTest, self).assertAlmostEqual(a, b, places=places)
AssertionError: -9.6512764254799896e-10 != 0.04 within 5 places
======================================================================
FAIL: test_int_prob (cvxpy.tests.test_mip_vars.TestMIPVariable)
Traceback (most recent call last):
File "/home/zxz/projects/cvxpy/cvxpy/tests/test_mip_vars.py", line 104, in test_int_prob
self.assertAlmostEqual(result, 0.04)
File "/home/zxz/projects/cvxpy/cvxpy/tests/base_test.py", line 41, in assertAlmostEqual
super(BaseTest, self).assertAlmostEqual(a, b, places=places)
AssertionError: 1.5064355097799085e-08 != 0.04 within 5 places

Ran 1818 tests in 211.205s
FAILED (failures=2)

You need to update ECOS. The old version had a bug in the branch and bound module.

Thank you for the quick reply.
My current ECOS conda package is 2.0.4, which is the latest:
https://anaconda.org/anaconda/ecos
This also appears to be the latest on github as well:
https://github.com/embotech/ecos-python
Which version and source would you recommend?
On Fri, Nov 17, 2017 at 5:45 PM, Steven Diamond<EMAIL_ADDRESS>wrote:

Closed #412 https://github.com/cvxgrp/cvxpy/issues/412.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/cvxgrp/cvxpy/issues/412#event-1348207421, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AgMrUUgKpqnDDaeOOUcV6TAoq8rplTWSks5s3jaigaJpZM4Qiy-7
.

Can you try installing ECOS from pip?
On Fri, Nov 17, 2017 at 6:55 PM smax1<EMAIL_ADDRESS>wrote:

Thank you for the quick reply.
My current ECOS conda package is 2.0.4, which is the latest:
https://anaconda.org/anaconda/ecos
This also appears to be the latest on github as well:
https://github.com/embotech/ecos-python
Which version and source would you recommend?
On Fri, Nov 17, 2017 at 5:45 PM, Steven Diamond<EMAIL_ADDRESS>wrote:

Closed #412 https://github.com/cvxgrp/cvxpy/issues/412.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/cvxgrp/cvxpy/issues/412#event-1348207421, or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AgMrUUgKpqnDDaeOOUcV6TAoq8rplTWSks5s3jaigaJpZM4Qiy-7
.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/cvxgrp/cvxpy/issues/412#issuecomment-345413392, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAfyIAdhBd-2ADVpaEH3OtOwJXvfZU-Jks5s3kctgaJpZM4Qiy-7
.

Version 2.0.4 appears to be the latest there too:
https://pypi.python.org/pypi/ecos
On Fri, Nov 17, 2017 at 7:21 PM, Steven Diamond<EMAIL_ADDRESS>wrote:

Can you try installing ECOS from pip?
On Fri, Nov 17, 2017 at 6:55 PM smax1<EMAIL_ADDRESS>wrote:

Thank you for the quick reply.
My current ECOS conda package is 2.0.4, which is the latest:
https://anaconda.org/anaconda/ecos
This also appears to be the latest on github as well:
https://github.com/embotech/ecos-python
Which version and source would you recommend?
On Fri, Nov 17, 2017 at 5:45 PM, Steven Diamond <
<EMAIL_ADDRESS>wrote:

Closed #412 https://github.com/cvxgrp/cvxpy/issues/412.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/cvxgrp/cvxpy/issues/412#event-1348207421, or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/
AgMrUUgKpqnDDaeOOUcV6TAoq8rplTWSks5s3jaigaJpZM4Qiy-7
.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/cvxgrp/cvxpy/issues/412#issuecomment-345413392, or
mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAfyIAdhBd-
2ADVpaEH3OtOwJXvfZU-Jks5s3kctgaJpZM4Qiy-7>

.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/cvxgrp/cvxpy/issues/412#issuecomment-345414697, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AgMrUQpV3gEVDptbZzvROinmSO-94xFbks5s3k03gaJpZM4Qiy-7
.

Try again now. I forgot to upload it apparently.

Thank you very much. After running '$ pip install --user --ignore-installed
ecos' all '$ nosetests cvxpy' pass.
On Sat, Nov 18, 2017 at 7:24 PM, Steven Diamond<EMAIL_ADDRESS>wrote:

Try again now. I forgot to upload it apparently.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/cvxgrp/cvxpy/issues/412#issuecomment-345488975, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AgMrUa13UNUu_NKEosIVr5gM2CO8WfDuks5s359-gaJpZM4Qiy-7
.

Sorry for the followup, but if this matters... Although the module version
is 2.0.5, it is installed into directory:
~/.local/lib/python3.6/site-packages/ecos-2.0.4
-py3.6-linux-x86_64.egg/ecos.py
, which implies version 2.0.4. Perhaps, setup.py also needs to be updated.
On Sun, Nov 19, 2017 at 12:40 AM, Stas Maximov<EMAIL_ADDRESS>wrote:

Thank you very much. After running '$ pip install --user
--ignore-installed ecos' all '$ nosetests cvxpy' pass.
On Sat, Nov 18, 2017 at 7:24 PM, Steven Diamond<EMAIL_ADDRESS>wrote:

Try again now. I forgot to upload it apparently.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/cvxgrp/cvxpy/issues/412#issuecomment-345488975, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AgMrUa13UNUu_NKEosIVr5gM2CO8WfDuks5s359-gaJpZM4Qiy-7
.


How do I resume training after an unexpected interruption in training?

Hi friend, Im a newer. For some reason, I can only train with a laptop, but halfway through the training, the computer restarts because the temperature is too high, how should I continue training? thank you for your help!

Hi there,
Did the machine finish the first epoch? If so, you should be able to find the saved checkpoint in the experiment path. In addition, when you train with a large dataset (with more than 200k samples), the script also saves the optimizer states.
https://github.com/YuanGongND/ast/blob/87a80043154eb4bb34ebceb4dc3e2d91a99235f4/src/traintest.py#L210-L216
The training progress is also saved at https://github.com/YuanGongND/ast/blob/87a80043154eb4bb34ebceb4dc3e2d91a99235f4/src/traintest.py#L39-L43.
You should be able to use above and torch.load and then torch.dataparallel to load the model and continue training, but we do not have an interface for continue training in this repo.
For training with lower computational overhead, you could consider (1) fine-tune our audioset pretrained model on your dataset, please check the ESC-50 recipe, and/or (2) using a smaller/no overlap in patch split, i.e., setting fstride=16 and tstride=16.
-Yuan

wow, thank you very much !! love you


Failed to load resource: net::ERR_INSECURE_RESPONSE

VSCode Version: 1.9.1
OS Version: win10

In markdown preview, when there is a link which has a Href attribute  using "Https", an error like the title come up:

But if I change it to Http, the error will disappear.
There is a small sample, looking forward your response.

Your example shows a css file failing to load. It is unclear how this is related to clicking on to a link the markdown preview. I am also unable to repo this using the sample you provided.
Please try upgrading to VSCode 1.10+ and running code with `--disable-extensions

My mistake, there is the situation

my point is that CSS source file use 'Https' can't load, but it works with 'Http', So actually, You got this problem.

As you can see, I used a different activationEvents and commands,

So, if you mean to avoid the influence of markdown extension of VSCode, I think, I need not use --disable-extensions .(BTW, it doesn't work).

I have tried VSCode 1.1.0, But it doesn't work either.

Looking forward your response.

Thanks for the additional detail. After a bit more investigation, I believe you are hitting this electron issue: https://github.com/electron/electron/issues/8366
We should pick up a fix when we upgrade to electron 1.5.x


Service Marketplace

Is your feature request related to a problem? Please describe.
Due to open service broker being to complicated for what we need and svc-cat being unmaintained, we need to look for a replacement.
I started a GH discussion here: https://github.com/epinio/epinio/discussions/854 where I added some high level requirements.
Describe the solution you'd like
I think Crossplane (https://github.com/crossplane/crossplane) is the right option to move forward. The main issue that I see is that they are missing local databases that give a clean way to bind to the application. This can be solved by working with some partners to build a "Local Database Provider".
Describe alternatives you've considered
See discussion

I had an idea:
If we end up using crossplane, it has a helm provider already built. The issue that I have with that provider is that it doesn't give any feedback to the platform on how to build the secret that gets mounted. This could likely be solved by using a composition of the helm provider with a provider we build that generates the right secrets to be consumed.
We could also fork a limited number of helm charts from bitnami and reduce the configurability of them to ease maintenance.

Useful explanation: https://blog.crossplane.io/crossplane-vs-cloud-infrastructure-addons/

I pushed a bunch of go code that lists the XRs defined in XRDs (as describe on the second step above): https://github.com/jimmykarily/epinio-issue-1012-poc

Postponing this a bit while we collect more information around Crossplane and how we want to use it. cc @agracey

@jimmykarily  Do we have more information on Crossplane now?

If we decide to go with it, it will be something like this:

Epinio discovers XRDs installed on the cluster
Epinio presents the XRDs as a "marketplace" to the user
Epinio provides commands to provision/delete services using the above XRDs

Epinio will not install or manage the Crossplane installation. We could come up with some "demo" XRD to create tests around it. Maybe something that can be used by users during development (some helm installed postgresql?).
If all the above is said and done, we won't have much of a marketplace to present because there aren't many XRDs available in public. Epinio users will only benefit if they or someone else creates some definitions for services.
With that in mind, I'm not sure if this is make it in the top priority list right now. I'll leave that to @agracey to plan it.

Ok, so the CLI would regain commands like
service market    # list xrds - "service classes"
service provision NAME XRD_NAME (XRD_PARAM...)
service unprovision NAME

This assumes that XRD's come with some kind of parameters you can set during provision, and that it is possible to query an XRD for such parameters. Type information maybe as well ?
Will be XRD based services be similar to the catalog based services we had before ? If so we may have to reintroduce the service interface type. As I suspect that the bind/unbind operations will need a different implementation, while we would like to keep the higher levels of the API server unaware of that.
The alternative would be to have the (un)provision commands not just create/delete the XRD service, but also manage a secret to hold the service location and credentials, like service create|delete do. I.e. manage a part which maps directly to what epinio currently uses for services.

Wrt priority, I am only asking at the moment because this is in our TODO right now.

Created a story for a spike here: https://github.com/epinio/epinio/issues/1281

Spike was done. Implementation happened here: #1332 . There are some improvements (e.g. #1359) but there is no reason to keep this epic open. Closing.


(Zen Back Forward) Zen Back Forward causes floating URL bar to go past the right side

With:

Without:

It work usually for me in all mode:

You might want to inspect your other mods!

I got no idea why it does that, because it happens on me with Compact Mode. I might need to reinstall...

After the zen ver 1.0.2-b.3 the back forward mod breaks Single toolbar layout in compact mode

enabling mod doesn't show the Single toolbar layout in compact mode

disabling the mod works fine


Karate Runner Java 8 updates

Description
Updated KarateRunner in karate-testng to use java 8 streams.
Updated Karate in karate-junit4 to include diamond operator in collections declarations.

Relevant Issues : N/A
Type of change :

[x] Code quality improvement

thanks but not merging for two reasons - right now we are on the cukexit branch (this is an exception for this release)
second I don't agree with needing the diamond operator. overall there are far more important things in this project and I prefer not to accept cosmetic PRs unless it is mixed with some concrete features, thanks. do consider contributing documentation - especially for the karate-core/README.md


[EntityBase] Remove DataContract and JsonProperty Attributes

Remove DataContract attribute

No need to reference System.Runtime.Serialization

Remove JsonProperty attribute

Make EntityIdentifier property public
No need to reference Json.Net in TE.Common

Update Clone method will first need to configure serializer to handle cycles #28
Acceptance tests will need configure serializers to handle cycles

Making the EntityIdentifier property public kind of orphans it.  I would like to include it in an interface which can be used on both server and client ends.  So I put it in an IMergeable interface, and am having IIdentifiable inherit from it.

Resolved by PR #40.


Replace print with logging in dataset scripts

Replaces print(...) in the dataset scripts with the library logger.

Maybe a script or even a test in test_dataset_common.py that verifies that a dataset script meets some set of quality standards (print calls and todos from the dataset script template are not present, etc.) could be added?

Yes definitely !


download dataset error

when I download the dataset with the following command: openxlab dataset get --dataset-repo GMAI/SA-Med2D-20M, I meet the following error

You need to go the the website and download a single file, for example the small JSON file, and fill in the form that pops up.
Afterwards the command should download the dataset successfully.


Restructure into a Neovim plugin via LazyVim

Thank you for your fantastic plugin! I came across your video on YouTube and decided to add the possibility of installing your plugin via LazyVim. While the original installation method remains available, I find it more convenient to manage plugins through a centralized manager like LazyVim. I've also included help documentation accessible within Vim via :h tsw.
To maintain backward compatibility, I've copied most of the content from ts-worksheet.lua to utils.lua. This ensures that everything continues to work seamlessly with the new installation method.
Please let me know if there's anything else you'd like me to adjust or improve!
Changes

Reorganized the codebase into a proper Neovim plugin structure
Split functionality into separate Lua modules:

commands.lua: Handles user commands and autocommands
config.lua: Manages plugin configuration
init.lua: Main entry point for the plugin
utils.lua: Contains utility functions and core logic

Copied core logic from ts-worksheet.lua to utils.lua
Added plugin doc file :h tsw

Rationale

Maintainability: The code is now split into modules, making it easier to maintain and extend in the future.

Organization: Each module handles a specific task, making the codebase more organized and understandable.

Configuration: The new structure simplifies configuration management with the config.lua file.

User Experience:

Added user commands and autocommands in commands.lua for better Neovim integration.
Help documentation is accessible with :h tsw.

Familiarity: The structure now follows standard Neovim plugin conventions, making it more user and contributor-friendly.

This is awesome. Thank you. I will check it out later today. Thanks for your work

@heilgar Thank you so much for the cool change. It was merged and I'm already using it like that myself. I'm a newby in neovim, so your help was greatly appreciated.

@chwoerz No problem at all. Thank you again for your plugin; it's been really helpful!
Just a heads-up: if you need to add or update functionality, you might need to support both files—one in the root and the other in utils—if you plan to keep the previous structure. If not, it might be better to remove the root file and have a single source of truth.
Here’s a reference on how it can be developed locally using a plugin-based approach:
https://github.com/heilgar/nvim-http-client/blob/main/doc/how-to-dev-plugin.md

I will most likely remove the other file 👍


Upgrade to rust 1.52.1

This PR depends on #9074.
This is part of upgrading from zexe to arkworks, which requires an upgrade to the latest rust (due to non-backwards-compatible typechecker changes in rust 1.49).
The changes in the marlin and zexe repos are minimal, and a build of this PR is able to connect and sync to mainnet, but this probably requires further testing to ensure that the new compiler doesn't cause issues with the existing OCaml-to-rust bindings.
Checklist:

[ ] Document code purpose, how to use it

Mention expected invariants, implicit constraints

[ ] Tests were added for the new behavior

Document test purpose, significance of failures
Test names should reflect their purpose

[ ] All tests pass (CI will check this if you didn't)
[ ] Serialized types are in stable-versioned modules
[ ] Does this close issues? List them:

@deepthiskumar  to help test this change

We've seen a segfault on test-labels (the net running this PR) :sob: I've pulled in #8774 to get more info if it happens again, and merged compatible to test that CI still passes against the latest changes.
This is the first memory corruption issue in 9 days of network uptime, across 8 nodes. Given the infrequency, and given that we've also seen exactly one segfault on devnet within the same period, I would prefer not to block merging this on discovering the source of that fault, especially since it may not be the cause at all.


ContentOffSet incorrect when use UIScrollView

It has strange display in a scrollView(Calls self.ss).
I set a UIScrollView on a UIViewController.view。
I set the scrollView the constraints that no space to the vc.view of each sides
[self.ss setParallaxHeaderView:self.header
                                     mode:VGParallaxHeaderModeTopFill
                                     height:300];

[self.ss setContentSize:CGSizeMake(self.view.bounds.size.width, 3000)];

self.header is a instance of UIView. It has a UIScrollView On it.When I built the project. Self.ss.contentOffSet was  {-600,0}
Everything is ok if  self.ss is UITableView

There is a observer there to watch for contentOffset changes to calculate new constraints. I will have a look next week. I've been super busy with regular work.


Can't connect to uhf reader from flutter app.

Hi, i need to read RFID/ UHF codes. I have tried this plugin with the sample code, but i looks like i cant open the serial connection as far as i see the logs.
trace:
Installing build/app/outputs/flutter-apk/app-debug.apk...
Debug service listening on ws://<IP_ADDRESS>:56251/fGm8wAMJTkw=/ws
Syncing files to device C72...
I/DeviceAPI_DeviceCon(22763): getDeviceInfoFromFile configPath=/storage/emulated/0/PDAConfig.txt
D/DeviceAPI_DeviceCon(22763): cw.model2:
D/DeviceAPI_DeviceCon(22763): cw.model:mtk.6765.c72
D/DeviceAPI_DeviceCon(22763): 最终cw.model:mtk.6765.c72
I/DeviceAPI_DeviceCon(22763): model=C72,uartPath=/dev/ttyMT3,uartPath_Fingerprint=/dev/ttyMT0,uartPath_RFID=
I/DeviceAPI_DeviceCon(22763): getModel() model=C72
D/DeviceAPI(22763): UHF device = C72
D/DeviceAPI(22763): UHF_Init----------->DevIsOpen = 1
D/DeviceAPI(22763): UHF uart = /dev/ttyMT3
D/DeviceAPI_SerialPort(22763): [SerialPort_Open] [2]: No such file or directory
D/DeviceAPI(22763): UHF uhf_uart_fd = -1
I/DeviceAPI_UHF(22763): init() Uart = /dev/ttyMT3
E/DeviceAPI_UHF(22763): init() err UHFOpenAndConnect result:-1
E/DeviceAPI_UHF(22763): setFrequencyMode() err :-1
I assume this is important
D/DeviceAPI_SerialPort(22763): [SerialPort_Open] [2]: No such file or directory maybe some path problems?
Thank you for answer. David

@dvamvakas-cyberheroes Try this plugin it supports Android 11


Update readme

The readme should probably have installation and running instructions sometime, so that others can do it if I'm not around, and for ease of people wanting to help contribute.

As a separate thing to address, the readme is currently a bit melodramatic.

That's part of it's charm.


Adding Force Pull Image

Adding Docker image force pull according to  https://jira.mesosphere.com/browse/METRONOME-148
This is based on the changes made by graceful task killing which needs to land first.
JIRA:  METRONOME-148

Needs tests and documentation.    Check initially to see reviews on the core work.

@kensipe LGTM!
Feel free to work on the tests. If you want a help with that, let me know :)

test to come...


02-load-data-with-power-query-in-power-bi-desktop.md

English:
When the Privacy Levels window opens, for each of the two data sources, in the corresponding dropdown list,select Organizational.
Current Translation:
当“私有级别”窗口打开时，对于两个数据源中的每一个，在相应的下拉列表中，选择“组织”。
Suggested Translation:
当“隐私级别”窗口打开时，对于两个数据源中的每一个，在相应的下拉列表中，选择“组织”。

Hello contributor,
Thank you for your contribution! We are processing your suggestions and we will update you shortly.
Kind regards,
Microsoft Worldwide Learning Team

Run the bugfix tool and waiting for result. ICMS QUERY

Fixed.


Autosync files to GDrive

When this task is complete, we should have a command-line program (using Python) that will automatically sync a folder (eg. the folder that's mounted from the GCE disk) into Gdrive. This will eventually be installed as a notebook extension but that's a task for later.
Don't worry about logging into GDrive yet. You can hard-code one of your personal auth tokens.

Done!


Flat flows and flat charges in blocksparse

Hi @mganahl, I believe there are some issues with .flows ._flows and .flat_flows in blocksparse. Consider the following example:
# define tensor
i0 = Index(U1Charge([-1,1]), flow=False)
i1 = Index(U1Charge([-1,0,1]), flow=False)
i2 = Index(U1Charge([-1,0,0,1]), flow=False)
i3 = Index(U1Charge([-2,2]), flow=True)
i4 = Index(U1Charge([-2,0,2]), flow=True)
i5 = Index(U1Charge([-2,0,0,2]), flow=True)
T0 = BlockSparseTensor.random([i0,i1,i2,i3,i4,i5])
# do transposition
T1 = T0.transpose([2,3,0,1,5,4]).reshape([8,2,3,12])

Then when I examine T1 the following is returned:
# T1._order == [[2, 3], [0], [1], [5, 4]]
# T1.flat_order == [2,3,0,1,5,4]

# T1.flows == [[False, True], [False], [False], [True, True]]
# T1._flows == [False, False, False,  True,  True,  True]
# T1.flat_flows == [False, False, False, True, True, True]

# T1.charges == [[i2,i3],i0,i1,[i5,i4]]
# T1._charges == [i0,i1,i2,i3,i4,i5]
# T1.flat_charges == [i0,i1,i2,i3,i4,i5]

So T1._flows and T1.flat_flows are exactly the same (similarly for the charges); they both give the flows before the lazy transposition and reshape. Is this intended behavior? It seems that it should instead be:
# T1.flat_flows == [False, True, False, False, True, True]
# T1.flat_charges == [i2,i3,i0,i1,i5,i4]

such that  T1.flat_flows is a flattening of  T1.flows rather than of T1._flows, which would be a useful attribute to have defined.

thanks @gevenbly! This is indeed intended behaviour, though it should be changed. I don't recall the exact problem, but both properties were used to ensure backwards compatibility with earlier code versions. At this point though they have become redundant, so we should remove them.
My suggestion is to remove them completely. Would you like instead to support the above described behaviour?

Supporting the behavior I describe above would be very useful, as the flat_flows and flat_charges are something that I would otherwise have to build myself each time I needed them (which is quite often).

sounds good, I will change it so that it has the desired behaviour.


New feature banner is stretched incorrectly

Description
There is a bug with the banner on the Wallet page, around 1700px breakpoint it stretches across the page, and then it is shown correctly again with the margins on the sides.
See screenshots below.
Expected result
The banner should preserve margins on all breakpoints.

Replaced by #404


Password Recover - No Success Message

Description
No success message is shown when you request a new password over /account/recover
Possible Solution
show a success message

Hi @cfreak,
actually there should always be a success message: "We sent you a confirmation email, if the provided email address is registered."

However the message is generic and does not differentiate if an account/email exists or not. We implemented this behavior due to security reasons. An attacker should not be able to guess if an email exists. An error message or no message at all should only be the case if there is something wrong with the system itself.
Do you get no message at all?
Best regards
Tobias


Add ModularFuelTanks v5.7.6 for KSP 1.1.3

Should enable KSP-CKAN/NetKAN#4488 to build

Checked and installs, merging


fix: :bug: chanage the CoreClasses to ModuleClasses

Description
chanage the CoreClasses to ModuleClasses.

good to go

@techoneel
Please don't approve failing checks PR.
Exception: Check configs are itself wrong. Please create Issue for PR checks.


[Bug]: WandB tracing not working when querying with a custom query engine

Bug Description
Running a custom query engine with wandb callbacks doesn't trace and log chains into wandb.
Version
0.10.20
Steps to Reproduce
I'm using a custom query engine:
`class CustomQueryEngine(CustomQueryEngine):
"""RAG Query Engine to integrate wandb"""
"""
https://github.com/run-llama/llama_index/blob/8eb59bdb84637c8043624f12736fc5f3995835ac/llama-index-core/llama_index/core/query_engine/custom.py#L39C1-L40C1
Tracing logic can be found here ^
"""
retriever: BaseRetriever
response_synthesizer: BaseSynthesizer
llm: OpenAI
qa_prompt: PromptTemplate
callback_manager: CallbackManager = Field(
default_factory=lambda: CallbackManager([Settings.callback_manager]), exclude=True
)
def custom_query(self, query_str: str):
    with self.callback_manager.as_trace("query"):
        nodes = self.retriever.retrieve(query_str)
        reranked_nodes = reranker.postprocess_nodes(query_str=query_str,nodes=nodes)
        context_str = "\n\n".join([n.node.get_content() for n in reranked_nodes])
        if context_str:
            response = self.llm.complete(
                qa_prompt.format(context_str=context_str, query_str=query_str)
            )
        else:
            return "Not enough context to generate a response"

        return str(response)`

and setting callbacks with:
 llama_debug = LlamaDebugHandler(print_trace_on_end=True)     run_args = dict(         project="rag_eval_test",     )     wandb_callback = WandbCallbackHandler(run_args=run_args)     Settings.callback_manager = CallbackManager([llama_debug, wandb_callback])
running the query engine with .query doesn't trace LLM and synthesis chains.
Looking at:
def query(self, str_or_query_bundle: QueryType) -> RESPONSE_TYPE: with self.callback_manager.as_trace("query"): # if query bundle, just run the query if isinstance(str_or_query_bundle, QueryBundle): query_str = str_or_query_bundle.query_str else: query_str = str_or_query_bundle raw_response = self.custom_query(query_str) return ( Response(raw_response) if isinstance(raw_response, str) else raw_response )
found here: https://github.com/run-llama/llama_index/blob/8eb59bdb84637c8043624f12736fc5f3995835ac/llama-index-core/llama_index/core/query_engine/custom.py#L39C1-L40C1
Tracing should automatically occur as query is inherited and custom_query is run through it
Am I doing anything wrong?
Relevant Logs/Tracbacks
**********
Trace: query
**********
Failed to log trace tree to W&B: list index out of range

I would expect this to be fixed as we move to the new instrumentation modules rather than the (now legacy) callback system


Click only

Is it possible on the bind function to click only? Because 'selectChoice' calls an event on the automatic dropdown like <a href> when you enter, tab or click of the mouse. I only want mouseclicks to call an event among the dropdowns

Did you try binding the click event ?

What do you mean bin the click event? Can you demo a code?
Here is my current code:
$('#searchbar').yourlabsAutocomplete({
url: '{% url "crew_view:navigation_autocomplete" %}',
choiceSelector: 'a',
}).input.bind('selectChoice', function(e, choice, autocomplete) {
window.location.href = choice.attr('href');
});

Actually, since you're using 'a' as choice selector, then you don't need to bind click at all.
Just remove that part:
.input.bind('selectChoice', function(e, choice, autocomplete) {
    window.location.href = choice.attr('href');
});

Also, please get your code formatting right on github next time, thanks.

What do you mean bin the click event? Can you demo a code?

That's not an autocomplete-light question but a generic jQuery question, here's an answer: http://api.jquery.com/on/

Actually, when you removed that part.. The click function is removed as well

When you click one of the choices, the choices container will just disappear on the moment of click

Then you've got another problem, because by default when clicking an <a> tag the browser should load the new url. I recon I'm not a fan of "blind debugging" so if you could reproduce this somewhere it'd be great else either find out why your browser doesn't load an url when clicking an <a> tag either use $.on() to bind the click event instead of the selectChoice event.

Could you produce a code on your end?
Because it is not really that hard trying to use other jquery event scripts to make it work. The problem is, it does not work with the django-autocomplete light

Just as everybody I love when people say "the problem is it does not work", and yes, I can produce a code, not sure when though.

Anyway, Just in case for other users who encountered the same problem.
I integrated the django autocomplete fields on search field of the django admin and my problem is I can not use the enter key anymore when you bind the choices with 'selectChoice'
I have solved the problem myself, I kinda cheated though(I set a timeout on the redirect function):
The code:
$('#searchbar').keydown(function(e){
if( e.which == '13' ){
$("#changelist-search").submit();
}
});
$('#searchbar').yourlabsAutocomplete({
url: '{% url "crew_view:navigation_autocomplete" %}',
choiceSelector: 'a',
}).input.bind('selectChoice', function(e, choice, autocomplete) {
setTimeout( function(){ window.location.href = choice.attr('href'); }, 500);
});

By the way I have another issue regarding keyboards, it is on another issue ticket can you please take a look as well?


Add support for wildcard characters when removing a view

php artisan scrap:view folder.* --force should remove all views in folder/. The wildcard character can only be used at the end of the argument, and will only work for removing files, not folders. So using php artisan scrap:view posts.*.index would not work in a situation like this:
posts/
|-authors/
| | index.blade.php
| | show.blade.php
--comments/
  | index.blade.php
  | show.blade.php

With the above situation, php artisan scrap:view posts.* would only remove the top-level files in the posts/ folder, the authors/ and comments/ folders (or files in them) will not be affected.

This package has been abandoned because of the introduction of the new make:view command included in Laravel since v10.23.0. For that reason, this issue will not be worked on.