feat: Defining Greedoid

The working branch for this project is greedoid-defs. This branch does not contain all features of the working branch, and is for real contribution to Mathlib.
The current branch have two files Predefs.lean and Basic.lean.
Before greedoid is properly defined, it defines two concepts on set systems, namely exchangeProperty and accessibleProperty in Predefs.lean.
Greedoid is a structure of set systems containing empty set, which satisfies both exchangeProperty and accessibleProperty, and is defined in Basic.lean.

Edit(4de12f4e7a4fb883e23b3447ed648d2bff6ac87d): Removed Basic.lean (will be added later), changed pull request title.
Edit(31f82971f047caa60bc9dc48116cc07dc1d307fc): Restored Basic.lean, but removed theorems related to exchangeProperty and accessibleProperty. Removed theorems are likely to be added in the next commit.

Currently it seems we can't find a technical reviewer for this one. If Mathlib is going to grow in this direction, we're going to need help from people who are already familiar with this, or adjacent, material. @apnelson1, would you be able to give this a review?

@qawbecrdtey, it's helpful if you can mark the conversations with Yury above as "resolved" if you've followed the suggestions there. It makes it easier for other reviews coming in to see what has been dealt with, and what is still active.

There are a few things to review:

Finset (Finset α) or Finset α -> Prop or Set α -> Prop: Currently it uses Finset (Finset α), as many lemmas on finite family of finite sets in the Combinatorics directory used it. Still I agree there are some conveniences which follow when the family of sets is implemented as Set α -> Prop just like Matroid, only if we discard computability. Finset α -> Prop is an option between those two.
Use of lemma instead of theorem: I could see a new keyword lemma in many other files. I wonder if this can be also applied to this file and afterwards.

I've never heard of a greedoid, so this is out of ignorance, but I don't see any reason to mention Finset in this file. Switching to Set a -> Prop seems a good idea.

I've never heard of a greedoid, so this is out of ignorance, but I don't see any reason to mention Finset in this file. Switching to Set a -> Prop seems a good idea.

I can see the reason for feasible sets being Finsets; you are in a setting where everything is finite by nature, and you get some computability consequences (being able to use cons for example) but I can't understand why feasible should itself be a Finset.
Presumably since with your setup there can only be finitely many feasible sets, you will never be considering a greedoid on an infinite type? If not, then why not just use Fintype a where it may be needed, and have feasible : Finset a -> Prop in the definition. 'feasible' is an adjective rather than a noun, after all.

Let me state again that I couldn't find any appearance of greedoids elaborating infinite sets. So for now it should be okay to state it as a family of finite sets. Even if we consider infinite greedoid of any type, the generalization of greedoids will probably not be what we were expecting, thus rewriting the library is a must in any case of generalization.
The reason I used Finset (Finset α) is simply because I referenced some files in Combinatorics/SetFamily (1, 2) while writing the first draft. This way we can easily know the feasible set of greedoid is also finite. However, the major problem with this approach which I noticed while implementing the draft is that it consumes too much memory. For matroids there is a method of converting a family of bases to a family of independent sets, but the similar thing doesn't work for greedoids. So we should store all the feasible sets, including the basis, if we're not dealing with ranks or closures. This should grow exponentially in most cases, so I also think that Finset (Finset α) is not a great idea in this sense. Although it is already implemented, and it was easy to take care of the structure using Finset of Finsets up to now, I think this should be replaced as it is not really 'computable'.
Set α -> Prop should work great as in matroids. As mentioned above we can restrict α as a Fintype, or we may simply restrict the feasibility to be True only if the set is finite. Of course there may be several other options, but all these can be simply managed using Finset α -> Prop (and probably assuming DecidableEq α?). Also,

'feasible' is an adjective rather than a noun, after all.

So maybe I should work on implementing again with family of feasible sets being Finset α -> Prop, with possible improvements. I'll bring the update at around mid-August. Thanks!

Let me state again that I couldn't find any appearance of greedoids elaborating infinite sets. So for now it should be okay to state it as a family of finite sets. Even if we consider infinite greedoid of any type, the generalization of greedoids will probably not be what we were expecting, thus rewriting the library is a must in any case of generalization.
The reason I used Finset (Finset α) is simply because I referenced some files in Combinatorics/SetFamily (1, 2) while writing the first draft. This way we can easily know the feasible set of greedoid is also finite. However, the major problem with this approach which I noticed while implementing the draft is that it consumes too much memory. For matroids there is a method of converting a family of bases to a family of independent sets, but the similar thing doesn't work for greedoids. So we should store all the feasible sets, including the basis, if we're not dealing with ranks or closures. This should grow exponentially in most cases, so I also think that Finset (Finset α) is not a great idea in this sense. Although it is already implemented, and it was easy to take care of the structure using Finset of Finsets up to now, I think this should be replaced as it is not really 'computable'.
Set α -> Prop should work great as in matroids. As mentioned above we can restrict α as a Fintype, or we may simply restrict the feasibility to be True only if the set is finite. Of course there may be several other options, but all these can be simply managed using Finset α -> Prop (and probably assuming DecidableEq α?). Also,

'feasible' is an adjective rather than a noun, after all.

So maybe I should work on implementing again with family of feasible sets being Finset α -> Prop, with possible improvements. I'll bring the update at around mid-August. Thanks!

Let me state again that I couldn't find any appearance of greedoids elaborating infinite sets. So for now it should be okay to state it as a family of finite sets. Even if we consider infinite greedoid of any type, the generalization of greedoids will probably not be what we were expecting, thus rewriting the library is a must in any case of generalization.
The reason I used Finset (Finset α) is simply because I referenced some files in Combinatorics/SetFamily (1, 2) while writing the first draft. This way we can easily know the feasible set of greedoid is also finite. However, the major problem with this approach which I noticed while implementing the draft is that it consumes too much memory. For matroids there is a method of converting a family of bases to a family of independent sets, but the similar thing doesn't work for greedoids. So we should store all the feasible sets, including the basis, if we're not dealing with ranks or closures. This should grow exponentially in most cases, so I also think that Finset (Finset α) is not a great idea in this sense. Although it is already implemented, and it was easy to take care of the structure using Finset of Finsets up to now, I think this should be replaced as it is not really 'computable'.
Set α -> Prop should work great as in matroids. As mentioned above we can restrict α as a Fintype, or we may simply restrict the feasibility to be True only if the set is finite. Of course there may be several other options, but all these can be simply managed using Finset α -> Prop (and probably assuming DecidableEq α?). Also,

'feasible' is an adjective rather than a noun, after all.

So maybe I should work on implementing again with family of feasible sets being Finset α -> Prop, with possible improvements. I'll bring the update at around mid-August. Thanks!

It feels to me like you are mixing ideas about computation and ideas about reasoning in a way that is causing confusion. What exactly are you hoping to achieve with greedoids in lean, computation-wise?

Let me state again that I couldn't find any appearance of greedoids elaborating infinite sets. So for now it should be okay to state it as a family of finite sets. Even if we consider infinite greedoid of any type, the generalization of greedoids will probably not be what we were expecting, thus rewriting the library is a must in any case of generalization.
The reason I used Finset (Finset α) is simply because I referenced some files in Combinatorics/SetFamily (1, 2) while writing the first draft. This way we can easily know the feasible set of greedoid is also finite. However, the major problem with this approach which I noticed while implementing the draft is that it consumes too much memory. For matroids there is a method of converting a family of bases to a family of independent sets, but the similar thing doesn't work for greedoids. So we should store all the feasible sets, including the basis, if we're not dealing with ranks or closures. This should grow exponentially in most cases, so I also think that Finset (Finset α) is not a great idea in this sense. Although it is already implemented, and it was easy to take care of the structure using Finset of Finsets up to now, I think this should be replaced as it is not really 'computable'.
Set α -> Prop should work great as in matroids. As mentioned above we can restrict α as a Fintype, or we may simply restrict the feasibility to be True only if the set is finite. Of course there may be several other options, but all these can be simply managed using Finset α -> Prop (and probably assuming DecidableEq α?). Also,

'feasible' is an adjective rather than a noun, after all.

So maybe I should work on implementing again with family of feasible sets being Finset α -> Prop, with possible improvements. I'll bring the update at around mid-August. Thanks!

Let me state again that I couldn't find any appearance of greedoids elaborating infinite sets. So for now it should be okay to state it as a family of finite sets. Even if we consider infinite greedoid of any type, the generalization of greedoids will probably not be what we were expecting, thus rewriting the library is a must in any case of generalization.
The reason I used Finset (Finset α) is simply because I referenced some files in Combinatorics/SetFamily (1, 2) while writing the first draft. This way we can easily know the feasible set of greedoid is also finite. However, the major problem with this approach which I noticed while implementing the draft is that it consumes too much memory. For matroids there is a method of converting a family of bases to a family of independent sets, but the similar thing doesn't work for greedoids. So we should store all the feasible sets, including the basis, if we're not dealing with ranks or closures. This should grow exponentially in most cases, so I also think that Finset (Finset α) is not a great idea in this sense. Although it is already implemented, and it was easy to take care of the structure using Finset of Finsets up to now, I think this should be replaced as it is not really 'computable'.
Set α -> Prop should work great as in matroids. As mentioned above we can restrict α as a Fintype, or we may simply restrict the feasibility to be True only if the set is finite. Of course there may be several other options, but all these can be simply managed using Finset α -> Prop (and probably assuming DecidableEq α?). Also,

'feasible' is an adjective rather than a noun, after all.

So maybe I should work on implementing again with family of feasible sets being Finset α -> Prop, with possible improvements. I'll bring the update at around mid-August. Thanks!

It feels to me like you are mixing ideas about computation and ideas about reasoning in a way that is causing confusion. What exactly are you hoping to achieve with greedoids in lean, computation-wise?

By the way, am I right in deducing that you are at Kaist? I will be giving a talk about matroid formalization on August 13 - see https://dimag.ibs.re.kr/event/2024-08-13/ . I'd be happy to discuss this in person.

By the way, am I right in deducing that you are at Kaist? I will be giving a talk about matroid formalization on August 13 - see https://dimag.ibs.re.kr/event/2024-08-13/ . I'd be happy to discuss this in person.

What a coincidence! I'll be looking forward to meeting you then!


[BUG] album pane broken

Describe the bug
The album pane is broken. No albums are shown. Images do show their albums in the new info panel. Also, creating a new album does not add it to the images submenu until you restart the application.
Version:
1.4
To Reproduce
Just open the application when you have albums.
Expected behavior
Albums should be seen in the list. Creating a new album should add it to the submenu inmediately.
Source Image
N/A
Source Image or Screenshots
N/A
Additional context
N/A

Is this starting from a new installation?

It works. Both the album pane and the update of the submenu. Thanks.


Uncaught overflow (#249 continued)

Same as the last tests in #249 with now different results

ulrich@g1:/tmp$ ulimit -v
100000
ulrich@g1:/tmp$ /opt/gupu/new-trealla/tpl -f
?- length(_,E),N is 2^2^E, catch(( length(L,N), length(L,M) ), Error,true).
   E = 0, N = 2, L = [_A,_B], M = 2
;  E = 1, N = 4, L = [_A,_B,_C,_D], M = 4
;  E = 2, N = 16, L = [_A,_B,_C,_D,_E,_F,_G,_H,_I,_J,_K,_L,_M,_N,_O,_P], M = 16
;  E = 3, N = 256, L = [_A,_B,_C,_D,_E,_F,_G,_H,_I,_J,_K,_L,_M,_N,_O,_P,_Q,_R,_S|...], M = 256
;  E = 4, N = 65536, L = [_A,_B,_C,_D,_E,_F,_G,_H,_I,_J,_K,_L,_M,_N,_O,_P,_Q,_R,_S|...], M = 65536
;  E = 5, N =<PHONE_NUMBER>, Error = error(resource_error(memory),length_rundown/2)
; Error: no memory
Aborted (core dumped)

Now:
ulrich@g1:/tmp$ time /opt/gupu/new-trealla/tpl -f
?- length(_,E),N is 2^2^E, catch(( length(L,N), length(L,M) ), Error,true).
   E = 0, N = 2, L = [_A,_B], M = 2
;  E = 1, N = 4, L = [_A,_B,_C,_D], M = 4
;  E = 2, N = 16, L = [_A,_B,_C,_D,_E,_F,_G,_H,_I,_J,_K,_L,_M,_N,_O,_P], M = 16
;  E = 3, N = 256, L = [_A,_B,_C,_D,_E,_F,_G,_H,_I,_J,_K,_L,_M,_N,_O,_P,_Q,_R,_S|...], M = 256
; ▮

▮ is the cursor position. It seems this is a buffering issue only, for after a C-c:
?- length(_,E),N is 2^2^E, catch(( length(L,N), length(L,M) ), Error,true).
   E = 0, N = 2, L = [_A,_B], M = 2
;  E = 1, N = 4, L = [_A,_B,_C,_D], M = 4
;  E = 2, N = 16, L = [_A,_B,_C,_D,_E,_F,_G,_H,_I,_J,_K,_L,_M,_N,_O,_P], M = 16
;  E = 3, N = 256, L = [_A,_B,_C,_D,_E,_F,_G,_H,_I,_J,_K,_L,_M,_N,_O,_P,_Q,_R,_S|...], M = 256
; ^C E = 4, N = 65536, L = [_A,_B,_C,_D,_E,_F,_G,_H,_I,_J,_K,_L,_M,_N,_O,_P,_Q,_R,_S|...], M = 65536
;
Action or (h)elp: c
 E = 5, N =<PHONE_NUMBER>, Error = error(resource_error(memory),length_rundown/2)
;
%terminated

Expected is to continue with resource errors.

?- length(_,E),N is 2^2^E, catch(( length(L,N), length(L,M) ), Error,true), writeq(E+N=Error),nl,fail.
0+2=_5
1+4=_5
2+16=_5
3+256=_5
4+65536=_5
5+4294967296=error(resource_error(memory),length_rundown/2)

%terminated unexpected

I am terminating a query after memory overflows. You can catch it, but then
it won't continue. Low level allocs are pretty much unmanageable at this
point.
On Fri, Dec 15, 2023 at 6:56 PM UWN @.***> wrote:

?- length(_,E),N is 2^2^E, catch(( length(L,N), length(L,M) ), Error,true), writeq(E+N=Error),nl,fail.
0+2=_5
1+4=_5
2+16=_5
3+256=_5
4+65536=_5
5+4294967296=error(resource_error(memory),length_rundown/2)
%terminated unexpected
—
Reply to this email directly, view it on GitHub
https://github.com/trealla-prolog/trealla/issues/429#issuecomment-1857511231,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFNKSEQC2MKAH7VJFYGYVQTYJQGDBAVCNFSM6AAAAABAUS2HUCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJXGUYTCMRTGE
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

If you do know that things have entered an inconsistent state, then issuing a system error (7.12.2 j) is more appropriate than the current behaviour.
And, regardless, the delayed printing of the answer for E = 4 and E = 5 is a problem.

Ok, i get the delayed printing now.
On Fri, Dec 15, 2023 at 7:17 PM UWN @.***> wrote:

If you do know that things have entered an inconsistent state, then
issuing a system error (7.12.2 j) is more appropriate than the current
behaviour.
And, regardless, the delayed printing of the answer for E = 4 and E = 5 is
a problem.
—
Reply to this email directly, view it on GitHub
https://github.com/trealla-prolog/trealla/issues/429#issuecomment-1857539985,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFNKSEQOQPGKH3H7WHN5MVTYJQISDAVCNFSM6AAAAABAUS2HUCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJXGUZTSOJYGU
.
You are receiving this because you commented.Message ID:
@.***>

And system_error is a good idea.
On Fri, Dec 15, 2023 at 7:17 PM UWN @.***> wrote:

If you do know that things have entered an inconsistent state, then
issuing a system error (7.12.2 j) is more appropriate than the current
behaviour.
And, regardless, the delayed printing of the answer for E = 4 and E = 5 is
a problem.
—
Reply to this email directly, view it on GitHub
https://github.com/trealla-prolog/trealla/issues/429#issuecomment-1857539985,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFNKSEQOQPGKH3H7WHN5MVTYJQISDAVCNFSM6AAAAABAUS2HUCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJXGUZTSOJYGU
.
You are receiving this because you commented.Message ID:
@.***>

UWN_open


add The new Windows Terminal

Adding The new Windows Terminal to "Terminal Emulation Applications" list

Checklist for submitting a pull request to Terminals Are Sexy:

[x] I have carefully read and comply with the Contributing Guidelines of this repo.
[x] I have searched the PRs of this repo and believe that this is not a duplicate.

Title: Windows Terminal
Link: https://github.com/microsoft/terminal
Category: Terminal Emulation Applications
Description: The new Windows Terminal, and the original Windows console host -- all in the same place!

Hi there @jeffrey-omega. Please change your commit so that only one line has been added. Thank you for your suggestion.


Minor fixes the README

Fixed spelling, grammar, and phrasing. I did my best to not change  the meaning of the text in the README and just fix the problems.  Because of that, some grammar issues remain,but changing them would  potentially change the meaning of the paragraph, so they were not  changed.

I can't merge the pull request because I don't have sufficient privileges. Could one of you do that? @wwwwww1020, @colinlaoaa, or @Xinchunran?
Also, I made an issue to go along with this pull request that we can link to it, which I also don't have permission to do.


Add Windows and Linux support

Please add windows and Linux desktop support using dart code

You might be better off using the Image library that is dart based and has a resize/compress option.


what's the difference between this TripletLoss  and pytorch predefined Torch.nn.TripletMarginLoss()

what's the difference between this TripletLoss  and pytorch predefined Torch.nn.TripletMarginLoss()

No Difference. The TripletMarginLoss was not provided when I implement this code.

thx


fix: design system package conflict with styled-component

Description
This PR updates the babel-plugin-styled-components since it was breaking design-system components in local development.
Fixes #15913
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Checked in local whether component are working fine or not

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

/ok-to-test sha=04fdf40

/ok-to-test sha=19c985a


Theme config changes

Merging this in and starting a prod deploy. Will address any comments in subsequent PRs.


Update documentation to include info on the Exporter

I needed to use the export functionality that @MaximePinot gracefully added a while ago and noticed that there was no documentation for it. I got to work with the information given in the PR and what I could gather from the code itself and wrote something that I think should be clear enough to get people going.
Maxime, would you be willing to review this?

Thanks! I applied all the suggested changes you made and I'll look into incorporating the other two suggestions into text tomorrow.

I think this commit should cover the two currently open conversations, unless I messed up my wording.

It looks good to me! :+1: Nice work!

Cool, thanks! Let's see what the big boss has to say. 😄 @curry684 ?

Big boss says awesoooooome   😉
Now I just have to fix the doc generator haha.


Exception in thread "main" java.lang.RuntimeException: Timeout of 120000 reached waiting for exclusive access to file

Unable to apply changes on device: UKYTWOVGTG69RK55. Error is: Failed to build plugin nativescript-ng-shadow :
Error: Command ./gradlew failed with exit code 1 Error output:
Exception in thread "main" java.lang.RuntimeException: Timeout of 120000 reached waiting for exclusive access to file: /Users/m.tahir/.gradle/wrapper/dists/gradle-4.4-bin/bgaq7vklkazwgxox0hdadxbvi/gradle-4.4-bin.zip
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:65)
at org.gradle.wrapper.Install.createDist(Install.java:48)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)
how to solve this issue?

HI @zeesulehria,
From the provided Error log it seems that you have some issue with the nativescript-ng-shadow plugin. Regarding that, I will suggest contacting the author of the plugin, who can help in resolving the issue.
This repository is used mainly for bug reporting and feature requests related to nativescript-angular.
Meanwhile,  I also tested the nativescript-ng-shadow's demo app with the latest NativeScript and the app seems to be built and executed correctly. You can review the attached project.
Archive.zip


Mleap XGBoost Extension calculating predictions different than the spark model

Hi
I have tried to use xgboost mleap extension. However, the results for mleap and spark are different.
Could you please guide me to fix this issue? I have also checked transformedTest.limit(1) always returns the same record which I have in MleapSample.json. If you need more info please send me your email.
val spark = SparkSession
   .builder()
   .master("local[*]")
   .appName("Trusted Payments Mleap")
   .config("spark.driver.memory","8g")
   .getOrCreate()
 val train = spark.read
   .format("csv")
   .option("header", "true") // Use first line of all files as header
   .option("inferSchema", "true") // Automatically infer data types
   .load("~/Documents/ml/RandomTrain.csv")

 val test = spark.read
   .format("csv")
   .option("header", "true") // Use first line of all files as header
   .option("inferSchema", "true") // Automatically infer data types
   .load("~/Documents/ml/RandomTest.csv")

 train.cache()
 test.cache()
 println(train.count())
 println(test.count())

 val categorical_columns = Array("Some list of columns")

 var indexers = mutable.ListBuffer[PipelineStage]()
 categorical_columns foreach(categoryColumn =>  {
   val indexer = new StringIndexer()
     .setInputCol(categoryColumn)
     .setOutputCol(categoryColumn + "_INDEX")
     .setHandleInvalid("keep")
   indexers += indexer
 })

 var encoderColumns = mutable.ListBuffer[String]()
 var encoders = mutable.ListBuffer[PipelineStage]()
 categorical_columns foreach(categoryColumn =>  {
   val encoder = new OneHotEncoder()
     .setInputCol(categoryColumn+"_INDEX")
     .setOutputCol(categoryColumn + "_VEC")
     .setDropLast(false)
   encoderColumns += categoryColumn + "_VEC"
   encoders += encoder
 })

 val features = train.columns.filter(!_.contains("label")).filter(!categorical_columns.contains(_))
 val assembler = new VectorAssembler()
   .setInputCols(features ++ encoderColumns)
   .setOutputCol("features")

 val pipeline = new Pipeline().setStages((indexers ++ encoders ++ Array(assembler)).toArray)

 val trainModel = pipeline.fit(train)
 val transformedTrain = trainModel.transform(train)
 val transformedTest = trainModel.transform(test)

 val paramMap = List(
   "eta" -> 0.1,
   "max_depth" -> 7,
   "objective" -> "binary:logistic",
   "num_round" ->10,
   "eval_metric" -> "auc",
   "nworkers" -> 8).toMap

 val xgboostEstimator = new XGBoostEstimator(paramMap)

 val evaluator = new BinaryClassificationEvaluator()
   .setLabelCol("label")
   .setRawPredictionCol("probabilities")
   .setMetricName("areaUnderPR")

 val cvModel = xgboostEstimator.fit(transformedTrain)

 val results = cvModel.transform(transformedTest)
 println("Final results "+evaluator.evaluate(results))
 cvModel.transform(transformedTest.limit(1)).select("probabilities", "prediction").show(1,false)
 val pipelineModel = SparkUtil.createPipelineModel(uid = "pipeline", Array(cvModel))
 val sbc = SparkBundleContext().withDataset(results)
 new File("/tmp/xgboostModel/xgboost.model.rf.zip").delete()
 for(bf <- managed(BundleFile("jar:file:/tmp/xgboostModel/xgboost.model.rf.zip"))) {
   pipelineModel.writeBundle.save(bf)(sbc).get
 }
 val mleapTransformerGBT = (for(bf <- managed(BundleFile("jar:file:/tmp/xgboostModel/xgboost.model.rf.zip"))) yield {
   bf.loadMleapBundle().get.root
 }).tried.get
 val s = scala.io.Source.fromURL("file:///~/Documents/ml/MleapSample.json").mkString
 val bytes = s.getBytes("UTF-8")

 for(frame <- FrameReader("ml.combust.mleap.json").fromBytes(bytes);
     frame <- mleapTransformerGBT.transform(frame);
     frameSelect <- frame.select("prediction");
     frameSelectProbability <- frame.select("probabilities")) {
   println("Result : " + frameSelect.dataset(0).getDouble(0))
   println("Probability : " + frameLrSelectProbability.dataset(0).getTensor(0).rawValues.mkString(","))
 }

Values from Mleap
Probability LR: 0.5955756452963958,0.40442435470360416
Values from Spark
+----------------------------------------+----------+
|probabilities                           |prediction|
+----------------------------------------+----------+
|[0.7874499559402466,0.21255002915859222]|0.0       |
+----------------------------------------+----------+

@fatihtekin Could you provide the test samples CSV file you are using?

Email is hollinrwilkins at gmail dot com

Hey,
So we got the model to train using a small set of features. We weren't able to find discrepancies between Spark and MLeap. Here is how we compared the outputs of the two:
val baseCols = transformedTest.schema.fields.map(_.name) :+ "mleap_probabilities"
val sparkCols = baseCols :+ "spark_probabilities"

val mleapPredictions = mleapTransformerGBT.sparkTransform(transformedTest).withColumnRenamed("probabilities", "mleap_probabilities").select(baseCols.head, baseCols.tail: _*)
val sideBySide = cvModel.transform(mleapPredictions).withColumnRenamed("probabilities", "spark_probabilities").select(sparkCols.head, sparkCols.tail: _*)

sideBySide.select("mleap_probabilities", "spark_probabilities").show()

+--------------------+--------------------+
| mleap_probabilities| spark_probabilities|
+--------------------+--------------------+
|[0.70909500179064...|[0.70909500122070...|
|[0.72755699620053...|[0.72755700349807...|
|[0.69017113816005...|[0.69017112255096...|
|[0.73918480641879...|[0.73918479681015...|
|[0.73918480641879...|[0.73918479681015...|
|[0.72755699620053...|[0.72755700349807...|
|[0.66373558475992...|[0.66373562812805...|
|[0.74395483672698...|[0.74395483732223...|
|[0.67537981034017...|[0.67537981271743...|
|[0.67325565942155...|[0.67325568199157...|
|[0.70351393514800...|[0.70351392030715...|
|[0.73246271740566...|[0.73246270418167...|
|[0.73246271740566...|[0.73246270418167...|
|[0.73648941675460...|[0.73648941516876...|
|[0.74395483672698...|[0.74395483732223...|
|[0.73648941675460...|[0.73648941516876...|
|[0.73648941675460...|[0.73648941516876...|
|[0.73648941675460...|[0.73648941516876...|
|[0.74395483672698...|[0.74395483732223...|
|[0.69546827110119...|[0.69546830654144...|
+--------------------+--------------------+

Hi Hollin
Thank you. You are right, I also tried your suggested code and got the same results.
But then, could you please help me what I am doing wrong with below code or with the JSON file? MleapSampleXGboost.json is already attached and it corresponds to the first record in the data file.
val s = scala.io.Source.fromURL("file:///Users/fatih.tekin/Documents/ml/MleapSampleXGboost.json").mkString
val bytes = s.getBytes("UTF-8")
for(frame <- FrameReader("ml.combust.mleap.json").fromBytes(bytes);
frameLr <- mleapTransformerGBT.transform(frame);
frameLrSelect <- frameLr.select("prediction");
frameLrSelectProbability <- frameLr.select("probabilities")) {
println("Price LR: " + frameLrSelect.dataset(0).getDouble(0))
println("Probability LR: " + frameLrSelectProbability.dataset(0).getTensor(0).rawValues.mkString(","))
}

@fatihtekin I looked at the JSON leap frame and it is a tensor with all of the features. Generating a leap frame with all of the original categorical/continuous features from the test dataset would help debug the issue you are seeing further.
Because the sample is just the feature tensor, I can't test with it, because I don't know exactly how you trained the model. We also can't verify that it is indeed the same exact data as the first record returned by Spark (which I'm not sure is guaranteed to always be the same unless we order the dataset first).
How do you create the leap frame json file? I can try generating one using the same method and running it through the pipeline I have trained.

I have checked the columns for the first record and they were matching with the first record. But it is also ok if you select any record you like as well.
To be able to get the feature tensor, I am calling below code and then pasting to the JSON file.
transformedTest.limit(1).collectAsList().get(0).getAsSparseVector.toDense

@fatihtekin Try using this code to generate a leap frame form the original test dataset, then run it through the entire pipeline instead of just the classifier:
import org.apache.spark.ml.mleap.SparkUtil
import org.apache.spark.ml.bundle.SparkBundleContext
import java.io.File
import resource._
import ml.combust.bundle.BundleFile
import ml.combust.mleap.spark.SparkSupport._
import ml.combust.mleap.runtime.MleapSupport._

val firstRow = test.toSparkLeapFrame.dataset.first
val schema = test.toSparkLeapFrame.schema
val dataset = ml.combust.mleap.runtime.LocalDataset(Seq(firstRow))
val leapFrame = ml.combust.mleap.runtime.DefaultLeapFrame(schema, dataset)
leapFrame.writer(ml.combust.mleap.runtime.serialization.BuiltinFormats.json).save(new java.io.File("/tmp/test_frame.json")).get

@fatihtekin Can you verify this is still and issue or can we mark this as resolved?

@fatihtekin Going to close this issue. Please reopen if you find any discrepancies!

@fatihtekin May I ask which libraries and versions you installed that enabled you to use this xgboost mleap extension code above?  This would be most helpful.   Much appreciated.


Datachannel close is broken

Error message: Could not set socket options for SCTP_RESET_STREAMS. errno=22 on amd64 and seems to somewhat work on ARMv7. Could be some SID related bug.

SCTP_STREAM_RESET_EVENT gets emitted more than once for every ResetSCTPStream call. This happens only for the python wrapper.

The issue is in incorrect/invalid SIDs inside the stream reset event.


⚠ Bofrost.de has degraded performance

In 44e6710, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1963 ms

Resolved: Bofrost.de performance has improved in b6a73b4.


1 and 2 electron overlap integrals for fragment

Hi, @a-p25 and I are trying to output the one electron and two electron overlap integrals of a fragment in the fragment's localized orbital basis, after we have performed the fragmentation of a system - does anyone have any initial advice?
The first thing we thought we'd try and do is look into the DMETFragment and Fragment classes, and see if they are generated in any calculations, but we couldn't find anything. Our current plan is to try and extract the logic from an OpenQEMIST (http://openqemist.1qbit.com/docs/#) method, which can calculate the overlap integrals from:

Molecule (pyscf.gto.Mole)
Mean field (pyscf.scf.RHF)
Active space, all orbitals in initial SCF calculation
Localization function (from pyscf.lo.orth.py)
Bath orbitals
Number of orbitals in fragment calculation
Core part of the one-particle RDM

We're now looking at getting these from the Fragment object and performing the manipulations on them to get integrals, but was wondering if anyone knows of another way we might be missing. Plus if you think this is useful to the vayesta repo - we'd be happy to create a PR for it.

Hi @hh2110 do you really mean overlap integrals? As in <i|j> ?
Or do you perhaps means 1- and 2-electron Hamiltonian elements <i|h|j> and <ij|r^-1|jk>

I mean <i|h|j> and <ij|r^-1|kl>

So we have an example here:
https://github.com/BoothGroup/Vayesta/blob/master/examples/ewf/molecules/20-dump-clusters.py
which is specifically for users who would like to use Vayesta to define fragment and bath spaces but then use the resulting spaces and integrals within their own code.
It dumps some cluster information to an HDF5 file - within a different group for each fragment. The 1- and 2-electron integrals, projected onto the cluster (=fragment+bath) space are in the fields hcore and eris.
There is also heff, which is hcore plus the Coulomb+exchange effective potential from frozen environment electrons.
If you only want these integrals in the fragment (without bath) space, you would have to do some extra work.
You would first have to form the matrix r = frac['c_cluster'].dot(ovlp).dot(frag['c_frag'])
which projectes from cluster basis to fragment basis and where ovlp is the AO overlap matrix, which you can get from PySCF as mf.get_ovlp() or from Vayesta as emb.get_ovlp().
Then you can perform hcore_frag = np.einsum('ij,ip,jq->pq', frag['hcore'], r, r) to project the 1e-core Hamiltonian onto the fragment space only. I hope this helps.

Yes, this is perfect thank you! I actually meant I wanted the integrals in the cluster (fragment+bath)


net: conn: Fix is_invalid_packet

In switch statement break statement was missing causing IPv4 part to
execute even if the packet is IPv6. Also logical negation is wrong in
this context.
Signed-off-by: Ruslan Mstoi<EMAIL_ADDRESS>

Codecov Report

Merging #7680 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #7680      +/-   ##
==========================================
+ Coverage   55.01%   55.01%   +<.01%
==========================================
  Files         483      483
  Lines       53950    53952       +2
  Branches    10495    10495
==========================================
+ Hits        29680    29682       +2
  Misses      19984    19984
  Partials     4286     4286

Impacted Files
Coverage Δ

subsys/net/ip/connection.c
78.35% <100%> (+0.1%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e7509c1...e2a687e. Read the comment docs.

@pfalcon: done


Cannot form industrial turbines

Hi, I have a huge problem with Mekanism V9. I tried all v9 versions just in case. I just can't form the turbine, I can't access to the interface, I reproduce the same turbine than in the video, I'm on a Superflat world, but it doesn't work. It's the only mod that I have, and I don't know why it doesn't work. Any solutions ?
EDIT : my configurator doesn't works too, and I can't change the mode of the Atomic Dissaembler

can you provide some pictures of the turbine?
i am not that far into mekanism to have a turbine at the moment or in the next few weeks, but maybe you missed some kind of blocks, which are needed for the turbine

Sure, there you go :

You don't have any glass above the pressure dispersers- a full cuboid still needs to be formed with glass and tank blocks.

I didn't understand, the last photo looks like your video... I'm sorry, I'm french I don't understand everything...

In the first image, it seems like you don't have glass covering the outside of your pressure dispensers and coils. It has to enclose the entire structure.

Sure, but this photo was just to show the inside of the structure. Look a the 4th photo, everythign is perfect, but it doesn't form..

In that case, which version of Mekanism are you using? I fixed an issue related to turbines not forming in other dimensions, perhaps this is related.

I use thé last version of Mekanism (9.0.2). But btw, I can't switch mode with the atomic disassembler and the configurator doesn't works too.

That's very strange, unless there's a keybind conflict there's something weird going on. Try playing on an instance with only Mekanism.

On Jun 2, 2016, at 6:38 PM, TheGeekdu74<EMAIL_ADDRESS>wrote:
I use thé last version of Mekanism (9.0.2). But btw, I can't switch mode with the atomic disassembler and the configurator doesn't works too.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.

But I found that the turbines are only with Mekanism Generators. I already tried to play only with Mekanism. But it doesn't works .... Maybe my problem is linked with the configurator or the atomic du disassembler ... i don't know ...

Specs : TechnicLauncher, superflat world, Minecraft 1.7.10, Forge <IP_ADDRESS>6, Java 1.8

Holy mother...  My bad, sorry...


[docs] warn that docker installation instructions are contrib/

Codecov Report

Merging #6925 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6925   +/-   ##
=======================================
  Coverage   63.96%   63.96%
=======================================
  Files         422      422
  Lines       20524    20524
  Branches     2230     2230
=======================================
  Hits        13128    13128
  Misses       7264     7264
  Partials      132      132

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5277fe...2231cd4. Read the comment docs.


[New Feature] Add Support for json logging

It would be nice if there is a option for json logging

@teimyBr is #64 according to what you had in mind?


Is it possible to edit a protected PDF with a "Validate" button?

Is it possible to edit a protected PDF with a "Validate" button?

This SDK can be used as server side dependency, and cannot be used directly in frontend JS based applications

Hmm... what does that have to do with my question???

@garrett-gottlieb : We are not able to fully understand your requirement. Would it be possible for you, elaborate more on this?

Hi @garrett-gottlieb - Following up on your question.
Could you please elaborate a little on your use case and requirement?
We have a remove password functionality through which you can remove the password protection, edit the PDF and reapply the protection. Regarding the "Edit functionality", are you using a custom application where you are editing the PDFs?

Closing this issue.
@garrett-gottlieb Please feel free to re-open the issue in case of any further queries.


LLM Prompt Adherence/Response Formatting.

Issues Persist when using ollama due to bad prompt adherence. As brought up by @dxcore35 in #15.
The "format" parameter that is used by ollama is not available using the OpenAI-Node library.
In a future overhaul of the api-call logic, it would be possible to make this work.
For now, I suggest testing different models and some improvements with regards to the prompt.
PR for a suggestion will follow shortly.

As a temporary "fix", this prompt-template works well for me with ollama and llama3.2:
"""
{{input}}
"""
Answer format is JSON {reliability:0~1, outputs:[tag1,tag2,...]}.
Even if you are unsure, qualify the reliability and select the best matches.
Respond only with valid JSON. Do not write an introduction or summary.
Output tags must be from these options:
{{reference}}
`;

🛑 gemma2:9b
🛑 qwen2.5


GSsOC '23 Contribution [Memory Game]

I'd like to express my keen desire to contribute to your project under GSsOC'23 by adding a Magic Match Game implemented using ReactJS.

Kindly assign me this issue for game

Hi @jyotideepjee1803 , could you please elaborate on the features of the  game with full details, demo / mock UI sample/Video? better share details documentation.
Because you have pasted the copy right claim pic here.

https://github.com/ssitvit/Games-and-Go/assets/98221699/24d0c7ab-2302-4916-ac83-9287609a611b
Demo


keg: don't delete main opt links of other formulae.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew tests with your changes locally?

If an old alias, according to the tab, of a formula being upgraded (e.g.
python) has the same name as the main opt link of some other formula
(e.g. python@2), don't remove the main opt link of that other formula.
Specifically,
brew install python@2
brew upgrade python
should not accidentally delete /usr/local/opt/python@2 even though
python@2 used to be an alias of python.
Fixes https://github.com/Homebrew/brew/pull/3893.

Looks good, thanks for the quick fix @ilovezfs!


New Bot Creation progress design feedback

Describe the bug
It would be good to have @emivers8 provide some feedback on how we are doing publishing progress.
At the very least, I would off two changes that could improve upon the current experience.

The progress spinner and the status text is very small, compared to what is happening.  See the screen shot below.
I would be good to provide some sort of task total count experience.

e.g.
1 of 6 Provisioning
2 of 6 Doing whatever is next
3 of 6 Doing the next thing in the process
Version
Version:  1.4.0-nightly.232669.4daad5c
Electron: 8.2.4
Chrome: 80.0.3987.165
NodeJS: 12.13.0
V8: 8.0.426.27-electron.0
Browser

[x] Electron distribution
[ ] Chrome
[ ] Safari
[ ] Firefox
[ ] Edge

OS

[x] macOS
[ ] Windows
[ ] Ubuntu

To Reproduce
Steps to reproduce the behavior:

Launch Composer
Click the '+' to create a new bot
Select any bot template, empty or whatever we're calling basic bot these days is good enough
Fill in the required data
Once the creation experience starts to template download, we display a spinner and update the text of that spinner ui

Expected behavior
Give that UX a little design love
Screenshots

Duplicate already completed in latest build


refactor(svelte/preprocess): use modern Svelte AST

@quentinderoubaix Thank you for your review! I have updated the code to take your comment into account.


I would like to add One Time Password Functionality to my bot.

Bot Info

SDK Platform: Node.js
SDK Version: v8.11.1
Deployment Environment: local development with Emulator

Issue Description
I am new to Bot Development, please be kind :)
I would like to add OTP Functionality to my bot through an email and I have no idea how to implement it, I would like to have some leads to add the functionality.
Expected Behavior
Bot Requests username, checks it in the database whether the username exists or not and confirms the email with the user. Once the user confirms, an email containing the OTP shall be sent to the user automatically. The user enters the OTP in the Bot, OTP is verified with the OTP in the DB.

I'm taking a look into possible solutions and will get back to you.

BotBuilder SDK v4.0

Hi @saranyuavh,
To do this, you will need to create an email bot. Following creating the email bot, you will need to build OAuth into the project. You can reference docs on DirectLine API 3.0 authentication or Authentication via ABS depending on your needs. There is also an OAuth example you can review.
As I don't know what service you plan on authenticating thru, Azure offers a variety of options for authenticating thru. This is done via the bot settings. The following image shows some of the options available.
Hope of help!


FIX: Fixes #14

Removed all refs to AssetAdmin.
Added support for service-based thumbnail generation.

This works for me, and will work for those who are using silverstripe/asset-admin. The only improvement I can think of is to add e.g. ThumbnailInterface so that the requirement for a generateThumbnail() method is baked-in at the language level, rather than relying on docs for users to read and understand.

i'm happy to merge as is, as long as it keeps working for existing users


hubble: Remove support for Service{Upsert,Delete}Notification

These events were not used by Hubble and on large clusters with lots of backends used up significant amounts of memory in the hubble ring buffer.
Service upsert and delete events are no longer emitted by Hubble as they were unused and used up significant amount of memory for services with many backends.

@cilium/sig-hubble does it make sense to stage this for deprecation in v1.17 and removal in v1.18?

does it make sense to stage this for deprecation in v1.17 and removal in v1.18?

yeah i think this makes sense. might be better to:

only mark it deprecated in .proto file without removing the functionality in v1.17
remove proto fields and the functionality in v1.18

but i'll make hubble folks make the call 🚀🙏

According to the notes from last week when we met to discuss this, we decided to mark it deprecated in the API to signal our intention to potentially remove, but instead of deleting the functionality outright we were going to turn it off by default.
Perhaps in a future release we can remove it altogether, but we probably want to be more cautious from the beginning.
Thank you @joamaki for putting up the PR. So we don't keep you on the hook, what do you think about one of the Hubble team members commandeering this PR and getting it merged in a slightly different state?

According to the notes from last week when we met to discuss this, we decided to mark it deprecated in the API to signal our intention to potentially remove, but instead of deleting the functionality outright we were going to turn it off by default.
Perhaps in a future release we can remove it altogether, but we probably want to be more cautious from the beginning.
Thank you @joamaki for putting up the PR. So we don't keep you on the hook, what do you think about one of the Hubble team members commandeering this PR and getting it merged in a slightly different state?

Well that of course sounds great to me!


[SmartContracts] Refactor Block Assemblers Part #1

This is the first PR refactoring the block assemblers to implement a better inheritance strategy, primarily to facilitate integrating the smart contract feature into the full node. This is also key to moving smart contracts to proof of stake after alpha release.
Currently the POS assembler inherits from the POW assembler. In the smart contract branch we also inherit from the POW assembler. We need to move base logic to a abstract base block assembler so that all the different block assemblers are logically more modular.
The next PR will rename the block assemblers to block builders and then we will move the assemblers to individual folder and create the base class.

yeah, i want to check this one

Yes. All unit tests and IntegrationTests.Miner - all are passing

How about syncing on mainnets Of bitcoin and Stratis?

Stratis mainnet- synced
BTC mainnet- synced first 40gb- all good.

Basically I agree with the points that Ivan raised.

@Aprogiena @noescape00 I have run all test and have synced main net and test net :)


Fehler beim AUslesen der Medieninformationen : 0

Version: 2.8.3
Build: 2016-08-15 09:58
OS: Windows 10 10.0
JDK: 1.8.0_101 x86 Oracle Corporation
What is the actual behaviour?
I get the Error Message in the TMM Messages during the Collection of the film data on my NAS.
The original TMM Data were created on another Computer.
I want to have TMM on my Notebook too so i install it there too.
During the Import the message appeared several times
What is the expected behaviour?
Steps to reproduce:
Additional
Have you attached the logfile from the day it happened?
tmm_logs.zip

thx.
Fixed with https://github.com/tinyMediaManager/tinyMediaManager/commit/56cb6e609f37f15e431da8a8fe0a1d64df146fdf
expect a hotfix release in a few hours...

thanks for help
yours sincerly

you also might want to try latest nightly/prerelease, to test if i really fixed your issue...

Of Course... from where do i get it?
Best regards

http://www.tinymediamanager.org/download/prerelease
just extract into a separate folder, if you want to keep your release version...

release is out


Update proto messages to support preemption

Currently, the executor (i.e., the in-cluster component of Armada) only reports free resources to the Armada server (the central component of Armada responsible for assigning jobs to executors). Available resources are reported on a per-node basis. To support preemption, the executor must report separately free resources and resources allocated to preemptible jobs.
More specifically, the executor must report total resources available and for each priority class the resources assigned to jobs of that priority class.
Implement by adding new fields to the proto message sent from the executor to the server with info on allocated resources. In particular, send a map from priority class to a report of the resources allocated to jobs with that priority class and a map of the total resources available.

I've made a prototype PR for this:

Hey team! Please add your planning poker estimate with ZenHub @dejanzele @jayofdoom @kannon92 @richscott


Fix committee fk

Description
Fixes a critical bug that can cause crashes under conditions in Conway.
Checklist

[x] Commit sequence broadly makes sense
[ ] Commits have useful messages
[ ] New tests are added if needed and existing tests are updated
[ ] Any changes are noted in the changelog
[ ] Code is formatted with fourmolu on version <IP_ADDRESS> (which can be run with scripts/fourmolize.sh)
[ ] Self-reviewed the diff

Migrations

[ ] The pr causes a breaking change of type a,b or c
[ ] If there is a breaking change, the pr includes a database migration and/or a fix process for old values, so that upgrade is possible
[ ] Resyncing and running the migrations provided will result in the same database semantically

If there is a breaking change, especially a big one, please add a justification here. Please elaborate
more what the migration achieves, what it cannot achieve or why a migration is not possible.

Fixes https://github.com/IntersectMBO/cardano-db-sync/issues/1831


Assertion Error Asyncio Socket

Getting these errors on Python 3.6
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/events.py", line 126, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 731, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 253, in data_received
    self._stream_reader.feed_data(data)
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 413, in feed_data
    assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof

My code
async def pubsub_loop(self):
	self.pubsub = self.redis.pubsub(ignore_subscribe_messages=True)
	await self.pubsub.subscribe('notsobot.rpc')
	while True:
		message = await self.pubsub.get_message()
		if message:
			load = json.loads(message['data'])
			self.bot.dispatch('rpc_receive', load)
		await asyncio.sleep(0.01)

async def publish(self, payload, snowflake):
	await self.redis.publish('notsobot.rpc', json.dumps({'shard_id': self.bot.shard_id, 'snowflake': snowflake, 'payload': payload}))

Not sure right now if it's my code or the library, the error is vague.
Also, everything is working fine, it's just these errors spam my console.

@NotSoSuper Thank you for your bug report!
According to your error, it seems like the aredis connection is used after it is closed. So i try to reproduce the error according to code given (i changed some part of it, for example, the bot attribute), but the code runs smoothly, if you don't mind, could you provide code more detailed?
Or can you tell me how you call publish() and pubsub_loop()?
My code is like below:
import json
import asyncio
import concurrent.futures
from aredis import StrictRedis

class ErrorCase(object):

    def __init__(self):
        self.redis = StrictRedis()
        self.shard_id = 1

    async def pubsub_loop(self):
        self.pubsub = self.redis.pubsub(ignore_subscribe_messages=True)
        await self.pubsub.subscribe('notsobot.rpc')
        for _ in range(10):
            message = await self.pubsub.get_message()
            if message:
                load = json.loads(message['data'])
                print(load)
                # self.bot.dispatch('rpc_receive', load)
            else:
                print('no message')
            await asyncio.sleep(0.01)

    async def publish(self, payload, snowflake):
        await self.redis.publish('notsobot.rpc', json.dumps({'shard_id': self.shard_id, 'snowflake': snowflake, 'payload': payload}))

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    error_case = ErrorCase()
    with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
        for _ in range(10):
            executor.submit(asyncio.run_coroutine_threadsafe, error_case.publish(1, 2), loop)
    loop.run_until_complete(error_case.pubsub_loop())

That's about all the code I use regarding aredis, the only other part would be in the rpc_receive function when I dispatch the event which calls at the end await self.publish({'command': 'received', 'response': r}, data['snowflake'])

@NotSoSuper
I am not sure, the exception is hard to trace because it is raised from asyncio code and the error can not be reproduced in my environment.
The only clue is the feed_eof
In aredis, the read stream is feed_eof only when connection is disconnectd, and after being feeded eof, the reader or writer stream will be set to None (which disable them to be used again). So it is wierd that the stream is used again after being feeded eof.
If you don't mind, can you help me to find out the line from which the error is raised? It's a good chance to make the code stronger.

I mean the line of aredis code in which the aredis connection is disconnected and reused again or something related.

Hey @NotSoSuper , do you have any idea about the bug?  Or may i close this issue?

Sorry for the late response, I've rechecked my code and used try/except blocks but none of them produced an error.

@NotSoSuper
Actually i did some experiments to prove my guess, but it seems like my guess is wrong.
In code below i try to disconnect the connection directly or use pubsub.reset to make the connection disconnected, but still can't the error be reproduced.
async def pubsub_loop(self):
        self.pubsub = self.redis.pubsub(ignore_subscribe_messages=True)
        await self.pubsub.subscribe('notsobot.rpc')
        for i in range(10):
            message = await self.pubsub.get_message()
            if message:
                load = json.loads(message['data'])
                print(load)
                # self.bot.dispatch('rpc_receive', load)
            else:
                print('no message')
            if i > 1:
                self.pubsub.connection.disconnect()
                # self.pubsub.reset()
                # await self.pubsub.subscribe('notsobot.rpc')
            await asyncio.sleep(0.01)

In PubSub.parse_response, if the connection can be read is checked when parsing response
async def parse_response(self, block=True):
        "Parse the response from a publish/subscribe command"
        connection = self.connection
        if connection is None:
            raise RuntimeError(
                'pubsub connection not set: '
                'did you forget to call subscribe() or psubscribe()?')
        if not block and not await connection.can_read():
            return None
        return await self._execute(connection, connection.read_response)

And the Connection.can_read will have a check on whether the connction is feeded eof by using at_eof function. If the connection is disconnectd as expected (its reader and writer stream will be set to None and then it will reconnect by self.connect() automatically)
async def can_read(self):
        "See if there's data that can be read."
        if not (self._reader and self._writer):
            await self.connect()
        return self._parser.can_read() or (not self._reader.at_eof())

@NotSoSuper
Hey~ Did you use StrictRedis or StrictRedisCluster as your redis client?
If the bug is raised when you were using StrictRedisCluster, there are two bug issues raised by melhin( #26 #28 ) which may lead to your question and is now fixed.
If you don't mind, could you install aredis from github (current version is not released yet) and try it again?
You can install by command
pip install git+https://github.com/NoneGG/aredis

@NoneGG I was using StrictRedis client, I'll update the package and test again!

@NotSoSuper
Thanks, but no big changes were made on StrictRedis and the bug may still be there.
Anyway, i will try my best to figure out the bug.

@NotSoSuper
To solve the problem, i discard usage of feed_eof(mainly used in connection.can_read and pubsub) and use another way to enable timeout when using `pubsub and to handle streams disconnection.
commit is here.
I hope that will help to solve the problem, it passes all test in my env and could you help to test in you code please?

It indeed has fixed the issue, thank you! I'll be switching back from aioredis 👍
>>> import aredis
>>> redis = aredis.StrictRedis()
>>> import asyncio
>>> loop = asyncio.get_event_loop()
>>> async def s():
...   pb = redis.pubsub(ignore_subscribe_messages=True)
...   await pb.subscribe('memes.1')
...   while True:
...     msg = await pb.get_message()
...     if msg:
...       print(msg)
...     await asyncio.sleep(1)
...
>>> loop.run_until_complete(s())
{'type': 'message', 'pattern': None, 'channel': 'memes.1', 'data': '{"lol": "meme"}'}

@NotSoSuper
Hope that will help you~
If you have other problems about aredis please tell me, i will fix it as soon as i can


PM policy: map device driver instances to power states disabling power to the devices

The problem
When a device driver activates a device, it should ensure that the power to the device won't be cut off. To do this, the device driver forbids the power management subsystem from entering the power states which would result in disabling power to the activated device. This can done with the following functions:
https://github.com/zephyrproject-rtos/zephyr/blob/7d7f7aaf03fd0728d298dce22c20cb3098c3d0aa/include/zephyr/pm/policy.h#L114
https://github.com/zephyrproject-rtos/zephyr/blob/7d7f7aaf03fd0728d298dce22c20cb3098c3d0aa/include/zephyr/pm/policy.h#L125
A device driver is expected to be a generic implementation for a given device. A single device may be integrated with different SoCs, by different vendors, or with different boards. Each of them can support other power states, and other power states may result in disabling power for this device. Moreover, in a single SoC, there can be multiple instances of a single device type (using the same driver) and each of these instances can be powered off in other power states.
The device driver implementation should be agnostic to the device integration environment, and because of this it shall not have hardcoded power state and sub-state values which are required in the void pm_policy_state_lock_get(enum pm_state state, uint8_t substate_id); function.
The solution
I would like to have a mapping between device instances and power states that disable power to a given device. I would like to express it in the device tree reusing already existing devices and power state nodes, like:
power-states {
                idle: system_on_idle {
                        compatible = "zephyr,power-state";
                        power-state-name = "suspend-to-idle";
                        min-residency-us = <100>;
                };

                suspend_to_ram: s2ram {
                        compatible = "zephyr,power-state";
                        power-state-name = "suspend-to-ram";
                        min-residency-us = <10000>;
                        exit-latency-us = <1000>;
                };
};

a_peripheral_bus {
        uart0: uart@8000 {
                compatible = "nordic,nrf-uarte";
                reg = <0x8000 0x1000>;
                interrupts = <8 NRF_DEFAULT_IRQ_PRIORITY>;
                power-states-disabling-power = <&suspend_to_ram>;
                status = "disabled";
        };

        uart1: uart@9000 {
                compatible = "nordic,nrf-uarte";
                reg = <0x9000 0x1000>;
                interrupts = <9 NRF_DEFAULT_IRQ_PRIORITY>;
                power-states-disabling-power = <>;  // This instance is powered on in all power states
                status = "disabled";
        };
};

With this mapping, the driver code could call a new power management function implemented as:
void pm_policy_device_power_lock_get(const struct *device)
{
    enum pm_state state;
    uint8_t substate_id;

    for (POWER_STATES_DISABLING_POWER_FOR_DEVICE(device, state, substate_id)) {
        pm_policy_state_lock_get(state, substate_id);
    }
}

void pm_policy_device_power_lock_put(const struct *device)
{
    enum pm_state state;
    uint8_t substate_id;

    for (POWER_STATES_DISABLING_POWER_FOR_DEVICE(device, state, substate_id)) {
        pm_policy_state_lock_put(state, substate_id);
    }
}

This way the driver gets decoupled from its device environment and the power tree in the product. The power tree is handled in the module dedicated to power management.
The alternatives
I considered using existing APIs, but it leads to hard to maintain macros and unnecessary coupling between the device drivers and the power tree in the products integrating the devices like:
#ifdef CONFIG_SOC_NRF...
    pm_policy_state_lock_get(1, 0);
    pm_policy_state_lock_get(1, 2);
    pm_policy_state_lock_get(2, 0);
#elif CONFIG_SOC_STM32...
    pm_policy_state_lock_get(0, 0);
#endif

I was also thinking about using "power domains" in the device tree, but I'm not familiar enough with this concept to include it in the proposal.

That makes a lot of sense, was thinking about it other day as well. Will work on PoC and move from there.

@hubertmis https://github.com/zephyrproject-rtos/zephyr/pull/70111 FYI !
Would mind take a look on that ? Nomenclature-wise is still a problem imho, so any feedback is welcome. Feature-wise it seems complete.


Does this framework's output have been compared with other features?

Does this framework's output have been compared with other features like wav2vec, hubert?

Hi,
No really, because HTS-AT itself is our proposed audio transformer, in this paper, we just use it for audio classification and SED tasks. But we use this HTS-AT architecture in other tasks, such as contrastive language-audio pretraining, CLAP. We compare this audio representation with other TF-domain SoTA. I think wav2vec can be compared, even though we did not conduct such experimenes before.


The Signature help appears twice

Describe the bug

Steps to Reproduce

Create empty c++ project named myexttest and add module.cpp

#include "Python.h"

static PyObject* AddTwoInteger(PyObject* Self, PyObject* Argvs)
{
	int Argv1(0), Argv2(0);
	if (!PyArg_ParseTuple(Argvs, "ii", &Argv1, &Argv2))
	{
		return 0;
	}

	return PyLong_FromLongLong(Argv1 + Argv2);

}

static PyMethodDef ModulesMethods[] =
{
	{ "Add", AddTwoInteger, METH_VARARGS, "Add Two Integers!" },
	{ NULL, NULL, 0, NULL }
};

static struct PyModuleDef ModuleDesc =
{
	PyModuleDef_HEAD_INIT,
	"AddTwoValue_Module",
	"This module is created by C++. And it Add two Integers!",
	-1,
	ModulesMethods
};

PyMODINIT_FUNC PyInit_myexttest(void)
{
	return PyModule_Create(&ModuleDesc);
}

Create Python Application project, add c++ project cppextcode as reference
Add code from  myexttest

import myexttest
print(myexttest.Add(10, 10))

Expected behavior
The Signature help of print just appears once.
Actual behavior
By the way, I restart the VS and can repro this.

This would likely be in the LanguageServer.Client dll (not in PTVS). It puts up this UI.

Created a bug on the language server here:
https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems/edit/1362812

@rchiodo This can repro in today's build. You can try to write a few more times print to repro this.

@ttSpace this issue is being tracked here:
https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems/edit/1362812
It should be a bug in VS proper as we don't provide the UI for that.

@ttSpace, I am not able to reproduce this. Could you leave some detailed repro steps on https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems/edit/1362812 or, even better, take a short screen capture video when you reproduce the issue?
Thanks a lot!


guard with "break"/"continue" no longer works in 1.2.3

This compiles and runs on 1.2.2 but not 1.2.3 master:
shared void run() {
    for (i in 0:1) {
        String? s = "";
        if (!exists s) {
            break; // or continue;
        }
        print(s.size);
    }
}

on master:
source/simple/run.ceylon:7: error: method or attribute does not exist: 'size' in type 'String?'
        print(s.size);
                ^
1 error

Yes, I removed that functionality.

oh? why? Wasn't it nice to have that kind of flow-typing?

I thought it was only disallowed in initializer sections?

Well the previous implementation was completely wrong, to begin with.

I have figured out how to make this work correctly.


Include PR title in merge commit messages

With this patch, the merge commit message changes from Merge #${pr_num} to Merge #${pr_num}: ${pr_title}, making the git history slightly more informative. This fixes #61.

Thank you! @OpsBotPrime merge

Pull request approved by @bertptrs, rebasing now.

Rebased as d991e85c409ccad72c9c19d5b995a3a758469010, waiting for CI …


docs: Fix lerna homepage link

What did you implement:
Fixed lerna broken link.

How did you implement it:
Updated readme with the correct lerna homepage link.
How can we verify it:
Click on the readme's lerna homepage link.

Todos:

[ ] ~Write tests~
[ ] ~Write documentation~
[ ] ~Fix linting errors~
[ ] ~Make sure code coverage hasn't dropped~
[x] Provide verification config / projects / resources
[x] Enable "Allow edits from maintainers" for this PR
[x] Update the messages below

Is this ready for review?: YES
Is it a breaking change?: NO

@rmax thanks!


"No common supertype of ..." while loading model

System information.

Have I written custom code (as opposed to using a stock example script provided in Keras): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Mint 21
TensorFlow installed from (source or binary): binary (tensorflow-cpu via pipenv)
TensorFlow version (use command below): v2.9.1-132-g18960c44ad3 2.9.2
Python version: 3.10
Bazel version (if compiling from source): -
GPU model and memory: -
Exact command to reproduce: Given the standalone code below saved as minimal-example.py, run python minimal-example.py

Describe the problem.
I can't load a model when passing a variable to a submodel. See the example below. This explicitly tells me to "[...] file a bug if [...] being hindered by this error." Code like that worked with previous versions of TensorFlow (2.7 I think).
Describe the current behavior.
Loading fails with No common supertype of TensorSpec(...) and VariableSpec(...).
Describe the expected behavior.
The model is loaded without errors.
Contributing.

Do you want to contribute a PR? (yes/no): no

Standalone code to reproduce the issue.
import tensorflow as tf

class SubModel(tf.keras.models.Model):
    def call(self, x):
        return ()  # My actual model returns something depending on x here, but even this fails.

class MainModel(tf.keras.models.Model):
    def __init__(self):
        super().__init__()
        self.sub_model = SubModel()
        self.x = tf.Variable(tf.zeros((1,)))

    def call(self, _):
        return self.sub_model(self.x)

model = MainModel()
model(())
model.save("model")
model = tf.keras.models.load_model("model")

Source code / logs.
Output of the above script:
Traceback (most recent call last):
  File ".../minimal-example.py", line 19, in <module>
    model = tf.keras.models.load_model("model")
  File ".../.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File ".../.venv/lib/python3.10/site-packages/keras/saving/saved_model/load.py", line 1151, in common_spec
    raise TypeError(f'No common supertype of {x} and {y}.')
TypeError: No common supertype of TensorSpec(shape=(None,), dtype=tf.float32, name=None) and VariableSpec(shape=(1,), dtype=<dtype: 'float32'>, trainable=True).

@ahasselbring I tried to replicate the issue on colab, could you find the gist here and confirm the same?
Thank you!

Indeed I get that error when I use TensorFlow 2.8, but with both 2.9 and 2.10 the error is as in the issue description:
No common supertype of TensorSpec(shape=(None,), dtype=tf.float32, name=None) and VariableSpec(shape=(1,), dtype=<dtype: 'float32'>, trainable=True).

coming from here (2.9) or here (2.10). Unfortunately I can't test with 2.7 anymore (I'm quite sure things like that worked with that version - maybe not this minimal example, but generally, passing variables to submodels, letting them do calculations and return them).

@ahasselbring With tensorflow 2.7, there is a warning that states
WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.
You can find the gist here.

That warning can easily removed by adding compile=False to the load_model call in 2.7 (or by actually doing some training with a useful model, but that doesn't make sense in a minimal example). In newer versions, the error occurs regardless of that option.

Summry: This model cannot be save and load. (Minor modification made to the original snippet, tested with colab)
class SubModel(tf.keras.models.Model):
    def call(self, x):
        return x

class MainModel(tf.keras.models.Model):
    def __init__(self):
        super().__init__()
        self.sub_model = SubModel()
        self.x = tf.Variable(tf.zeros((1,)))

    def call(self, x):
        self.sub_model(self.x)
        return x

Thanks for reporting the issue - at this point we're lacking bandwidth to look further into this and community contribution for a potential fix is welcomed.

I am working around this now by passing variable + tf.zeros_like(variable) instead of variable.

I'm also running into a related issue in a similar context, but I'm not exactly understanding what the error is.

I have a similar issue with a model that uses RaggedTensors. I see an error message that looks something like "No common supertype of TensorSpec and RaggedTensorSpec".
I seem to be able to fix this by rolling back the code for infer_inputs_from_restored_call_function from r2.10 (found here: https://github.com/keras-team/keras/blob/r2.10/keras/saving/saved_model/load.py#L1293) to r2.7 (found here: https://github.com/keras-team/keras/blob/r2.7/keras/saving/saved_model/load.py#L1172, which also requires adding back get_common_shape right above it), but leaving everything else in r2.10 the same. It looks to me like this is the only place where this function is used.
I didn't seem to notice any problems with this in the local testing that I did, does anyone have context for why this was changed or know of side effects of rolling it back?


Create Presto edge release pipeline

This pipeline runs weekly on every Tuesday 12 PM PT, automatically. It can also be triggered on-demand.
This pipeline check the latest stable release and the latest edge release, increments the edge build number and create a new branch. The corresponding release artifacts will be tagged based on the build number.

Jenkins pipeline
https://github.com/prestodb/presto-release-tools/blob/pipeline-release-edge/Jenkinsfile


ORACLE: Issue with Export resultset to SQL INSERT statements for a table field of CLOB datatype

Export resultset to SQL INSERT statements for a table field of CLOB datatype is done as is and it is ok for a data with the length < 2~4KB.
But when the data in CLOB field is bigger then generated SQL INSERT will fail when i try to execute it.
How to fix: such long CLOB value must be saved to the file using the following format:
INSERT INTO "T" (X,C) VALUES
(1,TO_CLOB('xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')
||
TO_CLOB('xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')
||
TO_CLOB('xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaaaaaa'));

Agreed, this makes sense.

Will be added in the next version.
It'd be a great feature to make something similar with BLOBs.


Memory leak

The following code works for a while but stops receiving messages very quickly. When run without the debugger, I actually get the following error messages:
15/01/09 21:36:50 WARN channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.OutOfMemoryError: Java heap space

To expedite the process I limit JVM memory with -Xmx32m. However, I first discovered this issue while running a production server with 6GB heap size. It was under very low load and received a small message every 20 seconds. At some point I noticed messages were not being delivered anymore and the logs threw the same OOM errors. A heap dump showed buffer in com.lambdaworks.redis.protocol.CommanHandler was a holding a contiguous 1GB array. It was trying to extend the buffer, but had no more huge contiguous blocks in the heap.
I am weirdly unable to reproduce the OOM in a debugger. However, when running in a debugger, I still do get the same issue with messages not being received after a while.
import java.util.Arrays;
import java.util.concurrent.ExecutionException;
import java.util.logging.Logger;

import com.lambdaworks.redis.RedisClient;
import com.lambdaworks.redis.RedisConnection;
import com.lambdaworks.redis.RedisURI;
import com.lambdaworks.redis.codec.Utf8StringCodec;
import com.lambdaworks.redis.pubsub.RedisPubSubConnectionImpl;
import com.lambdaworks.redis.pubsub.RedisPubSubListener;

public class LettuceMemoryLeak implements RedisPubSubListener<String, String>
{

    private static Logger LOGGER = Logger.getLogger( "LettuceMemoryLeak" );

    public static void main( String[] args ) throws InterruptedException, ExecutionException
    {
        RedisURI uri = RedisURI.Builder.sentinel( "localhost", 26379, "mymaster" ).build();
        RedisClient client = new RedisClient( uri );

        RedisPubSubConnectionImpl<String, String> subscriber = client.connectPubSub( new Utf8StringCodec() );
        subscriber.subscribe( "test" );
        subscriber.addListener( new LettuceMemoryLeak() );

        RedisConnection<String, String> publisher = client.connect( new Utf8StringCodec() );

        char[] chars = new char[8096];
        Arrays.fill( chars, 'a' );
        String str = new String( chars );

        while( true ) {
            publisher.publish( "test", str );
        }
    }

    @Override
    public void message( String channel, String message )
    {
        LOGGER.info( channel );
    }

    @Override
    public void message( String pattern, String channel, String message )
    {
    }

    @Override
    public void subscribed( String channel, long count )
    {
        LOGGER.info( "subscribed to " + channel + " " + count );
    }

    @Override
    public void psubscribed( String pattern, long count )
    {
    }

    @Override
    public void unsubscribed( String channel, long count )
    {
    }

    @Override
    public void punsubscribed( String pattern, long count )
    {
    }

}

I'll update this ticket once I have more information.

The problem seems to be with PubSubCommandHandler.decode() which never discards of read bytes in the buffer. It override the method that calls buffer.discardReadBytes() in CommandHandler but never calls buffer.discardReadBytes() itself. This means the buffer will keep growing forever while receiving messages from channel subscription.
Another oddity I found is that the connection is never closed on exception. Even if RedisStateMachine.decode() throws IllegalStateException the connection keeps going. Resetting the connection on unknown exceptions would have at least cleared the buffer in this case and let the application keep running, assuming the connection watchdog is enabled.

Thanks for the memory leak fix. I'll make up my mind about exceptions on connections and the result of exceptions on the whole state.

Thanks!

Released lettuce 3.0.2.Final to maven central.

Thx. An experimental feature could be close connection on internal errors, as you mentioned it. Will track it in #25

I found a better approach than closing the connection: Resetting the internal state. I updated #25 accordingly.