Deadlock with ScalaCheck 1.13.x, scalaz-concurrent, and Scala 2.12.0-RC2

After trying to port remotely to Scala 2.12.x, I found that some of the tests involving scala.concurrent.Future weren't terminating. The most likely cause is a deadlock. The issue doesn't appear in Scala 2.10.x or 2.11.x, and also not when running ScalaCheck properties through ScalaTest. It only manifests when using the plain ScalaCheck runner and 2.12.0-RC2.
A small reproduction project, including instructions, can be found here.
As of now I'm unsure whether this is a problem with scalaz.concurrent.Task, scala.concurrent.Future, or ScalaCheck (or a combination thereof). I'm opening this issue so that others who might have the same problem can find it.

Same result even with a different worker count:
> testOnly remotely.ResponseSpec -- -workers 2

deadlocks with w = 1 and 2

Thread dump? (logical or actual deadlock?)

https://gist.github.com/larsrh/7745732685193f908b64b0fe466966db

This throws after 10 second delay at object initialization time:
object ResponseSpec extends Properties("Response") {
  property("stack safety") = {
    import ExecutionContext.Implicits.global
    val f = Future(1)
    Await.result(f, 10.seconds) == 1
  }
}

Not sure if it's related or not but seems suspicious. JStack shows the same/similar stack trace as the hang in the original ResponseSpec:
"scala-execution-context-global-9" #9 daemon prio=5 os_prio=31 tid=0x00007fd89d435000 nid=0x4d03 in Object.wait() [0x0000700001149000]
   java.lang.Thread.State: RUNNABLE
        at remotely.ResponseSpec$$$Lambda$19/1836797772.apply$mcI$sp(Unknown Source)
        at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:12)
        at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:653)
        at scala.concurrent.Future$$$Lambda$21/1206883981.apply(Unknown Source)
        at scala.util.Success.$anonfun$map$1(Try.scala:251)
        at scala.util.Success.map(Try.scala:209)
        at scala.concurrent.Future.$anonfun$map$1(Future.scala:287)
        at scala.concurrent.impl.Promise$KeptPromise$Successful$$Lambda$22/1525037790.apply(Unknown Source)
        at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
        at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
        at scala.concurrent.impl.Promise$KeptPromise$Successful$$Lambda$23/727001376.apply(Unknown Source)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
        at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:140)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

hmm, I don't understand. why is remotely in the stack trace if not in the
code?
--
Cheers,
√
On Oct 29, 2016 6:31 PM, "Michael Pilquist"<EMAIL_ADDRESS>wrote:

This throws after 10 second delay at object initialization time:
object ResponseSpec extends Properties("Response") {
property("stack safety") = {
import ExecutionContext.Implicits.global
val f = Future(1)
Await.result(f, 10.seconds) == 1
}
}
Not sure if it's related or not but seems suspicious. JStack shows the
same/similar stack trace as the hang in the original ResponseSpec:
"scala-execution-context-global-9" #9 daemon prio=5 os_prio=31 tid=0x00007fd89d435000 nid=0x4d03 in Object.wait() [0x0000700001149000]
java.lang.Thread.State: RUNNABLE
at remotely.ResponseSpec$$$Lambda$19/1836797772.apply$mcI$sp(Unknown Source)
at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:12)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:653)
at scala.concurrent.Future$$$Lambda$21/1206883981.apply(Unknown Source)
at scala.util.Success.$anonfun$map$1(Try.scala:251)
at scala.util.Success.map(Try.scala:209)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:287)
at scala.concurrent.impl.Promise$KeptPromise$Successful$$Lambda$22/1525037790.apply(Unknown Source)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
at scala.concurrent.impl.Promise$KeptPromise$Successful$$Lambda$23/727001376.apply(Unknown Source)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:140)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rickynils/scalacheck/issues/290#issuecomment-257101302,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAqdwW_W-euLVJZ66nW2i4Mxh-sW-5Eks5q43S-gaJpZM4KkEq9
.

@viktorklang Only because the test class is in the "remotely" package.
Here's a further simplification which shows the same hang in jstack:
package remotely

import org.scalacheck._
import scala.concurrent.{Await,ExecutionContext,Future}
import scala.concurrent.duration._
import scala.util.{Success,Failure}
import java.util.concurrent.Executors

object ResponseSpec extends Properties("Response") {
  property("stack safety") = {
    implicit val ec = ExecutionContext.fromExecutor(Executors.newCachedThreadPool())
    Await.result(Future(1), 1000.seconds) == 1
  }
}

I think what's happening here is that when we run the tests via ScalaCheck, the properties are being instantiated as part of object initialization of ResponseSpec. However, we have a separate thread calling back in to ResponseSpec from the thread pool, and it looks like that thread spin-waits on ResponseSpec object initialization to complete (or otherwise blocks in a way that's not visible to JVM).
So overall, I don't think this is a bug, just an undesirable result of running futures from an object initializer.

One more simplification -- same hang here:
object Foo {
  implicit val ec = ExecutionContext.fromExecutor(Executors.newCachedThreadPool())
  Await.result(Future(1), 1000.seconds)

  def main(args: Array[String]) = ()
}

Dump:
2016-10-29 12:56:19
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.102-b14 mixed mode):

"Attach Listener" #10 daemon prio=9 os_prio=31 tid=0x00007fcc6c19b000 nid=0x360b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"pool-1-thread-1" #9 prio=5 os_prio=31 tid=0x00007fcc6a58c000 nid=0x4d03 in Object.wait() [0x0000700001149000]
   java.lang.Thread.State: RUNNABLE
        at remotely.Foo$$$Lambda$12/60830820.apply$mcI$sp(Unknown Source)
        at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:12)
        at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:653)
        at scala.concurrent.Future$$$Lambda$13/1698156408.apply(Unknown Source)
        at scala.util.Success.$anonfun$map$1(Try.scala:251)
        at scala.util.Success.map(Try.scala:209)
        at scala.concurrent.Future.$anonfun$map$1(Future.scala:287)
        at scala.concurrent.impl.Promise$KeptPromise$Successful$$Lambda$14/913190639.apply(Unknown Source)
        at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
        at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
        at scala.concurrent.impl.Promise$KeptPromise$Successful$$Lambda$15/922151033.apply(Unknown Source)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"Service Thread" #8 daemon prio=9 os_prio=31 tid=0x00007fcc6a029800 nid=0x4903 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007fcc6b845800 nid=0x4703 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007fcc6b833800 nid=0x4503 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007fcc6b832000 nid=0x4303 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007fcc6b83c800 nid=0x320f runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fcc6b814800 nid=0x3003 in Object.wait() [0x000070000092e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0000000795588e98> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)

Yep, @mpilquist says it well. See also http://stackoverflow.com/questions/15176199/scala-parallel-collection-in-object-initializer-causes-a-program-to-hang/15176433#15176433. This was true before 2.12.
New in 2.12 is that locking behavior is now different when lambdas are involved, as the lambda body is lifted to a member of the enclosing class (and thus an invocation locks on the class's monitor), instead of a method in a separate (anonymous) class.

Adrian,
this new behavior sounds plenty risky.
--
Cheers,
√
On Oct 29, 2016 7:57 PM, "Adriaan Moors"<EMAIL_ADDRESS>wrote:

Yep, @mpilquist https://github.com/mpilquist says it well. See also
http://stackoverflow.com/questions/15176199/scala-
parallel-collection-in-object-initializer-causes-a-program-
to-hang/15176433#15176433. This was true before 2.12.
New in 2.12 is that locking behavior is now different when lambdas are
involved, as the lambda body is lifted to a member of the enclosing class
(and thus an invocation locks on the class's monitor), instead of a method
in a separate (anonymous) class.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rickynils/scalacheck/issues/290#issuecomment-257106015,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAqd-BDK_Jd7g0E4xhWKJRmsKOpskozks5q44jNgaJpZM4KkEq9
.

A sympathy comment: I hit the same issue looking at https://issues.scala-lang.org/browse/SI-10007 and you scratch your head looking at thread dumps.

Viktor, it's a consequence of the new lambda encoding, which is also how Java does it. I don't know how we could change this and still use invokedynamic. There's a SIP for less deadlock-prone object/lazy val initialization that will likely get into 2.13. /cc @retronym

To make this more concrete:
object O {
  def f = (x: Int) => x
  f(1)
}

compiles to:
public final class O {
  public static Function1<Object, Object> f() {
    return O$.MODULE$.f();
  }
}

public final class O$ {
  public static O$ MODULE$;
  public static { new O$();  }
  private O$() {
    MODULE$ = this;
    this.f().apply$mcII$sp(1);
  }

  public scala.Function1<Object, Object> f() { return x -> x;  }
}

I probably over-interpreted it.
Is the problem only manifesting because the logic is executed as the
object initializer?
(I am a contributor to the improved lazy init so I'd of course want it as
soon as possible. 😊)
--
Cheers,
√
On Oct 30, 2016 2:47 AM, "Adriaan Moors"<EMAIL_ADDRESS>wrote:

Viktor, it's a consequence of the new lambda encoding, which is also how
Java does it. I don't know how we could change this and still use
invokedynamic. There's a SIP for less deadlock-prone object/lazy val
initialization that will likely get into 2.13. /cc @retronym
https://github.com/retronym
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rickynils/scalacheck/issues/290#issuecomment-257124760,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAqd9b0HUHxvh2bCVoaINxaIzzb0D8cks5q4-kPgaJpZM4KkEq9
.

So, to summarize: It's not a problem in ScalaCheck per se, but it is exhibited by the fact that the ScalaCheck runner

uses objects instead of instantiating classes, and
if a property is constructed from a plain Boolean, this is not done lazily

Is that assessment correct?

The simplest change which would fix this issue is something like this:
-  property("stack safety") = {
+  property("stack safety") = P {
     import ExecutionContext.Implicits.global
     fromFuture(Future(1)).run == 1
   }
 }
+
+object P {
+  def apply(body: => Boolean): Prop =
+    Prop(_ => Prop.Result(if (body) Prop.True else Prop.False))
+}

@larsrh @rickynils Do you think ScalaCheck should support something like this? If P was instead some method on Prop it would make fixing these issues relatively straightforward (albeit still annoying).

Is the problem only manifesting because the logic is executed as the object initializer?

yep, as far as I understand -- all access indirect through the module field, which is locked during initialization, and re-entry (e.g., by invoking a lambda which is now implemented as a method defined in the module class, and is thus invoked through the module field) will lock

I guess we'll see how much of a problem this will become—it's not going to
be easy or obvious to debug.
Are there any chances of getting the improved lazy val SIP retrofitted onto
2.12 somewhere down the line?
On Sun, Oct 30, 2016 at 8:30 PM, Adriaan Moors<EMAIL_ADDRESS>wrote:

Is the problem only manifesting because the logic is executed as the
object initializer?
yep, as far as I understand -- all access indirect through the module
field, which is locked during initialization, and re-entry (e.g., by
invoking a lambda which is now implemented as a method defined in the
module class, and is thus invoked through the module field) will lock
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rickynils/scalacheck/issues/290#issuecomment-257172955,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAqdwL_VvJsK3gNUVBAfHgOapcwxfNsks5q5PBggaJpZM4KkEq9
.

--
Cheers,
√

@larsrh @rickynils Do you think ScalaCheck should support something like this? If P was instead some method on Prop it would make fixing these issues relatively straightforward (albeit still annoying).

IMHO this should be the default behaviour (lazy evaluation), because I assume quite a few people will hit this issue.

Note this works:
object ResponseSpec extends Properties("Response") {

  def fromFuture[A](f: Future[A])(implicit E: ExecutionContext): Task[A] =
    Task.async { cb => f.onComplete {
      case Success(a) => cb(\/.right(a))
      case Failure(e) => cb(\/.left(e))
    }}

  property("stack safety") = Ext.suspend {
    import ExecutionContext.Implicits.global
    fromFuture(Future(1)).run == 1
  }
}

object Ext {
  def suspend[P <% Prop](p: => P): Prop = Prop { prms => (p: Prop)(prms) }
}

I don't think there's any other way to solve it in ScalaCheck 1.13.x -- the main culprit is the propBoolean implicit conversion, which is defined strictly:
  implicit def propBoolean(b: Boolean): Prop = Prop(b)

Theoretically, we could change that (and maybe other conversions) to be by-name like:
  implicit def propBoolean(b: => Boolean): Prop = Prop(_ => b)

@mpilquist We could keep that method but make it non-implicit, and add a new by-name implicit. That would maintain binary compatibility and fix the issue I think.

@non oooo good idea!

I'll try it out now.

Cool, I hope it is fixable.
On Mon, Oct 31, 2016 at 5:21 PM, Erik Osheim<EMAIL_ADDRESS>wrote:

I'll try it out now.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rickynils/scalacheck/issues/290#issuecomment-257341238,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAqd9-VaQieyl19Xl1YNeF3vP50bF8oks5q5hV4gaJpZM4KkEq9
.

--
Cheers,
√

So, the simple fix did not work.
I tried creating implicit def propBooleanLazy(b: => Boolean): Prop and then making propBoolean non-implicit. This doesn't seem to work -- in at least some cases when a Boolean => Prop is asked for, a (=> Boolean) => Prop is not sufficient. I tried creating a lower-priority implicit of the old type, but then that gets preferentially used even in the case we care about.
If I could get propBooleanLazy to be chosen at the property definition site then things would be fine. But I haven't yet found a good way to do this.

@non How about this instead?
diff --git a/src/main/scala/org/scalacheck/Prop.scala b/src/main/scala/org/scalacheck/Prop.scala
index cc0e92d..ce04f6b 100644
--- a/src/main/scala/org/scalacheck/Prop.scala
+++ b/src/main/scala/org/scalacheck/Prop.scala
@@ -300,6 +300,8 @@ object Prop {
   /** Create a property from a boolean value */
   def apply(b: Boolean): Prop = if(b) proved else falsified

+  def suspend(p: Prop): Prop = apply(prms => p(prms))
+

   // Implicits

diff --git a/src/main/scala/org/scalacheck/Properties.scala b/src/main/scala/org/scalacheck/Properties.scala
index 972bae2..5029ff1 100644
--- a/src/main/scala/org/scalacheck/Properties.scala
+++ b/src/main/scala/org/scalacheck/Properties.scala
@@ -93,7 +93,7 @@ class Properties(val name: String) {
    *  }}}
    */
   sealed class PropertySpecifier() {
-    def update(propName: String, p: Prop) = props += ((name+"."+propName, p))
+    def update(propName: String, p: => Prop) = props += ((name+"."+propName, Prop.suspend(p)))
   }

   lazy val property = new PropertySpecifier()

Oh, that looks great! Does that pass mimaReportBinaryIssues?

Yep, mima passes

:shipit:

mimaPreviousArtifacts wasn't configured -- once I changed it to compare against 1.13.3, it fails of course. I have a fix that passes tests and mima by making the old update method protected. PR incoming shortly.

FYI: I don't think this problem is inherent to invoke dynamic and lambdas.
The solution is to not never methods static in top-level unless you really have to.
I'm adding version by @mpilquist to Dotty test suite https://github.com/lampepfl/dotty/pull/2283
The solution comes from this PR, if I remember it right: https://github.com/lampepfl/dotty/pull/628

Sorry for being late for party. @smarter invited me to this discussion long ago, but I didn't notice the email.
ping @adriaanm


Use libsvm in hadoop

Hello, everyone.
I want to ask something. Can i use libsvm in apache hadoop ?
Is it work with map reduce programming model in hadoop ?

why do you want to use libsvm on hadoop?
I think it might be improper to apply kernel svm on map reduce settings cause currently it can not handle too large data sets

I want to do performance analysis in Hadoop and Spark using SVM algorithm.
If i only use less than 1 GB dataset, it is ok ?
I have read many paper that Hadoop can implement SVM algorithm, but no one explain what library they use. So, i start with libSVM.
So, What should i do ? Or are there another SVM library that support map reduce programing model ?

Spark ML contains an implementation of Linear SVMs, similar too, but not as comprehensive as those in LibLINEAR. As @infwinston mentioned, SVMs with kernels, which is what LibSVM is for, are not really suited for Hadoop and Spark, since they don't scale well to large datasets.

I think you may want to check out LIBLINEAR webpage.
in some cases, Linear SVMs give good enough performance and get way faster than Kernel SVM.


Update README.md to something approaching good

Closes #98

Codecov Report
Patch and project coverage have no change.

Comparison is base (fbb7aa0) 84.92% compared to head (898070c) 84.92%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #106   +/-   ##
=======================================
  Coverage   84.92%   84.92%
=======================================
  Files          13       13
  Lines        2149     2149
=======================================
  Hits         1825     1825
  Misses        233      233
  Partials       91       91

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Updated Big Fan of it to 1.0.7

^ Bumped Big Fan of it to 1.0.7

don't merge yet I'm in process of fixing something

superseded by #12


No more parse_raw_as in Pydantic v2

Initial Checks

[X] I have searched Google & GitHub for similar requests and couldn't find anything
[X] I have read and followed the docs and still think this feature is missing

Description
Methods such as parse_* have been removed in pydantic v2.
The motivation for this change is that <class>.model_validate() has a similar behavior.
But one use case is not covered anymore, parse_raw_as when the type is not a Pydantic BaseModel.
For example, I don't know how to do this in v2

from pydantic import BaseModel, parse_raw_as

class Env(BaseModel):
    name: str
    value: str

EnvList = list[Env]

my_list = parse_raw_as(EnvList, [{"name": "key1", "value": "value1"},{"name": "key2", "value": "value2"}])

Is there a function for this?
Affected Components

[X] Compatibility between releases
[X] Data validation/parsing
[ ] Data serialization - .model_dump() and .model_dump_json()
[ ] JSON Schema
[ ] Dataclasses
[ ] Model Config
[ ] Field Types - adding or changing a particular data type
[ ] Function validation decorator
[ ] Generic Models
[ ] Other Model behaviour - model_construct(), pickling, private attributes, ORM mode
[ ] Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

Selected Assignee: @adriangb

in v1, this simple work:
>>> pydantic.parse_raw_as('List[str]', '[1,2]')
['1', '2']

Anyway to get the same results with v2?

TypeAdapter(list[int]).validate_json(‘[1])`


[Prism] Implement PrismPipelineResult

This PR closes #31793 with a package private PipelineResult implementation. The goal of tests are to validate shutting down the Prism executable during termination-related methods (cancel, waitUntilFinish) and the storing of terminal State and metrics. To focus tests on this behavior without executing Pipelines, it uses the mockito library.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

R: @Abacn


Formatting code is showing warnings in the terminal

For example, I get this:
warning: parentheses are required when piping into a function call. For example:

    foo 1 |> bar 2 |> baz 3

is ambiguous and should be written as

    foo(1) |> bar(2) |> baz(3)

Ambiguous pipe found at:
  nofile:7:3

warning: an expression is always required on the right side of ->. Please provide a value after ->
  nofile:8:19

For code like this foo |> bar fn -> end. Maybe we want to show it to users? Or discard it?

If there is a syntax error while formatting, we are not showing those either. :)

@josevalim honestly it would be weird to add the warning to cell outputs as a result of formatting with ctrl + shift + i.
For cases like piping the formatter would pick one interpretation, and if that's not what the user wants then can use ctrl + z and correct it.
I'd suppress formatting warning altogether, ideally via an option to Code.format_string!/2 if adding that is viable.

Alright, I will tackle this on Elixir side then. I will let you know if we do this by default or behind an option. However, we need to handle the syntax error. :)

@josevalim for syntax errors the same point applies, showing the error on formatting shortcut is weird, and they will see the error as soon as they evaluate.

It doesn’t need to be output but today if there is a syntax error, then nothing happens and there is no feedback. I think we need at least some feedback. The best would be if we could rescue those errors and show those as squiggly lines in the editor!

The warning part has been fixed in Elixir. We still need to show errors though. If we can do it using editor integration (instead of part of output), then even better!

@josevalim fantastic, thanks!


Update README.md

Replaced old github links from Jo3-L to yagpdb-cc

Does not seem as an urgent change, as the old Links still point to the according folders and files in this repository, however for consistency it is indeed feasible to update them.

Also bois if u gimme perms i can review pr's and add stuff here without joe and satty having to do all work alone .-.

Those links have been updated already in #90 (if not there, then in the local copy of that branch that needs pushing)

Those links have been updated already in #90 (if not there, then in the local copy of that branch that needs pushing)

holy shizzle, you changed a lot of things in ur pr

lgtm, thanks!
also, check your invites, I invited you to the yagpdb-cc organization


fix(spot): Prevent both instance_requirements and instance_type

Description

I'm using this module to create ASG and resources associated to be used in my ECS cluster.
Motivation and Context

I searched to use for_each on this module to create on-demand and spot ASG/LaunchTemplate. I faced a problem because you can't declare instance_requirements if instance_type is set too.
To avoid this problem, I added a check that var.instance_type == null when you declare your instance_requirements.
Personal code example
if my asg is spot == false I keep the default instance_requirements as {} and set instance_type
if my asg is spot == true I set the instance_requirements, use_mixed_instances_policy and set the instance_type to null
locals.tf
auto_scaling_groups = {
        "on-demand" = {
          min_size     = 0
          max_size     = 5
          desired_size = 1 # only use at first deploy then manages by ECS CAS (Cluster Auto Scaling)
          spot         = false
          ec2_type     = "t3a.medium" # x64
        }
        "spot-burstable" = {
          min_size     = 0
          max_size     = 5
          desired_size = 1
          spot         = true
          spot_configuration = {
            allowed_instance_types = ["*"] # Must contains one or more wildcards (examples: m5.8xlarge, c5*.*, m5a.*, r*, *3*)
            memory_gb_min          = 4
            memory_gb_max          = 4
            vcpu_min               = 2
            vcpu_max               = 2
            burstable_performance  = "required"       # Valid Values: included | required | excluded
            cpu_manufacturers      = ["intel", "amd"] # Valid Values: intel | amd | amazon-web-services
          }
        }
      }

asg.tf
module "asg" {
  source = "./modules/terraform-aws-autoscaling/"

  for_each = local.env_vars.auto_scaling_groups

  # ... for brevity

  instance_type   = each.value.spot == false ? each.value.ec2_type : null

  # --------------------------------------------------
  # Spot configuration
  # --------------------
  instance_requirements = {
    allowed_instance_types = each.value.spot ? each.value.spot_configuration.allowed_instance_types : null
    burstable_performance  = each.value.spot ? each.value.spot_configuration.burstable_performance : null # Valid Values: included | required | excluded
    cpu_manufacturers      = each.value.spot ? each.value.spot_configuration.cpu_manufacturers : null     # Valid Values: intel | amd | amazon-web-services
    memory_mib = {
      max = each.value.spot ? each.value.spot_configuration.memory_gb_max * 1024 : null
      min = each.value.spot ? each.value.spot_configuration.memory_gb_min * 1024 : null
    }
    vcpu_count = {
      max = each.value.spot ? each.value.spot_configuration.vcpu_max : null
      min = each.value.spot ? each.value.spot_configuration.vcpu_min : null
    }
  }
  use_mixed_instances_policy = each.value.spot ? true : false
  mixed_instances_policy = {
    instances_distribution = {
      spot_allocation_strategy = "lowest-price"
      spot_max_price           = "" # on-demand price
      spot_instance_pools      = 2
    }
  }
  capacity_rebalance = each.value.spot ? true : false
  # --------------------------------------------------

  # ... for brevity
}

Breaking Changes
No breaking changes.

How Has This Been Tested?

[ ] I have updated at least one of the examples/* to demonstrate and validate my change(s)
[ ] I have tested and validated these changes using one or more of the provided examples/* projects

[x] I have executed pre-commit run -a on my pull request
[x] I tested it locally in my stack

why do you need to use this in a for_each loop? seems like the problem is solved by having two ASG definitions, and will be much more stable

@bryantbiggs yes it could be done with both ASG module declarations (150 lines for me right now for one ASG). In this example, I have two ASG, but you can go for example to 5 in your ECS cluster (small, medium, spot, on-demand instances)
The difference is not a huge change from the module perspective. And it fix an AWS limitation that when you declare instance_requirements you should not declare instance_type.
I hope you will find it helpful as it unlocks a new possibility.

but you can go for example to 5 in your ECS cluster (small, medium, spot, on-demand instances)

For what?

The difference is not a huge change from the module perspective. And it fix an AWS limitation that when you declare instance_requirements you should not declare instance_type.

There isn't anything broken so there isn't anything to fix in terms of limitations - its just that you are trying to force two very different configurations into one

For what?

To place your ECS services depending on the application requirements in small, medium or large node. And if you are allowed to use spot, you can either run your workload 100% on spot or balance it between on-demand and spot. All these considerations are base on finOps. So it's a common thing to have multiple ASG in an ECS cluster.

its just that you are trying to force two very different configurations into one

Maybe, but I prefer using for_each loop when it's possible. You seem preferring duplicate a hundred lines of HCL.
Your comments are not based on the contribution, as it's just a small thing unlocking a huge possibility. It's more based on your HCL writing vision.
Any third opinion would be helpful @antonbabenko


Explanation of scope

The readme says that cretonne is supposed to be a code generator for WebAssembly, but it's a bit unclear what that means, seeing how there doesn't appear to be anything to actually do with wasm in the code.

I heard that cretonne is meant to have inter-procedural optimizations as another layer built over top. Is this true, and always going to be the case?
Does another tool to translate wasm to cretonne IR need to be built in order to generate code for wasm?
Is it in scope for cretonne to output to wasm, or is it only intended to output to native code? (e.g. will I be able to use cretonne as a pure-rust binaryen alternative?)
Is cretonne meant to be useful in both JIT and AOT configurations?

Preferably, the readme or documentation should be able to answer these questions.

Cranelift (formerly Cretonne) is currently focusing on "backend" compilation, however if it is successful, we hope to extend it some day with high-level inter-procedural optimizations.
While Cranelift includes the cranelift-wasm crate for translating wasm, it isn't sufficient for complete wasm compilation. One needs additional support, such as the support now provided by Wasmtime.
It is theoretically possible that Cranelift could output to wasm some day, however it is not set up to do so right now.
Cranelift is indeed meant to be useful in both JIT and AOT configurations. Lucet and rustc_codegen_cranelift are examples of using it in an AOT configuration, and Wasmtime is an example of using it in a JIT configuration.
If you have any further questions, please re-open this issue or file a new one!


Add support for custom error pages

This sets up a custom 404 page and a generic error page which is used for all other 4xx and 5xx status codes.
In order to allow for both config and the built site to be uploaded we:

add nginx config for error pages which will be included within the location block
modify middleman config to build into /deploy/public
add the new build directory to .gitignore
modify the CloudFoundry manifest to upload the entire /deploy/ directory
add a Staticfile configured to serve only /deploy/public and include the custom nginx config

Thanks for the review!

Thanks for taking as though this PR Ollie, that was really helpful. To me the logic of it seems sound.


Want to add preloader to website

🚀 Feature
I want to add preloader in frontend website.
Have you read the Contributing Guidelines on Pull Requests?
yes

/assign


Add mdx serialization support

[ ] Add support for mdx_jsx and move over the tests

This PR is dependent on #152

@wooorm
This is roughly 60% finished, but I want to take your opinion on the way of how you're doing tracking and some of the (monkey patches) that are in the js version for mdx-js.
In mdx-js there's a copy of the containerFlow function here https://github.com/syntax-tree/mdast-util-mdx-jsx/blob/main/lib/index.js#L709
It's a modified copy which seems to have more logic in it for indent, but it's only called from within the mdx-jsx module, my question here is does that mean mdx-jsx doesn't care about any of the previous tracking that have happened while we were serializing the tree ? If it's not the case do you think we should add support for tracking in mdast-util-to-markdown?
If we did so the Rust version might be out of sync with the JS one, but we can probably also make the same PR in the JS version, and I honestly don't know how many of the tests are going to change (if any).
There are some other overridden options when we have a tree with jsx nodes here https://github.com/syntax-tree/mdast-util-mdx-jsx/blob/main/lib/index.js#L537
Does that mean in the Rust version at run time we need to know if we have a jsx node before we start serializing ? Or should we let the user configure this option by themselves and error or panic when we find a jsx node with the wrong configuration.
We can be explicit about the second option in the docs, what do you think ?

I don't have the bandwidth to address any of the issues in this PR if someone would like to build on top of this draft feel free


Fix incorrect name in "TopN queries" docs

@abbondanza can you sign http://druid.io/community/cla.html?

@fjy signed.

@abbondanza signed!
On Thu, Jun 8, 2017 at 9:07 PM Fangjin Yang<EMAIL_ADDRESS>wrote:

@abbondanza https://github.com/abbondanza can you sign
http://druid.io/community/cla.html?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/druid-io/druid/pull/4386#issuecomment-307267768, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AE3Ovy972bOwyF_uD5GzvZKlY6I3sIBqks5sCJrtgaJpZM4N0o_l
.

 But I say to you, Love your enemies and pray for those who
persecute you."
-Jesus; Matthew 5:43-44


create-app: add app-backend plugin to template

Hey, I just made a Pull Request!
Not actually verified yet, but up for some early review :grin:
Some more context:
With #3264 now merged, any config loaded by the backend will also be filtered by visibility and injected into the frontend by the app-backend plugins. With that the plugin is a lot less confusing to use and much more powerful than other deployment alternatives with separate app/backends. It's still perhaps not what we'd recommend for a scaled deployment with high availability, but it's gonna be good enough for 95% of all usage and what I think we should recommend going forward.
:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)
[x] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)

Not sure the best place to document this, or if it's documented already, but we should mention that this route will only serve the production / built version of the frontend, so it does not replace the yarn start in the frontend package for development


PyJWT version 2.0.0 dropped "ExpiredSignature" as an alias for "ExpiredSignatureError", which breaks AdfsBaseBackend.validate_access_token()

PyJWT version 2.0.0 dropped "ExpiredSignature" as an alias for "ExpiredSignatureError", so I monkey-patched it back to avoid an AttributeError that was being raised inside AdfsBaseBackend.validate_access_token()

Hmm, I'm not seeing the reason behind a monkeypatch, instead of a proper fix here.
As discussed in #118 the proper fix is to change the code, bump the requirement version of jwt to 2.x and release a new version of django-auth-adfs. The users who cannot upgrade jwt to 2.x will have to pin their version to jwt<2.0 in their requirements file.

Hmm, I'm not seeing the reason behind a monkeypatch, instead of a proper fix here.
As discussed in #118 the proper fix is to change the code, bump the requirement version of jwt to 2.x and release a new version of django-auth-adfs. The users who cannot upgrade jwt to 2.x will have to pin their version to jwt<2.0 in their requirements file.

Hi, @JonasKs.
I'm aware monkey-patching is not considered best practice, I was trying to avoid a breaking change in django-auth-adfs by dropping compatibility with PyJWT versions prior to 2.x. But I guess you are right.

Hi, @JonasKs.
I'm aware monkey-patching is not considered best practice, I was trying to avoid a breaking change in django-auth-adfs by dropping compatibility with PyJWT versions prior to 2.x. But I guess you are right.

I understand👍 I think it’s a bit too hacky, since it’s a reimplementation of a deprecated API in a third party package, which is as you say not a good practice.
Would you mind implementing the suggested solution in #118 ? 😊

I understand👍 I think it’s a bit too hacky, since it’s a reimplementation of a deprecated API in a third party package, which is as you say not a good practice.
Would you mind implementing the suggested solution in #118 ? 😊

Thanks for your contribution, but #120 has been accepted.

Thanks for your contribution, but #120 has been accepted.

Hey @VBobCat, no worries. Thanks for making a pull request! 🎉

Hey @VBobCat, no worries. Thanks for making a pull request! 🎉


Fix examples

bounds: BoundingObject -> IBoundingObject
engine: "BoundingAABB" object doesn't have properties set on it, "Spikes" does.

Hey, thanks for the fixes - appreciate it~!


Fix broken headings in Markdown files

GitHub changed the way Markdown headings are parsed, so this change fixes it.
See bryant1410/readmesfix for more information.
Tackles bryant1410/readmesfix#1

Cool, thanks!


Add trytond module dependencies

Motivation for this change
I am working on adding support for the modules of trytond. This PR contains a first chunk of dependencies needed to install them.
Details about the modules which are available in the project: http://doc.tryton.org/4.2/modules/index.html
Things done

[ ] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[x] NixOS
[x] macOS
[ ] Linux

[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Fits CONTRIBUTING.md.

On my NixOS box things did build cleanly after disabling tests for zope_configuration on python 3.
On darwin I was not able to build all packages via nox, mainly due to issues around the cffi package on python35, it has failing tests.

Failing tests with cffi? That is not good. Could you turn on the sandbox?

Hmm, not sure how the sandbox on darwin works, need to set up a few bits most probably.
I've tried to build things on master via nix-build python35Packages.cffi, it's failing in a similar way there:
====== 197 failed, 1588 passed, 122 skipped, 4 xfailed in 396.50 seconds =======
builder for ‘/nix/store/894al50a57swq9dyzl2h6r87jqrnar05-python3.5-cffi-1.9.1.drv’ failed with exit code 1
error: build of ‘/nix/store/894al50a57swq9dyzl2h6r87jqrnar05-python3.5-cffi-1.9.1.drv’ failed

Running the same on top of this change gave me this result:
====== 197 failed, 1588 passed, 122 skipped, 4 xfailed in 367.21 seconds =======
builder for ‘/nix/store/894al50a57swq9dyzl2h6r87jqrnar05-python3.5-cffi-1.9.1.drv’ failed with exit code 1
error: build of ‘/nix/store/894al50a57swq9dyzl2h6r87jqrnar05-python3.5-cffi-1.9.1.drv’ failed

My assumption is that the change in this PR did not really affect it. The build with python 2.7 just went fine on darwin.

Is there any check left which I could do for this PR?

thanks for the reminder


Comfy UI segmentation fault

Comfy UI closes with segmentation fault(core dumped) after importing this. Tried in a fresh comfy ui install with only ComfyUI-DragNUWA extension

Worked after downgrading to python 3.10


As a developer i want issues to automatically be added to the project board too minimise manual work

Issues should automatically be added to the project board in the appropriate location.
New issues should be added to Not started.
Issues with label bug should be added Bugs
Accaptance:

When a new issues is created that is not labeled bug it gets added to Not started
When a new issues is created that is labeled bug it gets added to Bugs

I've tried to use the .yml workflow file to automate the issues, but that shit don't work.


Restructure the controller tests

In this PR the controller tests were restructured to avoid race conditions

I rebased this PR on https://github.com/netbox-community/netbox-operator/pull/29 so we can see if the integration test is properly fixed!

Integration test CI is now passing!

Hey @bruelea sorry that I have no idea what happened that made this PR merged (I didn't push the button...), but it's actually not merged into the main branch (magic?)
Can you open a new PR with your branch? Thanks .. :)

Hey @bruelea sorry that I have no idea what happened that made this PR merged (I didn't push the button...), but it's actually not merged into the main branch (magic?)
Can you open a new PR with your branch? Thanks .. :)

Done. Thanks @henrybear327  for testing the PR in the pipeline!


JAV-469 Edge lb new instance cache manager

edge use new InstanceCacheManager, normal consumer use old InstanceCacheManager
if there is no problem, then old InstanceCacheManager will be deleted.

Coverage decreased (-0.01%) to 86.458% when pulling b5b95706a304a327b47b3faf7b6ead0690365f2b on wujimin:edge-lb-new-instance-cache-manager into 245ed36795905304709190546f3f459bc05ec47e on ServiceComb:master.


Fragment instance (again :( )

I'm using vfg in another project with webpack. And vfg is marked as a fragment instance. I don't get it...
Same Vue version as the one use in the dev example.

Could you show the template code?

I tested the new version it in another project and no problem there. I will try to pinpoint the exact problem.

Weird :confused:

Ok, I found the problem and it is dumb.
I imported the components in a 'normal' way:
components: {
    VueFormGenerator
}

And this is not how it work for this plugin. The correct way:
components: {
    VueFormGenerator: VueFormGenerator.component
}

I'm looking at a way to make it behave like normal components

The reason is that library contains validators and schema util functions too.
https://github.com/icebob/vue-form-generator/blob/master/src/index.js#L2
So you need to add as
components: {
    VueFormGenerator: VueFormGenerator.component
}

or register globally when your app start
Vue.use(VueFormGenerator);


fix the panic in table

Changes

:bug: Fix bug for issue #35
:broom: update current implementation for table printing

@maximilien ,  can you help to review again after rebase?

Seems like there was another update to deps. Let's see if this can still merge... Let's monitor jobs


Newsletter 9: Add ABstreet, Citybound and Scale updates

Part of #111

Tweaked formatting and wording a bit to make the PR more consistent with the rest of the newsletter.

Merged, thanks for the PR!

@aeplay fyi ^

Thanks for the feature! @ozkriff @Uriopass


Add option for ignoring TS errors

tsec is a wrapper around tsc and reports errors from both custom rules related to Trusted Types as well as standard TypeScript errors (e.g. strict checks). This is expected behavior in most cases as TypeScript errors might prevent tsec from doing it's job.
We have problem with this approach in complex CI/CD pipelines where we already have jobs that handle standard TypeScript errors. We would like to keep tsec job focused solely on issues related to Trusted Types - fail only on custom checks and ignore others as they are handled (and reported) elsewhere.
It would be great if we could add a switch/option into tsec that would be false by default and only enable this behavior when needed.

I will create a patch for this.
On a related note, would it be helpful in your case if we release an eslint-based tsec? Does that ease your CI/CD integration?

@uraj , if we could integrate it with https://github.com/microsoft/eslint-plugin-sdl/ that would be a big help. But we use eslint also on plain JS as well as TS without --project and thus without full type checking. We should be prepared to handle those scenarios.

Any progress on this issue?

I have a patch already and is going through internal code review.

Please try 0.2.8.


Problems running SWAN on macOS Catelina with XCode Command Line Tools 12.3 (12C33)

I am trying to run SWAN on my machine and I encounter an error while running the example file because the system couldn't load a library called libcast.dylib. Couldn't find any help regarding this library libcast.dylib on Google.
[~/projects/maple-lab]$ ./swan-release/cli/run-swan-single -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDK
s/MacOSX.sdk -path ./swan-release/test-files/Simple.swift
Could not find shared library!
java.lang.UnsatisfiedLinkError: /Users/moazin/projects/maple-lab/swan-release/lib/libswiftWala.dylib: dlopen(/Users/moazin/projects/
s/maple-lab/swan-release/lib/libswiftWala.dylib, 1): Library not loaded: @rpath/libcast.dylib
  Referenced from: /Users/moazin/projects/maple-lab/swan-release/lib/libswiftWala.dylib
  Reason: image not found
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817)
        at java.lang.Runtime.load0(Runtime.java:810)
        at java.lang.System.load(System.java:1086)
        at ca.maple.swan.swift.translator.sil.SwiftTranslatorPathLoader.load(SwiftTranslatorPathLoader.java:39)
        at ca.maple.swan.swift.translator.wala.SwiftToCAstTranslator.<clinit>(SwiftToCAstTranslator.java:55)
        at ca.maple.swan.swift.server.SwiftAnalysisEngineServerDriver.generateSDG(SwiftAnalysisEngineServerDriver.java:110)
        at ca.maple.swan.swift.test.GenericTester.doTest(GenericTester.java:66)
        at ca.maple.swan.swift.test.GenericTester.<init>(GenericTester.java:50)
        at ca.maple.swan.swift.test.GenericTester.main(GenericTester.java:182)
Exception in thread "main" java.lang.UnsatisfiedLinkError: ca.maple.swan.swift.translator.wala.SwiftToCAstTranslator.translateToCAstNodes(Ljava/util/ArrayList;)Lj
ava/util/ArrayList;
        at ca.maple.swan.swift.translator.wala.SwiftToCAstTranslator.translateToCAstNodes(Native Method)
        at ca.maple.swan.swift.translator.sil.RawData.setup(RawData.java:69)
        at ca.maple.swan.swift.server.SwiftAnalysisEngineServerDriver.generateSDG(SwiftAnalysisEngineServerDriver.java:112)
        at ca.maple.swan.swift.test.GenericTester.doTest(GenericTester.java:66)
        at ca.maple.swan.swift.test.GenericTester.<init>(GenericTester.java:50)
        at ca.maple.swan.swift.test.GenericTester.main(GenericTester.java:182)

I'm running macOS Catalina Version 10.15.6 and XCode Command Line Tools version is 12.3 (12C33). Java version is 1.8.0_271

Turns out that LC_RPATH in libswiftWala.dyld are hardcoded to @tiganov's machine, I had to build SWAN and WALA to grab the libcast.dyld file to get around this error.
But even with that out the way, building SWAN on Catalina is turning out to be a nightmare. Two errors come up from the Swift compiler (I guess):
<unknown>:0: error: unknown argument: '-enable-experimental-concise-pound-file'
<unknown>:0: error: unable to load standard library for target 'x86_64-apple-macosx10.15'

I think solving these would require rebuilding the translator code which would require building the Swift compiler at that specific tag.

Hi,
Compiling Swift/SWAN on Catalina did not work for us either. This is why we packaged the .dylib with the release. Unfortunately, it turns out this .dylib is hardcoded to my machine.
We are currently working on a new language frontend that will not require building Swift or hooking into the Swift compiler. We expect to push it within the next month.

Hi,
Compiling Swift/SWAN on Catalina did not work for us either. This is why we packaged the .dylib with the release. Unfortunately, it turns out this .dylib is hardcoded to my machine.
We are currently working on a new language frontend that will not require building Swift or hooking into the Swift compiler. We expect to push it within the next month.

Oh, okay, great. Should I keep the issue open?

Oh, okay, great. Should I keep the issue open?

Sure, you can keep it open for now until we push the new frontend.

Sure, you can keep it open for now until we push the new frontend.

Actually, I'm going to close this issue. We are actively working on fixing this problem, but we will not push anything until it is ready for use.

Actually, I'm going to close this issue. We are actively working on fixing this problem, but we will not push anything until it is ready for use.


fix(STADTPULS-605): Avoid wrong positioning of hero at 1024px

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Update  Azure.Sdk.Tools.NotificationConfiguration in globals.yml from 20230108.1 to 20230119.1

Updating to deploy this PR:

#5172

@weshaggard I updated the matcher logic. Please have a look once you have a moment :) Thanks!

/check-enforcer override


Blacken the source code and sort imports

Thanks for submitting a PR!
Please make sure to check for the following items:

[ ] Add unit tests and integration tests where applicable.
If you've added an error code or changed an error code behavior,
you should probably add or change a test case file under tests/test_cases/ and add
it to the list under tests/test_definitions.py.
If you've added or changed a command line option,
you should probably add or change a test in tests/test_integration.py.
[ ] Add a line to the release notes (docs/release_notes.rst) under "Current Development Version".
Make sure to include the PR number after you open and get one.

Please don't get discouraged as it may take a while to get a review.

@Nurdok bump

Sorry for the delay @samj1912. Between the first review and now, my wife managed to give birth to a healthy baby boy :)


flask's flash function does not work from inside a meld component

I have a meld component and the flask's flash function does not work after successful registration. Everything else is fine and working
Meld Component
class RegistrationForm(Component):
    def save(self):
        self.submitted = True
        if self.validate():
            # ...
            flash('Your account has been created!', 'success')
            return redirect(url_for('user.login'))

Meld Template
{% from "components/fields.html" import field %}

    <form meld:submit.prevent id="registerForm" action="" method="POST">
        {{field('first_name', fa_icon='user', form=form)}}
        {# ... fields here #}

            {{ form.submit(class="btn btn-grad",
                **{"meld:click": "save()",
                    "meld:loading.attr": "disabled",
                    "meld:loading.class": "opacity-80 cursor-not-allowed"}) }}

    </form>

Flask Template
{% extends "layout.html" %}

{% block title %}Register{% endblock %}

{% block content %}
    {% meld "registration_form" %}
{% endblock %}

Will have to dig into this one more to see if we can persist the flash messages

Any Update?

This may be a tough one to tackle: https://stackoverflow.com/a/59577538
Essentially, we lose the flash messages as the flask cookie does not get updated within a Meld component.


"None of [Index([382], dtype='int64')] are in the [index]"

Hi i'm getting the following error and have been unable to debug.
Thanks in advance!
kf = KFold(n_splits=5, shuffle=True, random_state=seed)

for fold, (train_index, test_index) in enumerate(kf.split(df)):
    print(f"Processing fold {fold + 1}")
    X_train, X_test = X.iloc[train_index].copy(), X.iloc[test_index].copy()
    y_train, y_test = y.iloc[train_index].copy(), y.iloc[test_index].copy()

    print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)

    clf = RandomForestClassifier(n_jobs=-1) #for categorical
    rgr = RandomForestRegressor(n_jobs=-1) #for numerical
    imputer = MissForest(clf,rgr)

    X_train_imputed = imputer.fit_transform(X_train, categorical = cat_col)
    X_test_imputed = imputer.transform(X_test)

    # Save imputed datasets
    train = pd.concat([X_train_imputed, y_train], axis=1)
    test = pd.concat([X_test_imputed, y_test], axis=1)
    train.to_feather(f'Data/Imputed/RFI_fold{fold + 1}_train.feather')
    test.to_feather(f'Data/Imputed/RFI_fold{fold + 1}_test.feather')

KeyError                                  Traceback (most recent call last)
Cell In[9], line 15
     12 imputer = MissForest(clf,rgr)
     14 X_train_imputed = imputer.fit_transform(X_train, categorical = cat_col)
---> 15 X_test_imputed = imputer.transform(X_test)
     17 # Save imputed datasets
     18 train = pd.concat([X_train_imputed, y_train], axis=1)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\missforest\missforest.py:475, in MissForest.transform(self, x)
    473 # Predict the missing column with the trained estimator
    474 miss_index = self._missing_row[c]
--> 475 x_missing = x_imp.loc[miss_index]
    476 x_missing = x_missing.drop(c, axis=1)
    477 y_pred = estimator.predict(x_missing)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\pandas\core\indexing.py:1192, in _LocationIndexer.__getitem__(self, key)
   1190 maybe_callable = com.apply_if_callable(key, self.obj)
   1191 maybe_callable = self._check_deprecated_callable_usage(key, maybe_callable)
-> 1192 return self._getitem_axis(maybe_callable, axis=axis)

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\pandas\core\indexing.py:1421, in _LocIndexer._getitem_axis(self, key, axis)
   1418     if hasattr(key, "ndim") and key.ndim > 1:
   1419         raise ValueError("Cannot index with multidimensional key")
-> 1421     return self._getitem_iterable(key, axis=axis)
...
-> 6248         raise KeyError(f"None of [{key}] are in the [{axis_name}]")
   6250     not_found = list(ensure_index(key)[missing_mask.nonzero()[0]].unique())
   6251     raise KeyError(f"{not_found} not in index")

KeyError: "None of [Index([382], dtype='int64')] are in the [index]"

Hi, I'm facing the same problem.
I think the error occurs because the transform method looks for dataframe row indices in the training set that have missing value for a certain feature.
Then when applied on the test set dataframe, the method does not find these indexes in there, since they belong to the training set dataframe.

I figured out what is happing, at least for my case.
Whenever the transform method is called, the get_missing_rows(x) method is also called. The latter populates the _missing_row dictionary with a series of dictionaries in which:
key -> feature: values -> list of indexes corresponding to the input df for which the feature has missing value.
When calling transform to impute a test or validation set, after applied fit_transform on a training set, the _missing_row dictionary is updated and not overriden.
I think the solution may be simply to insert:

self._missing_row = {}

at the beginning of the get_missing_rows method's definition.
I tried it and it seems to work.

@Sep905 you are right absolutely correct, thank you for figuring that out! I'll let you open the pull request for that change, and receive the glory.
But in the meantime, I realized that we can use python's unenforced access to private attributes to have a short-term workaround for this issue:
MissForest()
imputer.fit(X_train)
train_imputed = imputer.transform(X_train)
# reset the _missing_row attribute of the imputer object
imputer._missing_row = {}
test_imputed = imputer.transform(X_test)

worked for me without having to modify the library code.

Closes #39


HAWKULAR-744 : Reuse brush on chart when available

This, along with previous changes fixes the tooltip not showing unless
when selecting a range and also the sticky tooltip (HAWKULAR-744)

Merged #68


confusing error when not giving enough arguments to bump-formula-pr

Please follow the general troubleshooting steps first:

[x] Ran brew update and retried your prior step?
[x] Ran brew doctor, fixed as many issues as possible and retried your prior step?
[x] Confirmed this is problem with Homebrew/brew and not specific formulae? If it's a formulae-specific problem please file this issue at https://github.com/Homebrew/homebrew-core/issues/new

You can erase any parts of this template not applicable to your Issue.
Propose a feature:
$ brew bump-formula-pr jdnssec-tools
Error: no implicit conversion of nil into String
Please report this bug:
  http://docs.brew.sh/Troubleshooting.html
/usr/local/Homebrew/Library/Homebrew/download_strategy.rb:315:in `initialize'
/usr/local/Homebrew/Library/Homebrew/download_strategy.rb:315:in `new'
/usr/local/Homebrew/Library/Homebrew/download_strategy.rb:315:in `ext'
/usr/local/Homebrew/Library/Homebrew/download_strategy.rb:325:in `initialize'
/usr/local/Homebrew/Library/Homebrew/resource.rb:54:in `new'
/usr/local/Homebrew/Library/Homebrew/resource.rb:54:in `downloader'
/usr/local/Homebrew/Library/Homebrew/resource.rb:116:in `fetch'
/usr/local/Homebrew/Library/Homebrew/dev-cmd/bump-formula-pr.rb:138:in `bump_formula_pr'
/usr/local/Homebrew/Library/Homebrew/brew.rb:94:in `<main>'

I know my syntax is wrong (I'm missing arguments), but the output could be more helpful about that.

CC @ilovezfs for thoughts.

      elsif !hash_type
        odie "#{formula}: no tag/revision specified!"
 +    elsif !new_url
 +      odie "#{formula}: no url specified!"
      else

is probably good enough


Added BlockChain books

Added 2 books about blockchain technology.
What does this PR do?
Add Resources
For resources
Two books on blockchain.
Description
Added two books on Block
Why is this valuable (or not)
Because no resources on BlockChain exist.
How do we know it's really free?
One is a pdf, the other is an official guide.
For book lists, is it a book?
N.A.
Checklist:

[x] Not a duplicate
[x] Included author(s) if appropriate
[x] Lists are in alphabetical order
[x] Needed indications added (PDF, access notes, under construction)

I will format this properly.


Issue 307: Fix reference error in slash command processing.

https://github.com/howdyai/botkit/issues/307
There is logging happening here which introduces a reference error on slack_botkit. Removed this logging code because it already happens in SlackBot.js.

Thanks!


Ignore frequent spans

Using a Directive in an EnvFilter didn't seem to do anything at all so I landed on this. This was very painful as reordering anything at all in the registry init lead to completely broken output.
It is still not perfect as it will ignore every span, even if it has an error. I am opening this for now since I am stumped on how to solve this large issue and I've been trying to solve it for a few hours to no avail. This means this change is a bit far in the other direction - even erroring send offers, ping, and send heartbeat spans will be ignored. This will still reduce the load and log the error, but the error will not be attached to any span anymore.
As part of the implementation, I added a new quiet: bool field to send_interval, which is currently only true for SendHeartbeat. An alternative is a new send_interval_quiet, but I thought that it could be good to force the caller to consider whether to be quiet or noisy. For most situation, noisy will be fine, I think.

is there any reason to still be verbose when sending the heartbeats?
I like the explicit policy choice (but with an enum that states the policy).
zooming out: I'm not sure whether we should disable spans on offers and pings at this stage (maybe when we stabilize libp2p? or allow increasing that verbosity from the command line, so we don't need to redeploy a version if there are some issues lurking there and we'd like to debug it further? 🤔

Addressed the two concerns you had @klochowicz

Force pushed now - should work

bors r+


Hardcoded path to /bin/nvidia-smi

Currently nvidia-system-monitor doesn't work on Debian (or Ubuntu, etc.) because the nvidia-smi binary path is hardcoded to /bin/nvidia-smi, while on Debian/Ubuntu it's in /usr/bin/nvidia-smi. As a result, I get this error:
$ nvidia-system-monitor
Connecting to nvidia-smi...
nvidia-smi not found. Are you have NVIDIA drivers?
NVSMApp 0x55f24e528c00 deleted

I have added /usr/bin/nvidia-smi to search path. Please, test the latest master version

Somehow I still get the same error with the latest git.

I replaced fopen() with system(). Please, test the latest master version

It works now, thanks!


Give ServiceEntry resources exportTo . annotation

Otherwise by default they are assumed by Istio to have been exported globally?

I ran tests and it failed with the following error message.
• Failure [900.027 seconds]
PrincipalCloudFormationController
/workspace/controllers/principal_cloudformation_test.go:17
  Should create and destroy an IAM role [It]
  /workspace/controllers/principal_cloudformation_test.go:22

  Timed out after 900.002s.
  Expected
      <int>: 1
  to equal
      <int>: 0

  /workspace/controllers/principal_cloudformation_test.go:105

That probably indicates we need to rebase, I think we fixed those errors.


Export to tensor-flow model

Instead of exporting to an mlmodel, I would like to export to a tensor flow model so I can use the model on Android and other platforms. Is this possible?

This is not supported by Turi Create.
You could try exporting your model to Core ML, then converting it to ONNX using onnxmltools and finally converting the ONNX model to tensor flow using onnx-tf.

@TobyRoseman thanks for the suggestion.


Making this extension compatible with modeline from vim

Hi @deitry,
Would you be willing to rename the language id to make this extension compatible with modeline from vim supported by this other extension
Vim currently recognize this type of files as dosini, so by renaming the language id properties-alternative with dosini and having a modeline in your properties file as follow, vscode should be able to associate automatically the file type.
# ex: ft=dosini:

I just pushed updated version with language id alias.
https://code.visualstudio.com/docs/languages/identifiers
Could you please check if this helps?

Apparently this was not enough. I'll switch language id, that's not a problem

Should be fine since 0.0.3


Question

Does this package allow for Fan-Out i.e: multiple consumers can read the same stream message?
Thanks

Yes its a typical stream provider. It has all the same features.

Also have a look here: https://github.com/Surveily/Orleans.Streaming.Grains/tree/main/src/Orleans.Streaming.Grains.Tests/Streams/Grains


reaxt not found in Phoenix

Adding reaxt to a Phoenix app and starting the server results in this error:
** (File.Error) could not read file web/node_modules/reaxt/package.json: no such file or directory
    (elixir) lib/file.ex:245: File.read!/1
    lib/tasks.ex:44: Mix.Tasks.Compile.ReaxtWebpack.run/1
    (elixir) lib/enum.ex:977: anonymous fn/3 in Enum.map/2
    (elixir) lib/enum.ex:1261: Enum."-reduce/3-lists^foldl/2-0-"/3
    (elixir) lib/enum.ex:977: Enum.map/2
    (mix) lib/mix/tasks/compile.all.ex:15: Mix.Tasks.Compile.All.run/1
    (mix) lib/mix/tasks/compile.ex:64: Mix.Tasks.Compile.run/1
    (mix) lib/mix/task.ex:251: Mix.Task.run_task/3

I poked around and found the commonjs_reaxt/ directory.
I was able to clear up this error message by this action:
cd path-to-commonjs_reaxt/
npm link
cd path-to-phoenix-app/web/ # where webpack.conf.js lives
npm link reaxt

Now node seems to be able to find reaxt and things move along.
This works for development, but isn't ideal. Any hunches as to what might be going wrong?

Hello collin, actually the compiler :reaxt_webpack should do the "npm install /path/to/reaxt/priv/commonjs_reaxt" if the js reaxt lib is not found and that the installed version differs from the reaxt version.
See
https://github.com/awetzel/reaxt/blob/master/lib/tasks.ex#L39-L53
Maybe you forgot to add the compiler ?

to add the compiler, add compilers: [:reaxt_webpack | Mix.compilers] to your mix.exs


Use swagger (Swashbuckle ) with ASP.NET core 2.2 and IIS virtual applications

Hi,
I'm currently testing the integration between ASP.NET core and swagger by using the Swashbuckle.AspNetCore nuget package.
My current test environment is the following:

ASP.NET core 2.2 web application with one api controller
the development machine is a windows 10 machine
the installed version of IIS is IIS 10

I'm testing my application by using a virtual application under the IIS default web site. The virtual application has an alias of test-swagger. By following the documentation I configured the swagger middleware in the following manner (notice the . (dot) character at the beginning of the swagger endpoint):
app.UseSwaggerUI(c =>
      {
        c.SwaggerEndpoint("./swagger/v1/swagger.json", "My API V1");
      });

By configuring swagger this way the swagger UI does not work as expected, a pop-up is showed when the path /swagger is called and the message showed states that the swagger UI is not able to find the swagger.json file.
By following this blog I finally solved my issue. Basically, the only difference is that two . (dot) characters are required in the swagger endpoint configuration (while the official docs reports that just one dot character must be used in the swagger endpoint as a prefix).
To summarize, the working configuration for me is the following (notice the two . (dot) characters at the beginning of the swagger endpoint):
app.UseSwaggerUI(c =>
      {
        c.SwaggerEndpoint("../swagger/v1/swagger.json", "My API V1");
      });

Is there an error in the official docs ? Am I missing something ?
Thanks for helping !

It sounds like Swashbuckle isn't properly using the HttpRequest.PathBase value. Can you provide a sample project that reproduces the problem? What is the full absolute URL you expect for the swagger endpoint?

It sounds like Swashbuckle isn't properly using the HttpRequest.PathBase value. Can you provide a sample project that reproduces the problem? What is the full absolute URL you expect for the swagger endpoint?

Hi and apologize for the late reply,
the full absolute URL to which I would like to respond is something like http://localhost/test-swagger/swagger/index.html where test-swagger is the alias assigned to the application registered under the existing web site in IIS

It sounds like the issue is in Swashbuckle then, or possibly the doc. I'm surprised that you need ./ at all and isn't handling the path base properly itself.

It sounds like the issue is in Swashbuckle then, or possibly the doc. I'm surprised that you need ./ at all and isn't handling the path base properly itself.

I totally agree with you. In my scenario, the only way to get it working properly is by using the prefix ../

Have you filed an issue on the Swashbuckle repository?

Have you filed an issue on the Swashbuckle repository?

I'll try to open an issue directly to Swashbuckle, by referencing this issue. I'll write here in case of news

Even after trying with the double dot prefix, when I pull up http://localhost/myAspNetCore22App/swagger/index.html, I get a blank page. The browser console reports 404 on swagger-ui.css, swagger-ui-bundle.js, swagger-ui-standalone-preset.js.


Remove default Rx interface

Short description
Remove Rx as the default interface and offer it as an extension (subspec).
Setups

[x] Remove it from networking.
[x] Remove persistence.

GIF

Ready @sergigracia waiting for CI to pass.

          1 Error

  :no_entry_sign:
  Please include a CHANGELOG entry.

When the intern asks me if design patterns were made by designers

  Generated by :no_entry_sign: danger


fix(ref): checks if ref exists and also it passes the node dom

closes #30

:tada: This PR is included in version 1.2.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Update an account in the service

Read should accept an account_id and use Account.find() to find the account.
It should return an HTTP_404_NOT_FOUND if the account cannot be found.
If the account is found, it should call the serialize() method on the account instance and return a Python dictionary with a return code of HTTP_200_OK.

Update should accept an account_id and use Account.find() to find the account.
It should return an HTTP_404_NOT_FOUND if the account cannot be found.
If the account is found, it should call the deserialize() method on the account instance passing in request.get_json() and call the update() method to update the account in the database.
It should call the serialize() method on the account instance and return a Python dictionary with a return code of HTTP_200_OK.


Re-query speaker name when multiple speaker names returned during Group Chat speaker selection

Why are these changes needed?
During the speaker selection process (when in "auto" mode), the LLM returns the name of the next speaker. This is fairly reliable with OpenAI's models but with open-source/weight models they can sometimes have trouble returning, simply, the name of the next speaker. Often, they will return a sentence, paragraph, or even a large sequence of text. Furthermore, I have found that to get the correct next speaker name you often have to prompt these LLMs to provide an explanation in order to have a chain-of-thought that leads the LLM to the correct agent and the resulting response can often include the other agent names as part of the reasoning.
Currently, if there is more than one valid agent name it fails the speaker selection process by returning:
GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned: ...
This PR aims to provide a second-chance by prompting the LLM again, this time with a specific prompt text together with the returned text, asking the LLM to provide just the one agent name based on some rules. In my testing, I have found that this simpler step helps to overcome what would be a failing step and, depending on the model, this can drastically reduce the occurrence of the multiple agent names failure.
How does it work?

GroupChat has a new attribute, requery_on_multiple_speaker_names (bool) that enables a user to turn this feature on, default is off (False)
During the agent name matching step in speaker selection, GroupChat._finalize_speaker:

If a single agent name is detected in the response, it returns the agent (unchanged)
If multiple agent names are detected in the response and GroupChat.requery_on_multiple_speaker_names is True it will:

Compile a new chat message with a hard-coded prompt, including the response's text as context
Sends the message to the LLM (same LLM as the one used for the speaker selection)
If the new response has just one agent name, it is returned as the next agent (success!)
If the new response has no agent name or more than one, it fails with a similar message to the current error message

If no agent names are detected, it fails with the current error message (unchanged)

The prompt to select the name
I have put together a prompt that performs reasonably well in selecting a speaker name. However, I foresee that this prompt will be tweaked over time and, possibly, have the option to be overridden by the user.
I tried zero-shot prompts and few-shot prompts and, oddly, the zero-shot prompt worked better.
The prompt I have is (where {name} is the response with multiple agents names):
Your role is to identify the current or next speaker based on the provided context.
The valid speaker names are {[agent.name for agent in agents]}.
To determine the speaker use these prioritised rules:
1. If the context refers to themselves as a speaker e.g. "As the..." , choose that speaker's name
2. If it refers to the "next" speaker name, choose that name
3. Otherwise, choose the first provided speaker's name in the context

Respond with just the name of the speaker and do not provide a reason.
Context:
{name}

Testing
I tried the following multiple agent-name response texts, here is the result of my testing.
Tests are based on these 7 multiple agent-name responses (which I have seen through my testing of models):

#
Response
Expected Agent Name

1.
Product_Manager because they speak after the Chief_Marketing_Officer.
Product_Manager

2.
Thanks Chief_Marketing_Officer, as the Product_Manager my plan is to produce some amazing product ideas.
Product_Manager

3.
Product_Manager. Here are five ideas that I think will impress the Chief_Marketing_Officer and be great for a marketing strategy for the Digital_Marketer.
Product_Manager

4.
The next speaker, after the Chief_Marketing_Officer, is the Product_Manager.
Product_Manager

5.
As the Product_Manager I've decided that an infotainment system that links up with your phone will have the biggest impact on the marketplace. Digital_Marketer, over to you for an amazing strategy.
Product_Manager

6.
Thank you Digital_Marketer, the next speaker will be Chief_Marketing_Officer.
Chief_Marketing_Officer

7.
What a great team! Let's hear from the Chief_Marketing_Officer now that the Product_Manager has some ideas.
Chief_Marketing_Officer

Results

LLM
Correct out of 7

Open source/weight

llama2:13b-chat
ALL 7

mistral:7b-instruct-v0.2-q6_K
ALL 7

mixtralq4
ALL 7

mixtralq5
ALL 7

orca2:13b-q5_K_S
ALL 7

codellama:34b-instruct
6 / 7

gemma:2b-instruct
6 / 7

gemma:7b-instruct
6 / 7

neural-chat:7b-v3.3-q6_K
6 / 7

openhermes:7b-mistral-v2.5-q6_K
6 / 7

qwen:14b-chat-q6_K
6 / 7

codellama:34b-python
5 / 7

llama2:7b-chat-q6_K
5 / 7

yi:34b-chat-q4_K_M
5 / 7

deepseek-coder:6.7b-instruct-q6_K
4 / 7

solar:10.7b-instruct-v1-q5_K_M
4 / 7

phi
3 / 7

phind-codellama:34b-v2
3 / 7

codellama:7b-python
0 / 7

codellama:13b-python
0 / 7

nexusraven
0 / 7

OpenAI

gpt-3.5-turbo
6 of 7

gpt-4
6 of 7

gpt-4-turbo-preview
6 of 7

Mistral.AI

small
6 of 7

medium
6 of 7

large
ALL 7

I believe this capability will noticeably improve the reliability of GroupChat speaker selection
What is missing from this PR
Tests - As this PR is based on LLM responses, I need some guidance on what tests (if any) to create. Additionally, as this is focused more towards alt-models, can that be tested in any way. Finally, we would need to make sure that the responses are consistent.
Documentation - Along with the broader need for GroupChat documents (see #2243), I think this could be added to that PR as well as tips for Non-OpenAI models.
Thanks!
Related issue number
Based on shortcomings identified in #1746.
Checks

[ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
[ ] I've added tests (if relevant) corresponding to the changes introduced in this PR.
[X] I've made sure all auto checks have passed.

Codecov Report
Attention: Patch coverage is 16.66667% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 15.55%. Comparing base (4a44093) to head (c464f45).

Files
Patch %
Lines

autogen/agentchat/groupchat.py
16.66%
10 Missing :warning:

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #2304       +/-   ##
===========================================
- Coverage   38.14%   15.55%   -22.59%
===========================================
  Files          78       78
  Lines        7865     7874        +9
  Branches     1683     1824      +141
===========================================
- Hits         3000     1225     -1775
- Misses       4615     6620     +2005
+ Partials      250       29      -221

Flag
Coverage Δ

unittest
14.21% <16.66%> (?)

unittests
14.52% <16.66%> (-23.61%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks for the analysis! It looks like requerying aka give LLM a second chance makes the selection more robust.
I am wondering instead of further parametrize the "auto" method, if we can add another speaker_selection_method, such as "auto_with_retry", which would retry the selection until a single speaker is returned. Do you think this approach goes a step further to address the robustness issue?
Effectively, this will be a new built in speaker selection method. You can see how we can currently use user defined selection method like this: https://microsoft.github.io/autogen/docs/topics/groupchat/customized_speaker_selection

Thanks for the analysis! It looks like requerying aka give LLM a second chance makes the selection more robust.
I am wondering instead of further parametrizing the "auto" method, can we add another speaker_selection_method, such as "auto_with_retry", which retries the selection until a single speaker is returned. Do you think this approach goes a step further to address the robustness issue?
Effectively, this will be a new built in speaker selection method. You can see how we can currently use user defined selection method like this: https://microsoft.github.io/autogen/docs/topics/groupchat/customized_speaker_selection

Thanks for highlighting the possible approach, @ekzhu - I didn't think about it as a new method but it's definitely worth considering.
Regarding the retries until a single speaker is returned:
During my testing, I found that if it didn't succeed with the first re-query it was because it returned either:

text that wouldn't be useful for re-querying (it went off on a tangent)
blanks

It was rare for it to return text that still had agent names or useful context to then feed back for a re-query.
It is possible that we could introduce a second re-query prompt that is different to the first one (possibly simpler like "Select the most prominent speaker name from this list {[agent.name for agent in agents]}, Context: {name}". And, if this failed to identify a single agent, we could do the next suggestion (below) or return the failed response.
Alternatively, we could just take the first mentioned name from the original response (which seems to be, more often than not, the correct one) rather than throwing an error.
New method, auto_with_retry
In terms of adding a new method, I wasn't sure how much "auto" was already used for logic within the code and was wondering if adding a new, but similar, method would result in having to replicate changes for auto also for auto_with_retry whenever they came through.
In groupchat.py it doesn't look like it would be too much work to accommodate both auto and auto_with_retry. I'm happy to add as a method which would make it a more obvious approach for users.
Let me know if we do want to make this change and I'll update code.

During my testing, I found that if it didn't succeed with the first re-query it was because it returned either:
text that wouldn't be useful for re-querying (it went off on a tangent)
blanks
It was rare for it to return text that still had agent names or useful context to then feed back for a re-query.
It is possible that we could introduce a second re-query prompt that is different to the first one (possibly simpler like "Select the most prominent speaker name from this list {[agent.name for agent in agents]}, Context: {name}". And, if this failed to identify a single agent, we could do the next suggestion (below) or return the failed response.

Alternatively, we could just take the first mentioned name from the original response (which seems to be, more often than not, the correct one) rather than throwing an error.

With retries the select speaker function is becoming like a nested chat with agent and tool use. The key seems to be providing the right feedback to the group chat manager when the select next speaker operation fails to find a valid speaker. E.g., "The speaker selected was not in the valid spaker list", "No speaker was selected, please select a speaker..." etc.. The group chat manager maintains a separate temporary context within the select speaker function so all the retries are maintained in the same context, so that in each retry the group chat manager knows about the previous failures.

New method, auto_with_retry

I revise my previous suggestion. I think if we want to do the above, then this should be an upgrade to the auto method, in which the group chat manager performs the retries until a max_retries_for_selecting_speaker threshold is reached. No additional retry prompt is required -- retrying is an agent conversation.

With retries the select speaker function is becoming like a nested chat with agent and tool use. The key seems to be providing the right feedback to the group chat manager when the select next speaker operation fails to find a valid speaker. E.g., "The speaker selected was not in the valid speaker list", "No speaker was selected, please select a speaker..." etc.. The group chat manager maintains a separate temporary context within the select speaker function so all the retries are maintained in the same context, so that in each retry the group chat manager knows about the previous failures.

New method, auto_with_retry

I revise my previous suggestion. I think if we want to do the above, then this should be an upgrade to the auto method, in which the group chat manager performs the retries until a max_retries_for_selecting_speaker threshold is reached. No additional retry prompt is required -- retrying is an agent conversation.

Sorry, I don't quite understand the approach fully:

do you mean there's a new messages dictionary built up in the select speaker function and for each retry it adds the response and another message (e.g. "you didn't select a speaker, please select and return just one speaker") to the dictionary and then submits to the LLM up to the max retries. Would the starting prompt be the one I created or something else?

do you mean there's a new messages dictionary built up in the select speaker function and for each retry it adds the response and another message (e.g. "you didn't select a speaker, please select and return just one speaker") to the dictionary and then submits to the LLM up to the max retries. Would the starting prompt be the one I created or something else?

Right, this is what I meant. You can simply run a two-agent chat whenever you need to select a speaker:

First ConversableAgent with the select speaker system message and llm_config from the group chat manager
Second ConversableAgent with a registered reply function that validates the selected speaker from the first agent and returns a validation result based on the content of that message. If the result is valid, the registered reply function simply returns (None, True) to terminate the conversation.
The last message from the ChatResult of the conversation should contains the valid speaker, or, an error showing the speaker selection has failed.

Okay, thanks for the detailed summary, sounds logical to reuse agents to solve it.
I'll try that approach and see how it goes.

Just an update that I've implemented this and I'm testing it with multiple models to see how they behave with it.

Thanks for your patience on this PR - I've reworked the approach in line with @ekzhu's suggestion to use a nested chat for speaker selection.
I've implemented this approach by using a nested chat with two agents (one for speaker selection and one for checking). A custom function is used with the checking agent to validate the response from the LLM with the suggested next speaker's name. Just a reminder this is only for 'auto' speaker selection mode.
By default, when selecting the next speaker the agent chat will attempt at least one time and then, by default, two more times until it can identify a single agent as the next speaker.
A new attribute on GroupChat, max_retries_for_selecting_speaker, allows the user to nominate how many retries are allowed. As noted, default is 2.
For the retries, the validation function will append messages to the nested chat to guide the LLM by outlining the error (multiple names or no names) and providing prompt guidance on how to choose a speaker.
I've found this approach to be effective with non-OpenAI models which have typically required at least one retry. I have been able to get my debate challenge test to work its way through with this approach and some models, which was the catalyst for the changes.
I hope my code approach is right - please let me know if there are better ways of doing it.

To support the developer in understanding what is happening with speaker selection, I have added another attribute to GroupChat, select_speaker_auto_verbose, that provides visibility during execution on the select speaker responses. It defaults to False.
When it is true will output to the IOStream when:

The LLM returns a response that isn't just the agent name. E.g. "The next speaker is Product_Manager" will be shown but if it returned "Product_Manager" it would not.
The LLM returns a response that has multiple agent names or no agent names. E.g. "After the Chief_Marketing_Officer is the Product_Manager".
The successfully selected speaker.

The reason I added this is that it's difficult to see what's happening under the hood when there are issues and this can help dramatically to identify what the LLM is doing during speaker selection.
However, I understand this may not be the typical approach and if we need to change it please let me know. If there's an alternative solution that is equally helpful, then just let me know.
Examples of outputs (when select_speaker_auto_verbose=True):

and

There was one message that used to be output when select speaker failed:
logger.warning(
                f"GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned:\n{name}"
            )

I am not using logger.warning in my changes. I wasn't sure whether I should use it instead of IOStream or as well - any advice here is appreciated.

I've run these notebooks and they ran through the speaker selection okay:

agentchat_groupchat_RAG
agentchat_groupchat_research
agentchat_groupchat_vis
agentchat_groupchat
agentchat_society_of_mind

When running the existing tests, I noticed that in test_groupchat.py when it tests 'manual' mode, it ends up in the GroupChat's select_speaker function and I'm not sure if that's correct.
When I tried to create a manual chat it didn't end up in there and I'm not sure if it's just in the test scenario.
The reason I bring this up is I've had to add the following to the speaker_selection function (and a_speaker_selection) in order to cater for this and I don't think it's necessary:
        if self.speaker_selection_method == "manual":
            selector.update_system_message(self.select_speaker_msg(agents))
            final, name = selector.generate_oai_reply(messages)
            return self._finalize_speaker(last_speaker, final, name, agents)
        else:
            # auto speaker selection with 2-agent chat
            return self._auto_select_speaker(last_speaker, selector, messages, agents)

Once the changes are accepted I'll add some documentation.

When running the existing tests, I noticed that in test_groupchat.py when it tests 'manual' mode, it ends up in the GroupChat's select_speaker function and I'm not sure if that's correct.

I think in manual mode it still goes into that function but early terminates once the selected_agent is set.

When running the existing tests, I noticed that in test_groupchat.py when it tests 'manual' mode, it ends up in the GroupChat's select_speaker function and I'm not sure if that's correct.

I think in manual mode it still goes into that function but early terminates once the selected_agent is set.

Yep, I thought so as well but for the testing it doesn't seem to. I'll update for now as it seems like it may be just a test issue.

Thanks @ekzhu for all your suggestions. I've checked in updates.

I've made a number of changes but encountering a commit issue with mypy:
mypy.....................................................................Failed
- hook id: mypy
- duration: 7.04s
- exit code: 1

autogen/oai/openai_utils.py:695: error: "Beta" has no attribute "vector_stores"  [attr-defined]
autogen/oai/openai_utils.py:697: error: "Beta" has no attribute "vector_stores"  [attr-defined]
autogen/oai/openai_utils.py:702: error: "Beta" has no attribute "vector_stores"  [attr-defined]
Found 3 errors in 1 file (checked 33 source files)

It may be due to #2466, I've updated my openai through pip (in a Dev Container) and tried pre-commit clean but can't get passed the checks. Same issue mentioned here. Hopefully I'll get it sorted, if there are any tips, let me know :).

I've made a number of changes but encountering a commit issue with mypy:
mypy.....................................................................Failed
- hook id: mypy
- duration: 7.04s
- exit code: 1

autogen/oai/openai_utils.py:695: error: "Beta" has no attribute "vector_stores"  [attr-defined]
autogen/oai/openai_utils.py:697: error: "Beta" has no attribute "vector_stores"  [attr-defined]
autogen/oai/openai_utils.py:702: error: "Beta" has no attribute "vector_stores"  [attr-defined]
Found 3 errors in 1 file (checked 33 source files)

It may be due to #2466, I've updated my openai through pip (in a Dev Container) and tried pre-commit clean but can't get passed the checks. Same issue mentioned here. Hopefully I'll get it sorted, if there are any tips, let me know :).
UPDATE: I've updated the files through GitHub for now. My commits are in.

The solution is here: https://github.com/microsoft/autogen/pull/2455#discussion_r1581727843

@marklysze I hope you are unblocked now. There are a few comments from @ekzhu . Could you address them please?

@marklysze I hope you are unblocked now. There are a few comments from @ekzhu . Could you address them please?

Thanks @sonichi, appreciated... I'll review comments and resolve them

Okay, so the manual selection process is breaking when the user does not select an agent (response is blank or "q" in manual_select_speaker).
So that needs to be handled. As the on-screen direction is "enter nothing or 'q' to use auto selection", I will have it select the next agent if they don't select a valid one during the manual selection. I've committed a fix.

@ekzhu could you check whether your change request is addressed?


Planning revamp

What type of PR is this? (check all applicable)

[ ] Refactor
[x] Feature
[ ] Bug Fix
[ ] Optimisation

[required] Description
Added a new path planner implementation. Used by TU Berlin (FaSTTUBe) for global planning within a search window in front of the vehicle. This adds a ROS wrapper around the planner object:

Subscribes to cone map, car pose TF
Publishes path, boundaries, occupancy grid map of boundaries

[required] Documentation
Explanation on how this works:
https://papalotis-ft-fsd-path-planning-streamlit-main-63xmrt.streamlit.app/
[optional] Usability concerns or breaking changes?
N/a

LGTM pending carpark test

Carpark test pended


model.remove() can fail silently

Hi,
the method model.remove() seems to fail silently if the tag is not available. While this does not break anything, it can lead to strange behavior.
Additionally, this only works in root groups defined in self_groups since sub groups won't always have .remove()
This could be fixed with a check if the tag is in the list of tags. Otherwise, I will post an example for hierarchy soon which, among others features solves this issue.

I don't quite know what you mean ("tag is not available"), but since this part of the code base is currently in flux, I think it will sort itself out eventually.

... and now I cant follow you regarding

"tag is not available"

with my quick testing, you could call (group dosn‘t matter, old API)model.remove(‘functions‘, ‘interpolation‘) no matter if the node interpolation. If not available, it just does nothing. While this is not strictly an error, you would have a hard time tracing this on e.g. typos

I agree that it shouldn't fail silently. I just don't see how it does. Maybe I'm missing something. With current master (as model.remove() is new and not in the current release), I get this:
>>> import mph
>>> client = mph.start()
>>> model = client.load('tests/capacitor.mph')
>>> model.remove('functions', 'test_function')
>>> model.remove('functions', 'test_function')
No node named "test_function" in group "functions".
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "d:\home\coding\MPh\mph\model.py", line 411, in remove
    tag = self._node(group, node).tag()
  File "d:\home\coding\MPh\mph\model.py", line 116, in _node
    raise LookupError(error) from None
LookupError: No node named "test_function" in group "functions".

So it works the first time, because "test_function" exists in the demo model, and fails the second time, after that function has been removed. It also fails for any other name I pass for the second argument.

I double checked and can not reproduce. My bad. The reason I got confused is that this can happen when calling the java.remove() which some elements have - this does not check anything nor raises an error, e.g. model._group('physics')().remove('notthere') - however this is nothing we can catch.


add bio-photo.jpg

I want to add a picture to the .image DIRECTORY  (not into a blog post).
How do I upload the image from my PC-Desktop to the .Image directory pls ?

Not sure I understand the question. If you're wanting to push a file to your GitHub hosted page then add the file into the folder you want locally, commit the change with Git, the push it up to GitHub.