feat: declarative database management

This patch adds a prototype of the declarative database management feature by using a separate CRD named Database and exposing the relative status in its section.
This new CRD is only controlled by the instance manager of primary instance.
Database deletion is triggered by a finalizer which gets added by the reconciliation loop itself. By default the retention policy is set to "retain", preventing the controller from deleting the database.
Closes: #4679

This PR have been split from #5454 and contains only the code related to the declarative database management.

This PR has been split. I'm closing it in favor of #5329 and #5325.

/test limit=local ft=smoke

E2E test run on  EDB fork https://github.com/EnterpriseDB/cloudnative-pg/actions/runs/10814905112

Just two quick questions without having looked at the changes beyond the surface:

will it be possible to "adopt" existing databases into operator management?
the db-operator project, that I'm currently using for this task, has a mechanism to prevent accidental database deletion (if enabled they simple doesn't delete the underlying database), which seems useful in conjuction with CD tools. Is something like this planned?

Thanks for the work!

will it be possible to "adopt" existing databases into operator management?

If you create a Database object and the database already exists, the database is "adopted" and reconciled to the definition in the object.

the db-operator project, that I'm currently using for this task, has a mechanism to prevent accidental database deletion (if enabled they simple doesn't delete the underlying database), which seems useful in conjuction with CD tools. Is something like this planned?

The databaseReclaimPolicy property controls exactly that behavior.

/test d=push tl=4


Add redis+rq task queue to CLI docker compose file

Fixes LS-499
Untested until I'm able to build docker images again

Thanks! Will test once we can build


Interop with other Ingress Controllers

Is your feature request related to a problem? Please describe.
Pomerium and Pomerium Ingress Controller provide a feature rich ingress controller with built-in authentication, authorization and audit capabilities. Pomerium uses Envoy as its reverse proxy component and may be relied on for many demanding workflows and configuration options.
There could be situations when it is desirable to run Pomerium behind an existing Ingress Controller, i.e.

utilize an existing organization-wide traffic management, monitoring and alerting.
receive certain workloads Pomerium is not specialized to handle - i.e. API gateway.
perform request transformations, routing and scripting, configure rate limiting, etc.
use 3rd party plugins such as WAF, anti-DDOS, etc.

Historically, Pomerium supports forward-auth mode for NGINX and Traefik that allows you to authenticate and authorize requests using Pomerium. However, there are dozens of Ingress Controllers on the market, including those natively provided by the cloud vendors and integrating with each and every of them does not sound feasible.
Describe the solution you'd like
Suppose you have a typical Ingress resource that is talking to your app Service, and you would like to add a drop-in authentication policy that would only allow users from your domain to access the application.
Pomerium Ingress Controller would additionally scan Ingress resources with ingressClass other than pomerium that have ingress.pomerium.io annotations and and create a proxying Service that would implement authentication and authorization.
Thus if you kubectl apply the below spec, Pomerium Controller mutation webhook would

Create a pomerium-my-app Service, pointing to Pomerium Proxy.
Configure Pomerium Proxy Route to your original Service my-app, including applying all the rules set via pomerium-specific Ingress annotations.
mutate the incoming Ingress resource so that my-app would be replaced with pomerium-my-app Service.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minimal-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    ingress.pomerium.io/policy: |
      - allow:
          and:
            - domain:
                is: pomerium.com
spec:
  ingressClassName: nginx-example
  rules:
  - http:
      paths:
      - path: /testpath
        pathType: Prefix
        backend:
          service:
            name: my-app
            port:
              number: 80

Describe alternatives you've considered
Explain any additional use-cases
Additional context

I think we could skip (2) and have step (3) insert the pomerium-proxy service rather than creating a new one for each ingress.  I'm not sure there's any practical difference and would reduce the number of moving parts.

how will proxy know which host it's being called for then? the HOST would always be http://pomerium-proxy in that case.

I was thinking the external proxy would be responsible for ensuring the incoming host header is retained and we match directly on whatever is in the ingress rules.  However, in thinking about it, that might cause issues with routing if using TLS.
That said, we still might have an issue with TLS matching if we provision individual services too.  I'm not familiar enough with the listener config/behavior to be certain without trying it.  I believe the way TLS and routing are related in envoy has some sharp edges we'll need to be mindful of in either solution.

related: https://github.com/pomerium/ingress-controller/issues/132

Wildcard routing related:

https://github.com/pomerium/pomerium/issues/659
https://github.com/pomerium/pomerium/issues/1677

Considering the removal of forward auth at v0.21.0 - this seems to be an issue which will not be solved/addressed.
It is a shame, as Pomerium+fwdauth hit the sweet spot between doing one thing right and playing well with other services. Unfortunately will have to deploy an alternative as I require features ingress-nginx can provide, but Pomerium cannot.

I require features ingress-nginx can provide, but Pomerium cannot.

it would be of great help if you could help us identify the missing features.

In addition, working well with other proxies can help mitigate critical CVEs.


Use of quotation/referencing

Hi Jeppe
We don't quite know if we are allowed to ask this during the exam, but we are very much in doubt of whether to make a reference everytime we use the same setup/methods as you taught us in the problem sets and lectures. It is stated in the intro-part of the exam that, that indeed would be plagiarism, if it is text that is used from another source. Could you clarify the meaning of when to use quotations/references using code?

It is not necessary to reference the material in the course. I know it well, and can see where you deviate.


Add support for SSH ProxyCommand configuration

Terminus already supports the ProxyJump feature, however, in order for instance to connect to Cloudflare Access, the configuration of the SSH option ProxyCommand is required. Adding this feature would be very useful.

Thanks a lot for implementing this feature!


Fix Broken Email Links Behind Proxy

What kind of change does this PR introduce?
Fixes #1228
What is the current behavior?
supabase/auth behind a proxy with incorrect or internal-external X-Forwarded headers cause incorrect email account links
What is the new behavior?

Adds new environment variable API_FORCE_EXTERNAL_URL=[true | false] to force the API_EXTERNAL_URL to remain unchained. This new variable is turned off by default. Users reliant on the current (slightly broken) behavior have no changes.

Additional context
Total lines changed: 3

hello👋?  Anyone....

@hf Do you know why this PR hasn't been reviewed / merged yet? I believe this makes the changes as you proposed in the open issue by adding a new config variable. What can be done to to complete this?

@kangmingtay not sure who to ping on this. Are you perhaps able to assist on this PR? Any input / review would be great.
I believe the solution here is what hf proposed on the original issue: https://github.com/supabase/auth/issues/1228#issuecomment-1865912389

Any news on this?

Re: all interested, I'll continue to merge and keep up to date my feature repo based on main until it comes upstream to this repo.

Hi, anyone from the supabase team here? is it there a reason why this important fix is not merged? Thanks

@nickmitchko thanks for contributing to fix this issue - the team is quite tight on bandwidth right now so we'll need some time to review this
at first glance, i don't think we want to introduce a new config this way just to override the issue with the domain being set incorrectly by an external proxy service like kong
my initial thoughts on this would be to expose a config option specifically for the external url to be used in emails - similar to what we did recently for the SAML metadata (https://github.com/supabase/auth/pull/1599)

@kangmingtay Not the OP of this PR, but do want to point out that this PR aims to fix Supabase Auth's broken behavior introduced in version 2.67.1. When following Supabase's self-hosting documentation and code, the end result is simply broken, since version 2.67.1. As far as everyone in this issue (myself included) understands it, it's Supabase's own self-hosting setup / code that is broken, as that is what is being followed and used. If we're all mistaken about that, please spare a couple of minutes to let all of us in that issue know (or in the documentation) what you mean exactly by domains being set incorrectly* and/or how we should be reading the Supabase documentation / code / env files. Clearly, there are a lot of people running into this issue, and have been for 10 months.

Keeping in mind that in many setups, services should not use external URLs for other services

I use self-hosted version of gotrue
i have nginx that serves my react client on domain.com, and it also redirects all requests from domain.com/api/auth to gotrue
when I specify API_EXTERNAL_URL like this
API_EXTERNAL_URL=http://domain.com/api/auth/
only http://domain.com is used in the email, the route is trimmed. This essentially prevents me from using emails for auth handling.

@kangmingtay
Can we please get a resolution on this? @nickmitchko has offered an elegant 3 lines change solution.
Thanks

@kangmingtay Can we please get a resolution on this? @nickmitchko has offered an elegant 3 lines change solution.
Thanks
Edit: For anyone else struggling with this and that don't want to build an outdated image: MAILER_URLPATHS_RECOVERY="/auth/v1/verify" changing this in your .env to an absolute URL is a workaround.
Edit2: Another solution might be to go to https://studio.mywebsite.com/project/default/auth/templates (yes that's a hidden URL that doesnt appear on the left bar) and edit the text.

@y4my4my4m Thank you bro, this issue beat me so hard. Changing the MAILER_ in .env works

the kong issue still exists but turns out there are many hidden URLs that aren't shown on the UI and can be accessed by directly typing similar to the hosted cloud
idk why they would do something like this
ig to push people towards using their hosted cloud rather than self host
not good

MAILER_URLPATHS_CONFIRMATION="${API_EXTERNAL_URL}/auth/v1/verify"
MAILER_URLPATHS_INVITE="${API_EXTERNAL_URL}/auth/v1/verify"
MAILER_URLPATHS_RECOVERY="${API_EXTERNAL_URL}/auth/v1/verify"
MAILER_URLPATHS_EMAIL_CHANGE="${API_EXTERNAL_URL}/auth/v1/verify"
this works


Please Help: Changelog

People have requested a changelog.
But I don't want to write one by hand.
Please add a script that does this, and make it easy to run automatically.

something like https://github.com/rvagg/changelog-maker ?

Would you be open to having a commit message format for every commit?

"But I don't want to write one by hand."

Write or maintain?

If you want to have a script that does it automatically, you need have each commit in a specific format.
If you don't want to do that, what I've done in other projects is before each release look through the commits since the last release and determine which you should let users know about (interface changes, bug fixes, new features). You could use the following command to get the commits since the last release: git log <last-version-tag>..master.
Keeping a changelog is very useful for users who would otherwise need to look through the commits and changes since the last release. A maintainer should have much better knowledge of this and be able to determine the relevant changes much more easily.
I doubt backfilling the changelog to the initial version is all that useful anyway, so just starting one now and adding to it with each new release would be greatly appreciated.


Builder workplanes storage

The Builders are immediately storing what the user provides in the workplanes instance variable - the actual Plane objects should be stored there. Example:
    def __init__(
        self,
        *workplanes: Union[Face, Plane, Location],
        mode: Mode = Mode.ADD,
    ):
        self.workplanes = workplanes
        self.mode = mode
        self.sketch_local: Sketch = None
        self.pending_edges: ShapeList[Edge] = ShapeList()
        super().__init__(*workplanes, mode=mode)

Fixed with commit: f365ac2dd15f36809fc69db61001e29651972b4b


Addition and deletion of rows result in improper list

When the row is deleted, it just deletes the last row and not the row which is needed to be deleted. It's a critical bug!

Reported logs (10 Apr 2021)

When row was added, Fluttable realises the change and immediately adds a new list to the list of list, and send this as a callback to the main widget. So if you are doing a realtime computation on the basis of data sent by Fluttable, new row will be empty and your code could break unless you have custom conditions to check.
When row was deleted, Fluttable realises the changes and immediately removes the tapped row list from list of list and then send this as a callback to the main widget. This deletion does remove a row but text cells doesn't recognize this change and it looks as if the last row is deleted every time. This is a huge bug, since what you get as a data from callback is as expected and good, but what is shown by Fluttable in UI is completely off.

What is done?

Fluttable now has a separation of callbacks deprecating the previous callback as a result. It will still be available for some time but it would be highly recommended not to use it. Instead onTableEdited callback is added as a replacement. It's more or less a name change with a few functional updates but it plays a massive role in recognizing what this callback seems to do.
onRowAdded is another callback which will be added allowing you to save writing conditions as mentioned in Logs above. You can place custom logic to play with Fluttable. We recommend updating the editable data list and generating a new table key and setting the state. You can set your realtime result to null or maybe not display it at all since row added will be unfilled.
onRowDeleted is a major callback which solve the deletion log mentioned above. You can now simply sort it out similarly to what we recommended to do for onRowAdded. It should solve the problem.

Update 11 Apr 2021
Couple of changes:

callback will be deprecated in favour on onTableEdited
new callbacks: onRowAdded and onRowDeleted will be released

Since Fluttable will offer separate callbacks to events, you will need to handle list of data in the UI. This is done to generalize the overall behaviour and making row addition and deletion process easier and smoother and more controllable by custom business logic.
This new important changes will be released in the next version of Fluttable scheduled to release this week.


The terminal process terminated with exit code:1

Not able to open Integral terminal

To help diagnose this issue, I need a bit more information. Where in the course are you? What were you trying to do when you received this error? What have you tried so far to resolve it?

Closing due to lack of follow up.


Implement infra Transport for normal UDP

This change is

Git diff is quite confused; udp.go is completely new, while rudp.go is an unmodified move of old udp.go.

Review status: 0 of 4 files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

Reviewed 5 of 5 files at r1.
Review status: all files reviewed at latest revision, 2 unresolved discussions.

go/lib/infra/transport/udp.go, line 35 at r1 (raw file):

	conn net.PacketConn
	// While conn is safe for use from multiple goroutines, deadlines are
	// global so it is not safe to enforce two at the same time. Thus, to

Can you clarify this a bit? An example scenario would help. It looks like if a short deadline read comes in while a long deadline read is pending, the new read would just block unconditionally until after the existing read finishes, which seems a bit self-defeating.

go/lib/infra/transport/udp.go, line 67 at r1 (raw file):

		return nil, nil, err
	}
	b := bufpool.NewBytes()

What is the reason for using this function here? It means the reader needs to go look at another module, only to find out that this is effectively just calling make(common.RawBytes, common.MaxMTU). I started reading the module docs for bufpool before i realised it was actually irrelevant.

Comments from Reviewable

Review status: all files reviewed at latest revision, 2 unresolved discussions.

go/lib/infra/transport/udp.go, line 35 at r1 (raw file):
Previously, kormat (Stephen Shirley) wrote…
Can you clarify this a bit? An example scenario would help. It looks like if a short deadline read comes in while a long deadline read is pending, the new read would just block unconditionally until after the existing read finishes, which seems a bit self-defeating.

Damn, this means the same bug is present in RUDP. Hopefully fixed both now.
Also, neat discussion re: this issue of mixing old-school primitives and contexts: https://github.com/golang/go/issues/16620

go/lib/infra/transport/udp.go, line 67 at r1 (raw file):
Previously, kormat (Stephen Shirley) wrote…
What is the reason for using this function here? It means the reader needs to go look at another module, only to find out that this is effectively just calling make(common.RawBytes, common.MaxMTU). I started reading the module docs for bufpool before i realised it was actually irrelevant.

I'm thinking of having all packet buffer allocations going through the same library, to (1) avoid explicitly specifying the size everywhere, and (2) maybe amortize in the future (e.g., by actually grabbing the buffer from the bufpool itself).

Comments from Reviewable

Reviewed 2 of 2 files at r2, 1 of 1 files at r3.
Review status: all files reviewed at latest revision, 2 unresolved discussions.

go/lib/infra/transport/udp.go, line 67 at r1 (raw file):
Previously, scrye (Sergiu Costea) wrote…
I'm thinking of having all packet buffer allocations going through the same library, to (1) avoid explicitly specifying the size everywhere, and (2) maybe amortize in the future (e.g., by actually grabbing the buffer from the bufpool itself).

The size shouldn't be the same everywhere, depending on what you mean by "packet buffer". E.g. the BR uses 9k buffers, not 65k, and even that's only until we can support setting it based on the actual MTUs in use. It used to be 65k, but that wastes an obscene amount of memory. Using a buffer pool in the future is definitely something we can look at, but it would be cleaner to not add arbitrary references to that module until we're actually using it for this. At that point we can do a sweep of the codebase and transition use-cases to it, which we'll have to do anyway.

go/lib/infra/transport/udp.go, line 58 at r2 (raw file):

		return err
	}
	_, err := u.conn.WriteTo(b, address)

This doesn't verify that the entire message was sent.

Comments from Reviewable

Review status: 3 of 4 files reviewed at latest revision, 2 unresolved discussions.

go/lib/infra/transport/udp.go, line 67 at r1 (raw file):
Previously, kormat (Stephen Shirley) wrote…
The size shouldn't be the same everywhere, depending on what you mean by "packet buffer". E.g. the BR uses 9k buffers, not 65k, and even that's only until we can support setting it based on the actual MTUs in use. It used to be 65k, but that wastes an obscene amount of memory. Using a buffer pool in the future is definitely something we can look at, but it would be cleaner to not add arbitrary references to that module until we're actually using it for this. At that point we can do a sweep of the codebase and transition use-cases to it, which we'll have to do anyway.

Done.

go/lib/infra/transport/udp.go, line 58 at r2 (raw file):
Previously, kormat (Stephen Shirley) wrote…
This doesn't verify that the entire message was sent.

Done.

Comments from Reviewable

Reviewed 1 of 2 files at r4.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Why doesn't the cookbook start kafka service?

I'm working on a pull request to add support for Ubuntu.
The configure.rb recipe enables, but doesn't start the service.
service 'kafka' do
  supports start: true, stop: true, restart: true
  action [:enable]
end

Is there a reason why the start action isn't included by default?

On further comparison, it appears that I somehow downloaded a really old version of the cookbook off of github (0.2.0). Not sure how that happened. Closing


[Question] Safari support with trackInAppClick()

Hello.
I have a question. I'm using getInAppMessages with in-app method to show modals inside a webpage. Everything is fine but for Safari I see that trackInAppClick is not fired (looking through network). Based on the following link: https://github.com/Iterable/iterable-web-sdk#api this behavior is known.
My question: is there a fix or ideas how to make sure in-app click works on Safari as well? Or do I need to create my own workaround for that (like onClick listener, etc.)?
Appreciated!

Hi @Mapsai, onClick handler more than likely won't work as Safari doesn't support event handlers with our current i-frame security policy.


vite-plugin-dts incompatibility

This plugin seems to not be compatible with vite-plugin-dts.
I had expected this plugin to basically just automate the process of defining resolve aliases in my vite.config.ts.
And it does work fine for resolving imports during builds and typechecks.
However, the mentioned declaration-file-generating plugin seems to be incompatible.
Builds pass, but the tsconfig.json paths are not resolved to relative paths in the generated declaration files.
vite-plugin-dts works fine with resolve aliases, resulting in paths from my tsconfig.json being replaced by relative paths.
Is it possible/feasible to get both plugins working together?
Also see https://github.com/qmhc/vite-plugin-dts/issues/154.

Fixed by my new plugin vite-plugin-lib.

This is an issue with vite-plugin-dts, as it doesn't use Vite's path resolution pipeline. It merely looks at the resolve.alias option.
If someone else comes across this and vite-plugin-lib doesn't meet your needs, please open an issue in the vite-plugin-dts repo asking them to support Vite's path resolution.


[SPARK-26269][YARN][BRANCH-2.4]Yarnallocator should have same blacklist behaviour with yarn to maxmize use of cluster resource

What changes were proposed in this pull request?
As I mentioned in jira SPARK-26269, in order to maxmize the use of cluster resource,  this pr try to make YarnAllocator have the same blacklist behaviour with YARN.
How was this patch tested?
Added.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?


STL / Reset

Hi
Could you please post the STL file the case.
I noticed that the ESP32S3 does not reset sometimes on powerup, Removing C18 has completely solved the issue for me.
Thanks again from me and my Home Assistant :)

Hi @JmanNZ
Thank you for the feedback!
I've added the STL files for the enclosure to the repository. You can find them here:
hardware/KiCAD/esp32-energymeter-V2/enclosure
Tip: For best results, consider printing it with PETG filament.
I'm really glad you enjoyed the project! 😊


NamedTuple selection on the result of NamedTuple.Concat doesn't work

Compiler version
3.5.0-RC1-bin-20240516-c608177-NIGHTLY
Minimized code
import scala.language.experimental.namedTuples

object Test:
  type NT = NamedTuple.Concat[(hi: Int), (bla: String)]
  def foo(x: NT) =
    x.hi // error
    val y: (hi: Int, bla: String) = x
    y.hi // ok

Output
[error] ./try/nt.scala:6:5
[error] Found:    (x$proxy1 : (x : Test.NT) &
[error]   $proxy1.NamedTuple[
[error]     Tuple.Concat[
[error]       NamedTupleDecomposition.Names[
[error]         $proxy1.NamedTuple[Tuple1[("hi" : String)], Tuple1[Int]]],
[error]       NamedTupleDecomposition.Names[
[error]         $proxy1.NamedTuple[Tuple1[("bla" : String)], Tuple1[String]]]
[error]     ],
[error]     Tuple.Concat[
[error]       NamedTupleDecomposition.DropNames[
[error]         $proxy1.NamedTuple[Tuple1[("hi" : String)], Tuple1[Int]]],
[error]       NamedTupleDecomposition.DropNames[
[error]         $proxy1.NamedTuple[Tuple1[("bla" : String)], Tuple1[String]]]
[error]     ]
[error]   ]
[error] )
[error] Required: (Int, String)
[error]     x.hi // error
[error]     ^

Expectation
A valid selection.
More details
The selection desugars into something equivalent to:
NamedTuple.apply[("hi", "bla"), (Int, String)](x)(0)

With -Xprint:inlining I get:
        val $proxy1:

            NamedTuple.type{
              type AnyNamedTuple = Any;
                type NamedTuple[N <: Tuple,V <: Tuple] = V
            }

         =
          NamedTuple.$asInstanceOf[

              NamedTuple.type{
                type AnyNamedTuple = Any;
                  type NamedTuple[N <: Tuple,V <: Tuple] = V
              }

          ]
        val NamedTuple$_this:

            ($proxy1 :
              NamedTuple.type{
                type AnyNamedTuple = Any;
                  type NamedTuple[N <: Tuple,V <: Tuple] = V
              }
            )

         = $proxy1
        {
          val $scrutinee1: <error unspecified error> =
            NamedTuple$_this.toTuple[(("hi" : String), ("bla" : String)),
              (Int, String)](x$proxy1)
          val tup: <error unspecified error> = $scrutinee1
          tup.apply[(Int, String) & NonEmptyTuple](0).asInstanceOf[
            Tuple.Elem[(Int, String), 0.type]]

I'm not sure why we end up creating proxies for the top-level object NamedTuple here
The call to toTuple refers to an argument x$proxy1 but the pretty-printed tree doesn't contain any definition with that name.
Manually calling toTuple on my concatenated named tuple works.

I'm not sure why we end up creating proxies for the top-level object NamedTuple here

Ah I guess it's to deal with the combination of inline and opaque types

So it seems the inliner tries to deal with opaqueness by essentially taking the intersection of the original type and the type you get after replacing all the opaque types involved by type aliases. I think this sort of works when the opaque type is defined with an upper-bound but falls apart when it's defined with a lower-bound: (hi: Int, bla: String) & (Int, String) is equal to (Int, String) and not to (hi: Int, bla: String).
cf https://github.com/scala/scala3/blob/main/compiler/src/dotty/tools/dotc/inlines/Inliner.scala#L420-L430 introduced in https://github.com/scala/scala3/commit/1c2a8ded160fbf7609095f04dfb208796d4691bd

Actually this isn't specific to lower-bounds, we can minimize this to:
object Foo:
  opaque type Wrapper[T] = T
  inline def part[T](w: Wrapper[T]): T = w
  inline def part2[T](w: Wrapper[T]): T = part(w)
  type Rewrap[W] = Wrapper[Extra.Unwrap[W]]

object Extra:
  type Unwrap[W] = W match
    case Foo.Wrapper[t] => t
  type Rewrap[W] = Foo.Wrapper[Unwrap[W]]

object Test:
  type X = Extra.Rewrap[Foo.Wrapper[Int]]
  def foo(x: X) =
    Foo.part(x) // ok
    Foo.part2(x) // error

-- [E007] Type Mismatch Error: try/opaque-nt.scala:18:14 -----------------------
18 |    Foo.part2(x) // error
   |              ^
   |Found:    (w$proxy2 : (x : Test.X) & $proxy2.Wrapper[Extra.Unwrap[$proxy2.Wrapper[Int]]])
   |Required: Foo$_this.Wrapper[Int]
   |----------------------------------------------------------------------------
   | Explanation (enabled by `-explain`)
   |- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   |
   | Tree: w$proxy2
   | I tried to show that
   |   (w$proxy2 : (x : Test.X) & $proxy2.Wrapper[Extra.Unwrap[$proxy2.Wrapper[Int]]])
   | conforms to
   |   Foo$_this.Wrapper[Int]
   | but none of the attempts shown below succeeded:
   |
   |   ==> (w$proxy2 : (x : Test.X) & $proxy2.Wrapper[Extra.Unwrap[$proxy2.Wrapper[Int]]])  <:  Foo$_this.Wrapper[Int]
   |     ==> (w$proxy2 : (x : Test.X) & $proxy2.Wrapper[Extra.Unwrap[$proxy2.Wrapper[Int]]])  <:  Int
   |       ==> (x : Test.X) & $proxy2.Wrapper[Extra.Unwrap[$proxy2.Wrapper[Int]]]  <:  Int
   |         ==> (x : Test.X)  <:  Int
   |           ==> Test.X  <:  Int
   |             ==> Extra.Rewrap[Foo.Wrapper[Int]]  <:  Int
   |               ==> Foo.Wrapper[Extra.Unwrap[Foo.Wrapper[Int]]]  <:  Int
   |                 ==> Any  <:  Int  = false
   |         ==> $proxy2.Wrapper[Extra.Unwrap[$proxy2.Wrapper[Int]]]  <:  Int
   |           ==> Extra.Unwrap[$proxy2.Wrapper[Int]]  <:  Int
   |             ==> $proxy2.Wrapper[Int] match {   case Foo.Wrapper[t] => t } <: t  <:  Int
   |               ==> t  <:  Int

Match type reduction for  $proxy2.Wrapper[Int] match {   case Foo.Wrapper[t] => t } gets stuck in https://github.com/scala/scala3/blob/7d559ad06bdfe448450341c7fdf12cc0513456f3/compiler/src/dotty/tools/dotc/core/TypeComparer.scala#L3506-L3511
This is somewhat similar to https://github.com/scala/scala3/issues/20453 but trying to change match type reduction to get this to work seems a lot dicier /cc @sjrd .

Tentative fix: https://github.com/scala/scala3/pull/20457

Another workaround seems to be moving the apply method into NamedTupleDecomposition, and reexporting it from NamedTuple

I wonder whether we can come up with a general principle here:
In the scope of an opaque type (i.e. where the alias is known) we are not allowed to refer to a match type that contains the opaque type in one of its patterns. Can we maybe make this a warning or even an error? In that case, Jamie's fix which moved inline methods mentioning match types defined in NamedTuple outside of named tuple would address this issue. And we'd likely not have made the mistake in the first place.


Serverless manager should fail installation if the required dependency is missing

Description
Serverless manager should detects if Istio is installed and fail installation if it's missing and the internal docker registry is configured.
Manager should detect  presence or required Istio CRD (Virtual service) and succeed or fail depending on wether it is present and if its required for current serverless setup.
Additionally serverless charts include ServiceMonitor CR that is dependant on the kyma crds component.
Detect and fail if servicemonitors.monitoring.coreos.com CRD is not present
The status of serverless CR should reflect if required istio is missing.
Hint
The criteria to allow/fail installation will change in the future.
Enable the dependency check behind a feature flag.

The conditions are checked and the installation fails with an error log
1.66990002156049e+09	ERROR	error while checking for prerequisites	{"controller": "serverless", "controllerGroup": "operator.kyma-project.io", "controllerKind": "Serverless", "serverless": {"name":"default-kyma-serverless","namespace":"kcp-system"}, "namespace": "kcp-system", "name": "default-kyma-serverless", "reconcileID": "2e55bd95-e819-4aed-9747-c950f1fbf0ac", "error": "CustomResourceDefinition.apiextensions.k8s.io \"virtualservices.networking.istio.io\" not found"}

The Serverless CR is not yet reflecting the error because we need to first prove that we can set custom conditions in the Serverless CR status
https://github.com/kyma-project/serverless-manager/issues/16


Wrong sorting order with grouping for creation date

Description
I have a folder with a lot of files, and this software seems to be unable to sort and group the files based on the creation date correctly. See screenshots below.
It may or may not be related to an issue that makes this folder open showing older files, and then needing me to manually scroll to the top when the rest loads.
Steps To Reproduce
No response
Expected behavior
2022-02-20 should be considered to be after 2022-02-19, unlike it is now.
Files Version
Version: <IP_ADDRESS> OS Version: 10.0.22543.1000
Windows Version
Edition	Windows 11 Pro Insider Preview Version	Dev Installed on	2022-02-15 OS build	22543.1000 Experience	Windows Feature Experience Pack 1000.22543.1000.0
Relevant Assets/Logs

Specific to creation date, sorting by modified date appears ok.

Specific to creation date, sorting by modified date appears ok.

I experience the same issue with sorting and ordering by modified date:

Is this still an issue?

It seems to work in small folders, but I cannot negate the possibility that it may still happen in folders with a huge number of files, although I don't have such folders at the moment

@TheChilliPL can you try with the preview version? https://files.community/download/preview


Sluggish filtering

From: https://secure.helpscout.net/conversation/52771082/14257/

Sluggish filtering
Try picking a week or two in the calendar for 2 or 3 accounts. Sometimes it shows up, but sometimes you get an empty list for a while… then eventually the dones show up (usually but sometimes it seems stuck there forever so I have to refresh and try again)
I find the response time pretty slow given how simple the data is. I am not sure what is the backing store behind idonethis but even with hundreds of millions of entries (which probably it doesn't have) it shouldn't take this long to retrieve the data and render it.

oops this was meant for another project repo.


character  display abnormally,some disappear.

Hi,
I use freerdp to connect a linux server ,then linux server connect to a windows by xrdp.I found the character  display abnormally,I right-click on the desktop,some character can not display

Closing due to lack of activity.
Reopen/create a new issue if still relevant.


Migrate to new sortorder import path

See fvbommel/util#7.

Codecov Report

Merging #47 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #47   +/-   ##
=======================================
  Coverage   64.65%   64.65%
=======================================
  Files           5        5
  Lines         365      365
=======================================
  Hits          236      236
  Misses         90       90
  Partials       39       39

Impacted Files
Coverage Δ

rollup.go
78.82% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 50a7a8a...1d50604. Read the comment docs.

I am not sure what happened to travis. Can you push an empty commit to trigger CI?

I don't like empty commits, so I made a minor README fix instead :wink:.

Well .. still no dice. No idea what's happening. Travis status shows all green. Anyways, I ran tests locally and all seems good. I'll just force push this.
Thanks for the PR.


Arbitrary properties should not be overridden when a type is specified

When I use this class string: border-b border-[color:rgb(var(--color-gray-500-rgb)/50%))]
border-[color:rgb(var(--color-gray-500-rgb)/50%))] overrides border-b
I think it could be really useful to detect when a type is defined on an arbitrary property and override it accordingly (or don't when not).
In this case, border-[color:… should not override border-b

Hey @brandonmcconnell! 👋
Thanks for reporting this! I'm working on a fix.

Thanks! @dcastil

I discovered another bug I want to fix, so it will take a bit until I release a new version. If you need the fix right now, you can install<EMAIL_ADDRESS>already until a new prod release is out.


Feature sequential sign

Summary by CodeRabbit

New Features

Introduced a new e-signature service with configurations for creating and updating e-sign requests.
Added support for PostgreSQL and Flyway for database management.
Implemented new functionality for signing documents with customizable signature placeholders.
Enhanced the e-signature service with audit details tracking for better document management.

Bug Fixes

Updated method signatures for clarity and consistency in API responses.

Chores

Created a new Dockerfile for the e-sign service backend.
Updated various configuration files to enhance service integration and performance.

Tests

Added new mocks and updated test method signatures for improved clarity and structure.

1729245011829IsyeVQeXpX.pdf
sigend pdf on text

issue ticket https://github.com/pucardotorg/dristi/issues/1964


Getting broken image for predicted value at exp.show_in_notebook stage

Hello!
I am using tensorflow keras API for deep learning. the model has been built with Keras.
I do not know why my rendering section shows me not correctly on 'predicted value'. I thought I did a fine job. Can anyone think of why? Thank you.

Happened to me too when the predicted value was negative. When I translated it to a positive number, it showed correctly.

Happened to me too when the predicted value was negative. When I translated it to a positive number, it showed correctly.

Thank you for your comment. But as you see, my prediction is not negative. I do not know what causes this problem to me yet.

I have the same problem as well. Did anyone know the answer yet ?

Hi, I have this issue as well. This can be reproduced using the following code:
from sklearn.linear_model import LinearRegression
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from lime.lime_tabular import LimeTabularExplainer

boston = load_boston()
X = boston.data
y = boston.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)

linr = LinearRegression()
linr.fit(X_train, y_train)

explainer = LimeTabularExplainer(X_train, feature_names=list(boston.feature_names),
                                class_names=['MEDV'],
                                mode='regression',
                                random_state=0)
exp = explainer.explain_instance(X_test[25], linr.predict, num_features=5)
print(f"Predicted min: {exp.min_value}")
print(f"Predicted actual: {exp.predicted_value}")
print(f"Predicted max: {exp.max_value}")
exp.show_in_notebook(show_table=True)

Result:

After digging into the code, the problem lies in the bundle.js code line 27391 where Math.log10(min_value) evaluates to NaN when min_value is negative, causing the variable num_digits and corner_width to be NaN and bar_width to be negative. Maybe a fix is to take Math.abs as mentioned by @jkrabbenbos. I still don’t understand why the min predicted value is negative though.

Fixed in 5c7107e09e85c07a3417686673f2c81756c29280.
Thanks @JiaHong-Lee for identifying the cause of the bug and providing a reproducible example.


Changing fonts in Style Settings

Does not work at my end, even after reloading the vault.

Never mind.


Expo Build for PRs

What this does
Sometimes you need/want to do a full build test for a PR. Creating a Github Action to do that when manually built.
Checklist

[x] Initial version guessing at implementation
[x] Add support for choosing IOS and/or Android
[ ] Test

How to test
Add user steps to achieve desired functionality for this feature.

🧟 Resurrecting this PR. I've been doing this frequently on a recent project. You've also made a fix I was about to push related to #332
If you don't mind I'll take this over and try to finish it asap.


Passing null to stored procedure

I generate a stored procedure that is passed to request.query, for example
request.query("exec mysp 123,'this parameter can be null'");
sql server would require exec mysp 123,NULL
Is there a way in mssql to generate a null I can pass in using the 'exec ' method?

Create the parameter, and assign it a null value.


encryption: remove useless EncryptionKeyManager trait

What is changed and how it works?

Issue Number: Ref #15986
What's Changed:

remove useless EncryptionKeyManager trait

Check List
Tests

No code

Release note
None

/cc @overvenus Do you have any concerns?

/merge

/merge

/merge

/merge

/merge

/merge

/merge


Migrate ReactInterceptingViewGroup to kotlin

Summary:
Migrate ReactInterceptingViewGroup to kotlin
changelog: [internal] internal
Differential Revision: D51228374

Closing as abandoned internally


root endpoint request can crash the node if no RPCS are set

If there is no RPCS we send the following error:
error: HTTP:   Supported networks not defined
But we still try to send more info after that, including RPC related

closed on https://github.com/oceanprotocol/ocean-node/pull/575


Using fastlane for Swift Framework

I was just wondering can we use fastlane for a swift framework/lib without setting up an app identifier etc?
In this scenario the above is not relevant.
Thanks

Yeah, this might work, however you won't be able to use fastlane init, but instead have to manually build your Fastfile 👍

Ah thank you for the quick reply, I have tried this but unfortunately when running Scan it doesn’t register the fastfile existence. Does Scan make any other checks for files before running? Il take a look at the source code in the morning :) hopefully I can get it working Fastlane is awesome!

I'm using Fastlane to run native iOS and Android unit tests in an SDK: https://github.com/BranchMetrics/react-native-branch-deep-linking/blob/master/fastlane/Fastfile#L14. This is not Swift, but this might help you.

@jdee thanks for the info :) .. not sure why it was complaining for me then, will give it another go 👍

Thanks all I can confirm this is working as expected.


Handle consumder dir not configured, and cleanup old consuming mmap files if any

If the consumer directory is not configured, when we create the consumer dir in some
unknown place (depending on the path env of the application). Changed it to be under the datadir.
Also, it is possible that segments may be re-assigned after a server restart, causing old
mmaped segments to remain (if the last shutdown was a hard kill of the process).
It is best to clean out all mmap files under the consumer directory when the table data manager
is inited.

Codecov Report

Merging #2048 into master will decrease coverage by 0.33%.
The diff coverage is 46.66%.

@@            Coverage Diff             @@
##           master    #2048      +/-   ##
==========================================
- Coverage   69.68%   69.35%   -0.34%
==========================================
  Files         830      830
  Lines       40102    40386     +284
  Branches     5405     5483      +78
==========================================
+ Hits        27947    28010      +63
- Misses      10289    10509     +220
- Partials     1866     1867       +1

Impacted Files
Coverage Δ

...ata/manager/realtime/RealtimeTableDataManager.java
46.34% <46.66%> (+0.88%)
:arrow_up:

...in/pinot/transport/common/AsyncResponseFuture.java
57.36% <0%> (-23.36%)
:arrow_down:

...ot/broker/requesthandler/BrokerRequestHandler.java
65.94% <0%> (-22.06%)
:arrow_down:

...not/transport/scattergather/ScatterGatherImpl.java
41.74% <0%> (-16.37%)
:arrow_down:

...inot/transport/netty/NettyTCPClientConnection.java
69.71% <0%> (-16.32%)
:arrow_down:

...ore/query/scheduler/resources/ResourceManager.java
81.81% <0%> (-13.64%)
:arrow_down:

...in/pinot/core/query/reduce/NotEqualComparison.java
55.55% <0%> (-11.12%)
:arrow_down:

...mpl/dictionary/DoubleOffHeapMutableDictionary.java
70.17% <0%> (-8.78%)
:arrow_down:

...e/impl/dictionary/LongOnHeapMutableDictionary.java
80.85% <0%> (-6.39%)
:arrow_down:

...n/pinot/transport/netty/NettyClientConnection.java
68.42% <0%> (-5.87%)
:arrow_down:

... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 02aef39...4d9a0e8. Read the comment docs.


issue with px-datetime-picker

i am using px-datetime-picker component

//setting a date to the datetime picker component using dateTime attribute
var displayDate= '2017-03-01';
document.getElementById('dateOfInspectionId').dateTime = displayDate;
Now after setting the date and when i read the same value again
console.log(document.getElementById('dateOfInspectionId').dateTime);
2017-02-01
I am seeing a one month difference in the output.
could you please tell me the way to fix this

As I said on the forums, I'm not able to replicate this. Per the issue template (which you've ignored), can you provide us a CodePen or JSFiddle of this happening in the standalone component?

By any chance are you using the date-format property of YYYY-DD-MM and it is appearing off by one day instead of one month? We have seen that issue before, the root cause being the difference between UTC and whatever time zone you're located in.

we are getting a difference of one day to the date that we have set . if i set the date as 14-02-2017 , i am getting 13-02-2017
is this something that can be fixed

@Menkhaus-ge - can you please explain the UTC / timezone discrepancy and how to get around it?

Per the docs, your datetime should be formatted as an ISO 8601 string. Your dateTime is not a properly formatted ISO 8601 string and is probably getting interpreted incorrectly as a result. This value is probably also different in different browsers too as each will interpret it differently.
To solve this, 1) provide a full ISO 8601 string: "2013-01-07T22:44:30.652Z", 2) if you want a specific timezone, provide a timeZone. See: https://www.predix-ui.com/#/components/px-datetime-picker/


Standard formatting for those who want to contribute

I just made my first pull request and then I realise that I had a different formatting because of my local settings, so it will  be really nice if we had a standard  formatting.
Prettier ??

You can use deno fmt to format code to align with the the rest of the repo.

Use deno fmt. The CI enforces this. We should probably add a CONTRIBUTING.md file...

perfect, I'll work on this

done on #457


Motherboard model

https://github.com/diva-eng/sega-nu/blob/main/01-hardware/motherboard.md
it should Advantech AIMB-582QG2-00A1E with custom BIOS firmware

Thanks for the info! Document has been updated


@see breaks output when there is more than 1 space and a @link following it

In version 0.5.2 when using @see, the output is a broken link when there is more than 1 space after the tag and it includes a @link tag. I would expect the @see tag to ignore the whitespace that immediately follows it.
I'm able to reproduce this on try.esdoc.org.
Below is a sample of the modified Source Code:
/**
 * this is BaseClass.
 */
export class BaseClass {
  /**
   * this is a method of BaseClass.
   * @returns {string} a greeting.
   */
  baseMethod(){
    return 'this is base method';
  }
}

/**
 * this is MyClass.
 */
export default class MyClass extends BaseClass {
  /**
   * creates a instance of MyClass.
   * @param {number} value - initial value.
   */
  constructor(value){
    /**
     * this is property of MyClass.
     * @type {number}
     * @private
     */
    this._property = value;

    /**
     * This is where `@see` with `@link` is broken if it has `1` or more spaces following `@see`.
     * @type {number}
     * @see method
     * @see  method
     * @see  {@link method}
     * @see  {@link MyClass}
     * @see  {@link MyClass._property}
     * @see  {@link MyClass#_property}
     * @see  {@link _property}
     * @see  {@link _property} for further information (free-form description).
     * @see   method
     * @see   {@link method}
     * @see   {@link MyClass}
     * @see   {@link MyClass._property}
     * @see   {@link MyClass#_property}
     * @see   {@link _property}
     * @see   {@link _property} for further information (free-form description).
     */
    this._linkBroken = value;

    /**
     * This is where `@see` with `@link` works if it has only 1 space following.
     * @type {number}
     * @see {@link method}
     * @see {@link MyClass}
     * @see {@link MyClass._property}
     * @see {@link MyClass#_property}
     * @see {@link _property}
     * @see {@link _property} for further information (free-form description).
     */
    this._linkWorks = value;

    /**
     * This is where `@see` without `@link` works if it has `1` or more spaces following `@see`.
     * @type {number}
     * @see https://google.com
     * @see  https://google.com
     * @see   https://google.com
     */
    this._noLinkWorks = value;
  }

  /**
   * this is method of MyClass.
   * @param {number} a - this is a 1st number value.
   * @param {number} b - this is a 2nd number value.
   * @returns {string} repeated Hello
   */
  method(a, b){
    return 'Hello'.repeat((a + b) * this._property);
  }
}

@psyrendust Thank you for this issue. I will fix in next version.


finish system requirements

closes #98
closes #94

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#114  👈
#113
#112
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @dyc3 and the rest of your teammates on  Graphite

Merge activity

May 12, 3:49 PM EDT: @dyc3 started a stack merge that includes this pull request via Graphite.

@smcdonou02 Yes, it's intentional. "Must" indicates an absolute. "Should" indicates that failing to meet the requirement doesn't render the entire project a failure.


WebUIServerLauncher not running : java.lang.ClassNotFoundException: org.magic.api.main.WebUIServerLauncher

Hi!
I just got the last release (2.26), but when I want to launch the WebUIServerLauncher (may it be with web-ui.bat or web-ui.sh), I got the following exception :
Error: Could not find or load main class org.magic.api.main.WebUIServerLauncher
Caused by: java.lang.ClassNotFoundException: org.magic.api.main.WebUIServerLauncher

hoppsy my bad... i forgot to change theme with new release.
here is new files to replace :
web-ui.bat.txt
web-ui.sh.txt


[OSPP] Command Line Tools

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
BanyanDB needs a command line tool to

Register/Unregister native resources using schema operations APIs
CRUD the data using relevant APIs
Monitor the health status of servers using observability APIs

This task depends on https://github.com/apache/skywalking/issues/9044
This is a task of Open Source Promotion Plan
Use case
No response
Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I'm really interested in this task, could you pls explain what does the “observability APIs” in the third purpose above refer to?

Oh, I see , that might be this https://github.com/apache/skywalking/issues/9048. :)

Would you consider to enhancing https://github.com/SkyAPM/go2sky SDK first? We should be able to have. go2sky + meter APIs first, then we could have trace(optional only) + meter (required) + logs (optional) collecting for BanyanDB natively.

Oh, I see , that might be this #9048. :)

Would you consider to enhancing SkyAPM/go2sky SDK first? We should be able to have. go2sky + meter APIs first, then we could have trace(optional only) + meter (required) + logs (optional) collecting for BanyanDB natively.

I believe (9048) depends on the go2sky OSPP task? Let's focus on other subtasks first then see how that task go.

Could we create one on go2sky repo? (Out of ASF though)

@arugal FYI :)

I believe (9048) depends on the go2sky OSPP task?
Could we create one on go2sky repo? (Out of ASF though)

Out of curiosity, if there is dependency between multiple OSPP projects, how/when do we expect the students of each project to finish the project? Every single student should have equal time to do the work without being stressed by the fact that there is another student's work depends on  his/hers

Out of curiosity, if there is dependency between multiple OSPP projects, how/when do we expect the students of each project to finish the project? Every single student should have equal time to do the work without being stressed by the fact that there is another student's work depends on his/hers

#9048 is not an OSPP or GSoC project. It's an independent intro-level task for normal contributors.


Fix bug in addScheme

The bug was causing the NamespaceShadowingOfVariables test case to fail but was also resulting in schemes from the root namespace of the environment to be added to other namespaces.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 82.27%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #233      +/-   ##
==========================================
+ Coverage   82.11%   82.27%   +0.16%
==========================================
  Files          25       25
  Lines       10561    10563       +2
  Branches     1515     1515
==========================================
+ Hits         8672     8691      +19
+ Misses       1889     1872      -17

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Upgrade to OSP 1.12

/lgtm
Do you know what min version of OCP is required for OSP 1.12?

Nevermind. I found it "Before upgrading to the Red Hat OpenShift Pipelines Operator 1.12, ensure that you have at least installed the OpenShift Container Platform 4.12.19 or 4.13.1 version on your cluster."

Nevermind. I found it "Before upgrading to the Red Hat OpenShift Pipelines Operator 1.12, ensure that you have at least installed the OpenShift Container Platform 4.12.19 or 4.13.1 version on your cluster."

good catch @enarha  .... and stage/prod are beyond 4.12.19, so we should be good there

PR to test in RHTAP: https://github.com/redhat-appstudio/infra-deployments/pull/2507

there may be a consistently occurring chains/verify-enterprise-contract e2e failure occurring with 1.12 ... see https://github.com/redhat-appstudio/infra-deployments/pull/2507#issuecomment-1747406372

/retest

/retest

/retest

https://github.com/redhat-appstudio/infra-deployments/pull/2507 is now green. Merging the change.


Formatting of AppLens

The suggestions are overwriting the view of recycle events, so it is not possible to check the complete events. Check the point 4 suggestion in the attachment. This is tested on Chrome.

thanks. this is a known bug. See #https://github.com/ShekharGupta1988/AppLensV2/issues/398

closing this one. tracked as #398


AWS <-> AWS

From a private conversation:
Would be nice if into could move files in S3 to e.g., Redshift
data = into('redshift://conn_info::table', 's3://my/s3/bucket/*.csv')

There are presumably a few things in AWS world.  I can imagine a separate connected component in the into graph to manage these similar to how we might handle HDFS (#29).
My current thoughts on this is that we resort to metaclassing as we did with Chunks and have types S3(CSV), S3(JSON), etc. as well as standard sqlalchemy.Table for things like redshift.
A good first step would be to make functions like discover(S3(CSV)) that perhaps downloaded a small sample of a file, then called discover locally.
I don't think any of this is hard from an into/blaze perspective.  I think that most of the work here is just becoming familiar with the AWS technology stack.

for reference:
https://github.com/binarydud/redshift_sqlalchemy

redshift can only take data from s3 or dynamo
http://docs.aws.amazon.com/redshift/latest/dg/t_Loading_data.html

that doesn't limit us necessarily, except that ultimately to get something into redshift you have to through one of those things first

If only we had some mechanism to chain a sequence of conversions through a complex network...

:)

I think that @TomAugspurger and @wrobstory may have some experience with the AWS stack.  This might interest them.

oh hi
So I've written a very small library to perform the Postgres -> Redshift data shuffle : https://github.com/wrobstory/pgshift
The pipeline looks like pg_dump -> Pandas DataFrame -> S3 -> Redshift, with some gzipping + chunking to take advantage of Redshift's parallel-ity to improve load/COPY times. So, that code is probably a good starting point for something like into('postgres://conn_info::table', 'redshift://conn_info::table'), into(DataFrame, 'redshift://conn_info::table'), and into('s3://my/s3/bucket/*.csv', 'redshift://conn_info::table')
For data already in S3, I have a tool that will reconcile S3 keys to those that have already been loaded in Redshift, which could also be useful.
Let me get my head around whats happening in this library- at first glance it looks like I should be able to wrap my code with your @convert decorator and get this done pretty quickly.

For wrapping one's head around into I've also been creating this tutorial which might be worth skimming.
https://github.com/mrocklin/blaze-tutorial

Not much to add to what Rob said.
I haven't looked closely at the into library, but if you have two routes to a target, is there any kind of cost calculation to choose the faster route? It's so much faster to create a table in redshift via s3 and a copy than a straight df.to_sql.

Yes, convert edges/functions can be annotated with a cost.  We use networkx to find minimum length paths.

Great. The only other thing I'd add is that pandas has an open issue to support writing DataFrames to s3 URIs (we support reading right now). I've been lazy and have just used Rob's pgshift, since I only ever write to s3 to go on to redshift. I can implement a DataFrame.to_csv with an s3 path if you think that'd be a useful edge for into.

@wrobstory how does your Postgres -> Pandas -> RedShift process compare to using insert statements through SQLAlchemy?  Presumably into would already be able to manage Postgres -> Python tuples -> RedShift.
I'm also excited by helping users interact with CSV or JSONLines data already on S3.  I would normally manage this by

Implementing discover on CSV files in S3 by downloading a small bit (can boto help with this?) and then calling discover on the local file.  This would give me a datashape
SQLAlchemy + a datashape can probably already an empty table in Redshift with the right types
Implementing a function to generate the right COPY command like this one for postgres

I'm not sure if this would work differently in AWS land though.

@TomAugspurger Rather than implement DataFrame -> S3(CSV) an alternative would be to implement CSV -> S3(CSV) and then use the existing DataFrame -> CSV route.

To be clear, when I mention the type S3(CSV) I'm using S3 as a sort of parametrized type.  This is the behavior that I have in mind
In [1]: from into import *
In [2]: run into/s3.py

In [3]: data = S3(CSV)('path/to/myfile.csv', delimiter=',')

In [4]: type(data)  # Parametrized type
Out[4]: S3(CSV)

In [5]: data.path  # Still a normal CSV file
Out[5]: 'path/to/myfile.csv'

Because into relies on the type of the data to choose which function we want to run, we need some way to differentiate between a local CSV file and one on S3 or one on HDFS, etc..  One approach is to make many subclasses CSV_S3, CSV_HDFS, JSON_S3, ... another approach is to construct these subclasses on the fly, as with this S3 function.
from toolz import memoize

class _S3(object):
    pass

@memoize
def S3(cls):
    """ Parametrized S3 type """
    return type('S3(%s)' % cls.__name__, (cls, _S3), {'subtype': cls})

@wrobstory out of curiosity why is gnureadline a dep of pgshift?

@mrocklin i see, i sort of converged to something similar after playing around with the API a bit, but I like yours a bit better.

@cpcloud I'm not sure how that slipped in there- I did a pip freeze to a requirements file, and it's probably not actually needed.
@mrocklin, the COPY should be much faster than a straight INSERT statement, but that's totally anecdotal; I haven't benchmarked one against the other. I do know that it's easy to get i/o bound on COPY statements from S3, so you definitely want to chunk + gzip if at all possible.
I think steps 1 & 2 of your process for generating a COPY statement are what's missing in the pgshift code- right now you have to manually build the table yourself. Using a datashape to do that would be really great.

Looks like with a little effort it is possible to get only a small part of the file using a Range request header:
import boto
from into import discover, CSV
import tempfile

conn = boto.connect_s3(anon=True)

def stream(uri):
    bucket, key = uri.split('/')
    resp = conn.make_request('GET', key=key,
                             bucket=bucket,
                             headers={'Content-Type': 'text/csv',
                                      'Range': 'bytes=0-3000'})
    with tempfile.NamedTemporaryFile(mode='w+b', delete=False) as f:
        raw = resp.read()
        f.write(raw[:raw.rfind('\n')])
        return CSV(f.name)

print(discover(stream('nyqpug/tips.csv')))

python s3stuff.py

gives
var * {
  total_bill: float64,
  tip: float64,
  sex: string,
  smoker: string,
  day: string,
  time: string,
  size: int64
  }

That's awesome.  Next thing to check might be, will SQLAlchemy + resource successfully create a table for us on RedShift with that datashape?

@quasiben might know if Continuum has an aws account that one can use.

Just a quick, non-scientific benchmark. I made a 1000 x 4 DataFrame of floats and compared df.to_sql, which IIRC uses SQLAlchemy INSERTs by row, with a monkey-patched df.to_redshift method that goes df -> GZipped BytesIO -> s3 -> redshift.
to_sql took 2.5 minutes
to_redshift took 16 seconds.
My method doesn't do any chunking right now (maybe I'll try that later), so you could get even more of a speedup by loading chunks in parallel.

How does Redshift know how to interpret the bytes?

you have to tell it in the COPY command

i haven't added this yet, but i was going to do that tonight along with chunked uploading using multiprocessing

gzip is easy to add, it's simply an option in the COPY command, multipart uploads are not difficult but there are some details to straighten out

In the binary case what happens to things that don't have obvious standard binary representations, like datetimes?  Is Redshift expecting a certain format?

I'm not sure I understand, we'd have to go through CSV first. We couldn't just pipe a gzipped numpy array to redshift

it's quite liberal in the formats that it acceppts

Ah, sorry, my impression was that Tom was piping raw bytes to an S3 bucket and that Redshift had a way to interpret these raw bytes.

you can do neat things:
In [36]: import gzip

In [37]: from cStringIO import StringIO

In [38]: sio = StringIO()

In [39]: with gzip.GzipFile(fileobj=sio, mode='wb') as f:
    df.to_csv(f)
    print(len(sio.getvalue()))
   ....:
10

Does odo have support for redshift UNLOAD to an s3 prefix? I'm not seeing much in the way of this in the source at a glance.


Unnatural Instinct + Might of the Meek interaction

Actually doesn't work in PoB (small passives grants normal value -- not boosted by Might of the Meek). I don't know if working in PoE but it should be from wording.

Yep, just noticed this as well. It works in PoE, but PoB doesnt take the Meek-Modified Values of the Minor Nodes into Account when granting their Boni through Unnatural Instinct.

Fixed in the next update.

@Openarl was there a regression?
I have v 1.4.121 and putting the jewel into the socket doesn't grant any passive bonuses.

This is not working once again...
Also I added militant faith for the 3 jewel scion combo...
While Unnatural instinct and militant faith seems to work... Adding might the meek doesent do anything.


there should only be one Asset struct

now there is
build.Asset
and
horizon.Asset

@jedmccaleb looks like build.Asset is used by all the services other than horizon and it also looks like it's being replaced by txnbuild.Asset. Did you want to take Asset out from protocols/horizon/base and put it in the top level of protocols and have every single service use it? cc @ire-and-curses

In txnbuild, I ended up implementing an Asset interface, and then simple structs for NativeAsset and CreditAsset. I'm not sure how much value I see in combining the types here.

@ire-and-curses is this still relevant? It was assigned to me a long time ago.


binding inside #each

template:
{{#each items bind=true}}

Item {{bind "index"}}: {{bind "name"}}

    {{#if "foo" bind=true}}
        Hello {{foo}}
    {{else}}
        Goodbye
    {{/if}}

{{/each}}

code:
var context = {
    items: [
        {name: "tom", foo: "a"},
        {name: "cat", foo: "b"}
    ]
};

var templateScript = $('#book').html();
var template = Handlebars.compile(templateScript);
var nodes = Handlebars.parseHTML(template(context))

$(document.body).prepend(nodes);

setTimeout(function() {

    // works
    context.items[0] = {name: "Alex", foo: ""};

    // don't works
    context.items[0].name = "Alex";
    context.items[0].foo = "";
}, 1000*5);

I think it is clear from the code, what is the problem.
if use other template:
{{#each items var="item" bind=true}}

Item {{bind "index"}}: {{bind "item.name"}}

    {{#if "item.foo" bind=true}}
        Hello {{item.foo}}
    {{else}}
        Goodbye
    {{/if}}

{{/each}}

then it works, but the error is visible in the console (both methods context changes):

observe.js:753 Exception caught during observer callback: TypeError: Cannot read property 'appendChild' of null

thanks for reporting this one, I was rarely using this case you've pointed out.. it is clearly a bug related to #each helper without an item alias ("var" attribute) defined. Please update to 0.1.9 and let me know if it works out now.

btw, I recommend you to always declare an alias variable for #each item so it won't get accidentally conflicted with root context properties since we don't support binding with "../" yet. (and IMO it's better to read/understand as well)

Thanks for fix. It works now:
context.items[0].name = "Alex";
context.items[0].foo = "";

But the error remained. An exception is thrown always, regardless of how context changed (inside #each).

observe.js:753 Exception caught during observer callback: TypeError: Cannot read property 'appendChild' of null

hmm, weird.. I'm working on it

May be browser? I am use Chrome 48.0.2564.116 m
I will try to correct himself. Maybe I find out something.

If add:
if (!nodes[index - 1].parentNode) continue;
at the beginning of cycle on handlebars.element(106 line). Than the error disappears))

the error was deeper than I thought.. check out 0.2.0, I fixed this issue and also added more tests to prevent future bugs related to #each block bindings

oh, you're probably tired of me)
here is how I see this error again:
I'm use my old template:
{{#each items var="item" bind=true}}

Item {{bind "index"}}: {{bind "item.name"}}

    {{#if "item.foo" bind=true}}
        Hello {{item.foo}}
    {{else}}
        Goodbye
    {{/if}}

{{/each}}

and default context:
var context = {
    items: [
        {name: "tom", foo: "a"},
        {name: "cat", foo: "b"}
    ]
};

and now it work without errors:
context.items[0] = {name: "Alex", foo: ""};
but this is still causes the error (the same)
context.items[0].name = "Alex";
context.items[0].foo = "";

Btw, if you remove the "{{#if}}", then there is no error.

I got this issue fixed now (as you can see: https://jsfiddle.net/mateusmaso/v6dvqwyz), check out the lastest version and thanks for reporting it again.

Great! It worked. Thank you.


Enhance the UI of footer for the website

The footer of the website is not well designed , the github logo is too far left , other links are missing  as well.
I would like to work on the footer section and make it better looking and responsive

Feel free to do a mock design before making any changes to the code so I can see the kind of improvements you'd like to make.

@EmmaDawsonDev I can work on this if no one is working?

I am working on this @mud1tx , will PR soon

I would like to suggest this design for the footer.

We will have 2 cols to group the links under the category. This will remain the same for small and large screen
Copyright statement and social media links would be centre aligned below the links cols. This will stay the same for small and large screen

This design is responsive for small to large screens.

wyt @EmmaDawsonDev

I really like how you’ve categorised them into 2 columns. I think this will work great on larger screens but it may need to become 1 column on mobile (the Contribute section could go under the About us section) so that it’s not too cramped.

I really like how you’ve categorised them into 2 columns. I think this will work great on larger screens but it may need to become 1 column on mobile (the Contribute section could go under the About Us section) so that it’s not too cramped.

If we change the font size for mobile to 12px. This is how 2-cols will look

I think it looks to cluttered and I'd rather not go below a minimum font size of 0.875rem. I would rather go with one column only on mobile.

@EmmaDawsonDev  how about this, Yes details should be changed.

@EmmaDawsonDev how about this, Yes details should be changed.

How does it look on mobile?

Not Designed yet.


API command listPublicIpAddresses lists IPs only with forvirtualnetwork=true

ISSUE TYPE

Bug Report
Improvement Request

COMPONENT NAME

API

CLOUDSTACK VERSION

Since long long time ago ... (10 years)

CONFIGURATION

Zone Network type: Advanced network with Security Groups enabled
Network isolation: VXLAN
Virtualization: KVM

SUMMARY

The API command listPublicIpAddresses does not list IP addresses for any network with Advanced Networking + Security Groups + VLAN / VXLAN.
Command example: list publicipaddresses networkid=a7f6b609-211f-4157-a016-62c5563f8085 listall=True
The API command lists IP addresses with a select such as:
SELECT user_ip_address.id, user_ip_address.account_id, user_ip_address.domain_id, user_ip_address.public_ip_address, user_ip_address.data_center_id, user_ip_address.source_nat, user_ip_address.allocated, user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat, user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address, user_ip_address.source_network_id, user_ip_address.network_id, user_ip_address.uuid, user_ip_address.physical_network_id, user_ip_address.is_system, user_ip_address.vpc_id, user_ip_address.dnat_vmip, user_ip_address.is_portable, user_ip_address.display, user_ip_address.rule_state, user_ip_address.forsystemvms, user_ip_address.removed, user_ip_address.created
    FROM user_ip_address
        INNER JOIN vlan ON user_ip_address.vlan_db_id=vlan.id
        INNER JOIN account ON user_ip_address.account_id=account.id
    WHERE user_ip_address.source_network_id = 216 AND user_ip_address.display=1 AND user_ip_address.forsystemvms=0 AND user_ip_address.allocated IS NOT NULL AND user_ip_address.removed IS NULL  AND  (vlan.vlan_type = 'VirtualNetwork'  AND vlan.removed IS NULL ) AND (account.type != 5 ) ORDER BY user_ip_address.public_ip_address DESC  LIMIT 0, 500

The problem here is that the WHERE conditional vlan.vlan_type = 'VirtualNetwork'  filters to VLANs of type  'VirtualNetwork'; however, all networks of Advanced Network + SG are deployed with VLAN of type  'DirectAttached' which leads to an empty result.
This is due to the following code at ManagementServerImpl.java:
       VlanType vlanType = null;
        if (forVirtualNetwork != null) {
            vlanType = forVirtualNetwork ? VlanType.VirtualNetwork : VlanType.DirectAttached;
        } else {
            vlanType = VlanType.VirtualNetwork;
        }

When adding the parameter forVirtualNetwork at the API Command then it lists all the allocated IPs as expected.
list publicipaddresses networkid=<network-uuid> listall=true forvirtualnetwork=true;

This also affects the UI (both legacy and primate):

DISCUSSION
Considering the context presented here I see this as a bug, however, I would appreciate any feedback.
Is this a bug, or simply the desired behavior? If it is not a bug, we should then enhance the UI and API documentation.
I still don't see the "why" of such behavior in the case of VLAN on Advanced networking, as we hold networks with public IPs but none of them are listed on UI.
STEPS TO REPRODUCE

 🐱 > list publicipaddresses networkid=<network-uuid> listall=true;
 🐱 > list publicipaddresses networkid=<network-uuid> listall=true forvirtualnetwork=true;
count = 7
   ...
   ...
   ...

EXPECTED RESULTS

 🐱 > list publicipaddresses networkid=<network-uuid> listall=true;
count = 7
   ...
   ...
   ...

@GabrielBrascher this should be fixed by #4675 and #4676

@weizhouapache PRs #4675 and #4676 are interesting. I will be reviewing them.
The idea of this issue was regarding the use of the parameter forvirtualnetwork. On advanced networking with SG, it is not possible to list IPs unless forvirtualnetwork=true.
My question/suggestion is related to make Admins able to list all IPs when listall=true, even if forvirtualnetwork is not provided.

@weizhouapache PRs #4675 and #4676 are interesting. I will be reviewing them.
The idea of this issue was regarding the use of the parameter forvirtualnetwork. On advanced networking with SG, it is not possible to list IPs unless forvirtualnetwork=true.
My question/suggestion is related to make Admins able to list all IPs when listall=true, even if forvirtualnetwork is not provided.

@GabrielBrascher I understand your concern. however it works like it since 10+ years ago... it is not a good idea to change it.
it will break backwards compatibility.

@GabrielBrascher @weizhouapache - what's the consensus, close this issue in favour of the two already raised PR that fix the issue without breaking backward compatibility?

I will close the issue for now then, the main concern is with the UI bug as well.
Maybe I can think of a way to improve the API description as well or propose changes that would not break compatibility.


Hiding original source

It would be nice to if i could hide/mask the original location/link and show only the new output

I don't think that will be very useful and will be required only in exceptional cases.


No computational changes. Only some structural changes to ensure consistency with other repositories.

Validation steps:
git clone https://github.com/cabouman/gmcluster.git
cd gmcluster
git checkout prerelease
cd dev_scripts
source clean_install_all.sh
cd ..
cd demo
python demo_1.py
python demo_2.py

I tested it, and it works.


ZeroDivisionError: float division by zero

Traceback (most recent call last):
File "train.py", line 73, in
weights = make_weights_for_balanced_classes(dataset_train.imgs, len(dataset_train.classes))
File "/home/zzh/桌面/face.evoLVe/face.evoLVe.PyTorch-master/util/utils.py", line 47, in make_weights_for_balanced_classes
weight_per_class[i] = N / float(count[i])
ZeroDivisionError: float division by zero

Traceback (most recent call last):
File "train.py", line 73, in
weights = make_weights_for_balanced_classes(dataset_train.imgs, len(dataset_train.classes))
File "/home/zzh/桌面/face.evoLVe/face.evoLVe.PyTorch-master/util/utils.py", line 47, in make_weights_for_balanced_classes
weight_per_class[i] = N / float(count[i])
ZeroDivisionError: float division by zero

Hi, It seems that there exist some problem with your training data, please follow the dataset organization as indicated in our repo. Please re-check your training data and related paths.


[CP] Update to clean up messaging for package:unified_analytics

issue_link
https://github.com/flutter/flutter/issues/124808
Commit Hash
51b23bc, ca3a900
Target
beta
pr_link
https://github.com/flutter/flutter/pull/125445
Impacted Users
All flutter users
Impact Description
With the new analytics package being introduced, we wanted to automatically opt out users that have already opted out in the past in the new system. This makes the necessary changes to flutter to allow the package to handle the legacy opt out. Originally, we incorrectly put that logic into flutter when it should have lived in the package.
Workaround
No workarounds at the moment.
Risk
low
Test Coverage
yes
Validation Steps
Events won't be sent for legacy opted out users. It can also be checked by looking at $HOME/.dart-tool/dart-flutter-telemetry.config

@christopherfujino could you take a look?

LGTM

Completed yesterday, closing

Reopening, will close once it’s landed in a release.


feat(toplevel): add vf help subcommand + improve documentation consistency/look

Add help subcommand for convenience/consistency
with other tools
Make the --description text of all functions consistent and
under 80 char wide so that the help display looks nice
Sort alphabetically the docs/usage.rst commands and make
references to 'currently-activated virtualenv' consistent

Address part of Issue #40 (most of it was already addressed :+1:  )

Looks awesome, thanks! I've added a few comments about minor things, but otherwise I'm pretty happy with this.

Thank you. Let me know what you think about the responses and I can update the request again. I think I would also like to do #29, but I am wondering about versioning: would we even have to bother with it?

Done. Don't worry about #29, I've actually been thinking about that one during the week and I'll probably do it myself today.
It's definitely going to be versioned; I'll just call the first version that contains a setup.py 1.0, then follow PEP 440 from then on.

I have added --help and -h options. This should be ready to be pulled in now.


migrate-groups: add auth-type to identity

Description
Add basic auth-type to identity for the groups migration script
FIXES: https://issues.redhat.com/browse/THEEDGE-3555
Type of change
What is it?

[X] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Documentation update
[ ] Tests update
[ ] Refactor

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[X] I have added tests that prove my fix is effective or that my feature works
[X] New and existing unit tests pass locally with my changes
[X] I run make pre-commit to check fmt/vet/lint/test-no-fdo

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (4b1dd1c) 62.47% compared to head (e655343) 62.44%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2316      +/-   ##
==========================================
- Coverage   62.47%   62.44%   -0.03%
==========================================
  Files          87       87
  Lines       12979    12980       +1
==========================================
- Hits         8108     8106       -2
- Misses       4206     4208       +2
- Partials      665      666       +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

/restest


Improve snackbar centering

Using pixels as default for centering an element is bad pratice.
Use dynamic values such as, 50% or 50vw
Using just left makes placement wrong. Use portals on body instead?

Dynamic value was already set; left: 50% and transform: translateX(-50%) but it's the iframe Storybook uses that makes it seem like the position is off.
We also have the prop leftAlignFrom which defaults at 1200px, so larger screens will get a left aligned Snackbar. Should we increase the default here?

Dynamic value was already set; left: 50% and transform: translateX(-50%) but it's the iframe Storybook uses that makes it seem like the position is off.
We also have the prop leftAlignFrom which defaults at 1200px, so larger screens will get a left aligned Snackbar. Should we increase the default here?

Maybe using vw unit for left positioning in css would work better then 🤔


docs(tutorial): creating observables docs fixing typo

new Subject() does something very different from Subject.create().
fixes #1975

Coverage remained the same at 97.062% when pulling e58a4ab7569dc1e40f2bfeeb3b44ca63e304df81 on blesh:fix_tutorial_basics into d7355c7938b3aae5bccbcbcce290d03477dddbfc on ReactiveX:master.

LGTM

Coverage remained the same at 97.065% when pulling 197e2ac8c9ed405053d9fbc0bb95378a331cd351 on blesh:fix_tutorial_basics into 1ba0358d623b0bc9ef170f0a8a8cb4613961e199 on ReactiveX:master.

Coverage remained the same at 97.065% when pulling c25d88c75d8d0236e29237cf1a726b4895130fcb on blesh:fix_tutorial_basics into 5448f873d60226e5a409c6bdfcab0f9e95cceb09 on ReactiveX:master.

LGTM. Rebased for ya. omg I freaking love the this new ability to modify PRs!!! <3


[Feature request] Chat with docs, i.e. Retrieval-based QA etc

I like the user interactive design of this project. Is it possible to combine retrieval based QA with the user-friendly interaction of this project?

Thanks! I currently have no plans to add a chat-based layer of functionality on top of Semantra.
My primary motivation for building Semantra was to see how useful of an interface you could make without using any chat completions on top of embeddings. I have experimented with semantic search applications that use chat LLMs, and I've found that no matter what you try there is a risk they will a) fabricate plausible information, b) produce truthful information but not guided by context that uses outside knowledge, or c) provide truly unexpected responses (e.g. via prompt injection). On top of that, they are slow, computationally expensive, and cost significantly more to use than embeddings.
I do want to understand more though: what do you feel that you get with a chatbot experience that you don't feel is being met via the current interface of Semantra that shows relevant search results and even highlights relevant phrases within each result?

@freedmand
I've experimenting semantic search areas with most of the new LLM models. Here are my observations that related to the issues you highlighted.

Hallucination / fabricate information not guided by context:

this can be solved by using the right model and prompt

some of the models will always answer regardless of whether the question related to the context or not, they just ignored the context.

some models you have to prompt it in the right way. e.g:
`
Instruction:

lorem ipsum ....

Query:
`
In this prompt, the model will just answer without context
`
Instruction:
[Context] lorem ipsum .... [/Context]
Query:
`
In this prompt, the exact same model will say no if there are no answer in the context

Much better way that I found is we do mini CoT (chain of thought) two stages questioning.
First, we asked the model if it's possible to answer the question if we give the following context - just yes or no answer
If yes, then answer the question, if no, terminate the request. This gave lower hallucination result.

Regarding slow, and expensive computation:
Bigger parameters doesn't always better in term of semantic search (answer within context). There are <7b models that can correctly answer QA compared to 40B model. My observation is its more on the training data rather than the size. Answering context requires following instruction more than chatting.
We can solve this by deploying small local LLM.

Regarding : what do you feel that you get with a chatbot experience that you don't feel is being met via the current interface of Semantra that shows relevant search results and even highlights relevant phrases within each result?

Initially, I also thought a good search is enough without having any bot to answer the question. But when I use multiple documents, it quickly become confusing for the user.
For example, if I'm indexing a local law pdf, and then I asked, "What is the maximum jailtime for stealing a car?"
there are many paragraphs that returned by the search results and the user need to read one by one and understand the context of each paragraph. Sure, eventually the user will get the answer, but it will take a long time.
But if we have a bot that first answer the question, the user just need to verify if the answer is correct from the highlighted answer. Which is much faster and efficient way to resolve any questions. The digesting and understanding of all the context now are handled by the bot and human just need to validate the answer.

By the way semantra is designed, I think it should be possible to build on top of its api? Then this project could focus on providing better research results across a wide range of documents, and a separate project could build on top of it to provide the chat interface.
@hazxone regarding the case with legal documents, and just potentially numerous and confusing results for the user, I feel like the right step forward is to build/fine tune better embedding models that can deal with this kind of queries better. My tests so far is that whatever that might confuse the user might confuse the LLM even more. Our team got way better results for legal documents after putting each clause in association with each section and chapter, and then embedding each clause and those metadata one by one instead of relying on arbitrary length cutoffs (e.g., Chapter 2. Natural Person - Section 1. Civil Rights - Clause X. Loren ipsum…)

I think that's exactly right that with a stable API adding any integrations including chatbots would be very easy. With v0.2 (which is under development but may take some time) I want Semantra to be easy to use as an application, backend API server, and even an importable Python library with a simple interface.
Interesting to hear the use case of splitting text. I'm also looking into being able to define plug-ins to cover a variety of splitting cases. Embeddings are often sensitive to the length of the input, meaning that sometimes short queries match short texts well so it's not always good to have tons of chunks at different sizes (arbitrary equivalent-sized chunks thus works well as a good default). But having a way to delineate sub-sections within documents that should always be cut-off and never part of overlapping chunks would certainly be useful to improve results quality.

Thank you for the answer. Understand and I agree that chatbot should be separated from the main program.
Just a suggestion, it would be nice to have one small button in the main UI for curious user, so they can trigger the API to see what the AI answer is, without having to launch another separate script/program.
@yych42 Great ideas on separating the document by logical section. Did you separate it manually? because that would be difficult to automate/find the right pattern to separate for general documents.
And also (idea for v0.2) apart from documents, I think search semantically into a bookmarks/webpage also quite useful.
Let say the user index few medium posts, reddit threads, and in the next few months they would like to find back some important paragraph from all their bookmarks. As of right now, we can only search the page title or search again in google (which most likely it gave totally different updated result).


Alternate implementation of InMemoryTrackerStore

So there are a few Python3 isms in this snippet (that could be designed around), but as with quite a bit in rasa_core you can actually achieve more by getting rid of (what I assume are) odds and ends that have piled up.
I needed to write my own implementation of a TrackerStore, so I figured I'd share it here.
Benefits:

actually supports serializing to JSON (just pass the encoder / decoder arguments)
implements the collections.abc.Mapping protocol
is significantly denser and more legible
is well-enough typed.

import collections.abc
import logging
import inspect
import typing
import pickle

from rasa_core.tracker_store import TrackerStore
from rasa_core.trackers import DialogueStateTracker
from rasa_core.conversation import Dialogue
from rasa_core.events import deserialise_events

logger = logging.getLogger(__name__)

def serialize_dialogue(dialogue: Dialogue) -> typing.Dict[str, typing.Any]:
    """
    Dumps the `rasa_core.conversation.Dialogue` into a dict.
    """
    return {
        'name': dialogue.name,
        'events': [e.as_dict() for e in dialogue.events]
    }

def deserialize_dialogue(data: typing.Dict[str, typing.Any]) -> Dialogue:
    """
    Loads the `rasa_core.conversation.Dialogue` from a dict.
    """
    return Dialogue(
        name=data['name'],
        events=deserialise_events(data['events']),
    )

class InMemoryTrackerStore(TrackerStore, collections.abc.Mapping):
    def __init__(
            self,
            domain: str,
            encoder: typing.Callable[
                [typing.Dict[str, typing.Any]],
                typing.Any,
            ] = pickle.dumps,
            decoder: typing.Callable[
                [typing.Any],
                typing.Dict[str, typing.Any],
            ] = pickle.loads,
        ):
        super().__init__(domain)
        self._data = {}
        self.encoder = encoder
        self.decoder = decoder

    # collections.abc.Mapping methods
    __getitem__ = lambda self, key: self._data.__getitem__(key)
    __setitem__ = lambda self, key, value: self._data.__setitem__(key, value)
    __delitem__ = lambda self, key: self._data.__delitem__(key)
    __iter__ = lambda self, key: self._data.__iter__()
    __len__ = lambda self, key: self._data.__len__()

    def save(self, tracker: DialogueStateTracker) -> None:
        data = serialize_dialogue(tracker.as_dialogue())
        self[tracker.sender_id] = self.encoder(data)

    def retrieve(self, sender_id: str) -> DialogueStateTracker:
        if sender_id not in self:
            return None
        data = self.decoder(self[sender_id])
        dialogue = deserialize_dialogue(data)
        tracker = self.init_tracker(sender_id)
        tracker.recreate_from_dialogue(dialogue)
        return tracker

we'd be happy to accept a PR for this, that'll make it easier to review the code. But I'm afraid it does still have to be python 2.7 compatible for now :)

@dfee Thanks! I was having trouble with getting these Dialogs serialized.


Enhancement - Constructor which accepts a JSON String

@jelic98

Adding a new Constructor to manage a JSON String as source / PR #2 was created.
Updating the Gradle and Android Environment

Feature branch merged. I will close this issue now.


pip install pypi

安装成功之后   import pclpy 提示下面的错误
Traceback (most recent call last):
File "", line 1, in
File "D:\Python36\lib\site-packages\pclpy_init_.py", line 3, in
import pclpy.pcl as pcl
ImportError: DLL load failed: 找不到指定的模块。

Install OpenNI2, it will solve the issue. Make sure you copy C:\Program Files\OpenNI2\Samples\Bin\OpenNI2.dll.Copy OpenNI2.dll to Lib\site-packages\python_pcl-0.3-py3.6-win-amd64.egg\pcl

I also have the same problem.but there is no "python_pcl-0.3-py3.6-win-amd64.egg" in “site-packages”.The same situation occurred when I used “pip” to install it.
I installed it directly using pycharm. Pycharm uses the local environment. The computer has installed pcl(1.8.1，2015). By the way , I used python3.6.
How can I fix it?

After a day of hard work, I have I finally solved the problem。After trying various methods, I used a relatively silly method. I deleted the ".PyCharm2019.2" folder and reimported the local interpreter.
Although the problem was solved, the reason was still unclear. If anyone knows the reason, please help me explain the reason when you are free. Grateful!

I had the same problem, but after some time, I decided to copy all .dll files from pclpy_dependencies to pclpy folder and this worked for me! Don't know why

I experienced the exact same problem and solved it by recovering the file "C:\WINDOWS\System32\msvcr100.dll" from another system.
I hope this helps!

I have the same issue. None of the proposed solution is working for me

you may need to install c++ tools in Visual Studio Installer

I had the same problem, but after some time, I decided to copy all .dll files from pclpy_dependencies to pclpy folder and this worked for me! Don't know why

I did this and  it works for pclpy. BUT then it went wrong with import pcl. Is there a conflict between them? They don't seem to work at the same time.


License

What is the license you use to publish your code

I have none right now.
But I think I could use the MIT license.
It seems pretty straightforward.

OK MIT it is.


Part of exception from command is not excepted.

I found some exception is not excepted when Codeforces was crowded.
I forget what exception is. It may be RetryError or Timeout.
Anyway, except RequestException is desirable.

Already resolved.


RaSCSI crashes when MacOS reboots

This issue was observed using MacOS 8.1, but likely applies to other versions as well.
When the Mac starts the reboot process, it will send a Flush command to RaSCSI that is not supported. This causes RaSCSI to immediately crash and will cause the Mac to freeze (before it reboots).
To recover from this situation, you need to re-start RaSCSI.
Received an invalid flush command 15!!!!!
rascsi: disk.cpp:7864: void SASIDEV::FlushUnit(): Assertion `0' failed.
make: *** [Makefile:108: run] Aborted

Some observations:
While the drive is in DataOut mode, the initiator is sending a mode select command with the page code 0x08.
From void FASTCALL SASIDEV::FlushUnit().......
Operation Code: [15]
Logical Unit 0, PF 1, SP 0 [10]
Reserved: 00
Reserved: 00
Parameter List Len 18
Reserved: 00
Ctrl Len: 00000000
From SCSIHD::ModeSelect()......
Received page code 8 with length 10
08 0A 00 00 00 00 00 00 00 00 00 00
Mode Page Code 8 is defined in this old Sony interface manual, but the size of the command doesn't match.
http://www.bitsavers.org/pdf/sony/cdrom/Sony_CDU-541_SCSI_Interface_Manual_Mar1990.pdf
For now, updating RaSCSI to accept and ignore Mode Page Code 8 without throwing an error.

I'm going to consider this fix sufficient for now. Issue #1 will need to clean-up the Mac driver-specific commands. Everything seems to still work when this mode select command is ignored (with the patched driver)

@akuker Can you please verify whether this issue still exists? This issue looks very strange, and the work around is not such a good solution. (I have never seen the warning from the work-around, by the way.)
In order to test this you would have to remove the work-around in scsihd.cpp.

Can you please verify whether this issue still exists? This issue looks very strange, and the work around is not such a good solution.

This is an old one! I'm a little swamped at work right now, but will try it out as soon as I get a chance.

@akuker @rdmark Half a year has gone ;-). I suggest to close this ticket and remove the related debug code. There were so many code changes anyway that it's not clear whether this ticket is still valid.

No objections closing this. Thank you for doing some backlog grooming, @uweseimet !


VSC not recognizing system env. variables (Windows 11)

I reset my PC recently and was needing to install node.js and yarn.  So, I installed both and triple checked to make sure that the env. variables were set up correctly in Windows 11 (my Windows 11 version is 10.0.22000 build 22000).  Then, I verified that they were working in the terminal, and they were.  However, VSC failed to recognize node.js and yarn.  So, I restarted my computer several times and reinstall VSC.  Same problem.  Then, I installed a different IDE (atom) to see if node.js and yarn worked there.  It did.
I could not fix this myself.  VSC is broken on my PC.
However, I found a hacky work-around.  If I start VSC from the terminal using "code", then it recognizes node.js and yarn.  However, if I launch VSC from an .exe, then it breaks.
Please fix this soon.  This bug consumed two hours of my work day, and it'll likely waste time for many other people as well.

I believe this is already fixed, can you see if you can reproduce in the Insiders build?