IMPORTANT: PEP 563, PEP 649 and the future of pydantic

Update: see below this has been resolved by a changes in python 3.10 from the python steering council.
Thanks everyone for your thoughts, patience and hard work.

PEP 563 "Postponed Evaluation of Annotations" was introduced in python 3.7 behind the from __future__ import annotations switch, it basically meant all annotations are strings, not python objects.
Pydantic has tried to supported Postponed Annotations since v0.18.0, see #348.
The problem however is that trying to evaluate those strings to get the real annotation objects is really hard, perhaps impossible to always do correctly, see #248 #234 #397 #415 #486 #531 #545 #635 #655 #704 #1298 #1332 #1345 #1370 #1668 #1736 #1873 #1895 #2112 #2314 #2323 #2411
The reasons are complicated but basically typing.get_type_hints() doesn't work all the time and neither do the numerous hacks we've introduced to try and get fix it. Even if typing.get_type_hints() was faultless, it would still be massively slower than the current semantics or PEP 649 (see below).
In short - pydantic doesn't work very well with postponed annotations, perhaps it never will.
The Problem
The problem is that postponed annotations are set to be come default in 3.10, features of which will be frozen in about three week.
Even worse, there's no way to switch back to the current behaviour.
The Solution
The solution to this is PEP 649 developed by Larry Hastings.
As far as I can tell this is the best of both worlds.
The Second Problem
The sad reality however is that it seems very possible that PEP 649 will get rejected and when python 3.10 is released it will break much of pydantic, and thereby FastAPI and all the other libraries that rely on pydantic (as well as other libraries that use annotations at runtime like enforce and typer presumably).
See this very long discussion of the issue on python-dev about whether PEP 649 is a good idea.

The Point
So why am I saying all this apart from whinging about something I don't agree with?
1. This is fair warning
That pydantic might break in a big way if python's core developers continue value principle over pragmatism.
2.  You can help
Thousands of developers and organisations big and small rely on pydantic. Type hints might have been conceived to help readers and static type checkers, but they're now used to great effect at runtime by lots and lots of people - they make python better.
In the end I don't think python's core developers and steering council want to make your experience of python worse. They just haven't realised how important this is. (Even Larry hadn't heard of pydantic or FastAPI until yesterday when he emailed me, I guess they don't read the python developer survey 😉)
~If you value pydantic or FastAPI or other libraries that use annotations at runtime, please (constructively and respectfully) let the python steering council know that you would like PEP 649 to be accepted.~ Please don't contact members of the python community, they're aware of this issue (see below) and are taking it seriously.
I understand the decision on PEP 649 will be made over the next few days, ~so if you're going to do anything do it today~.

Could you please provide some suggestions for the best way to accomplish this ask:

If you value pydantic or FastAPI or other libraries that use annotations at runtime, please (constructively and respectfully) let the python steering council know that you would like PEP 649 to be accepted.

Is there an email I can write to? Some forum to post on? What's the most impactful course of action?

Not really, it's precisely because I don't know how to make my concerns clear that I created this issue.
I've posted to python-dev supporting the PEP, but I suspect that more people chiming in to just support what I've said might do more harm than good.
I've you've spoken to core devs in the past and they know you, perhaps prompting them might be the best thing to do.

As a long time reader of the Python Dev mailing list I would say that it is not necessary to contribute to the mailing list beyond what @samuelcolvin has already posted, unless there are specific technical questions that you are highly qualified to answer.
The post is already extremely forthright and I have no doubt it will generate a lot of discussion. One thing I would say though @samuelcolvin is it will probably help your case a lot to go in to more details about why typing.get_type_hints() doesn't work for pydantic, either because of specific issues, real world performance examples, or if there is some fundamental flaw in the approach itself.
There's a lot of smart people in the Python core Dev team, so it makes sense to state the problem not the solution. It may be that typing.get_type_hints() can be fixed or some other method could be implemented that would breach the gap.
My 2 cents anyway, hope it helps.

I take your point, but the problems with typing.get_type_hints() and PEP 563 are well described in PEP 649 and debated at length in the other thread, I didn't want to bring all that up again.

I'm going to be honest, I skimmed over many dozens of mails for this one.
But to me it seemed the discussions were talking about problems in the abstract, without a real motivation or need for them to be fixed. Whereas specifics on actual problems implementing it, with an actual library, that's used by lots of users, is a more much compelling position.
I would guess some core devs are going to ask questions about this though anyway, so responses to those questions will likely fill the gap.
Best of luck! Don't use this tool on a day to day basis, but it's been great when I have 😄

I think you should explain better why it's impossible to implement lazy evaluation of type hints. The long list of issues doesn't help get a better understanding of the problem in achieving this, and in many it's never explained why it's hard to do.

I take your point, but the problems with typing.get_type_hints() and PEP 563 are well described in PEP 649 and debated at length in the other thread, I didn't want to bring all that up again.

Even there is a long thread, I don't know what point is actually breaking pydanitc.

Do note that as a steering council member it is more important to me to hear from you directly, @samuelcolvin , in PEP 649 itself on how its acceptance/rejection will impact Pydantic as a maintainer of a popular package using runtime annotations. But inundating us with a flood of emails or trying to get a high upvote count is actually a negative as it implicitly tries to put the SC in a position of "us versus them" for this decision (on top of having to sift through more emails with what limited time I have for SC topics).
I am personally already well aware of the popularity of Pydantic, FastAPI, and Typer, so there's no need to prove that to the SC.

My recommendation here is to gather as much detailed, objective and technical information on how PEP 563 is impacting pydantic in a way it cannot adapt without compromising the core of the package. As a member of the steering council, I normally spend many hours parsing the mailing lists, https://discuss.python.org/ and other sources to gather a complete and holistic view of all the pros, cons and different views and how different parties should be impacted in order to make the best decision possible. The fact that this is bringing to our attention very close to beta freeze gives us very limited time to react in a way that keeps out standards, so it would be very very helpful to get this information directly from you without needing to be extracted from tons of messages.
Also, as a release manager of Python 3.10, it makes me sad that the first issues that are mentioned here:

#248 #234 #397 #415 #486 #531 #545 #635 #655 #704 #1298 #1332 #1345 #1370 #1668 #1736 #1873 #1895 #2112 #2314 #2323 #2411

go back to 2018 but we are hearing about all these problems and it's critically dangerously close to beta freeze. My priority as a release manager is to veil for the highest stability for the release. A considerable amount of features are merged days before feature freeze and this already has a considerable amount of chaos and work from the release management team and the buildbot team and the potential of reverting or including such big changes on top of all this with too little time close to feature freeze makes my job much harder as you can imagine.
In any case for us making sure that all our user base is taken into account is a very serious matter so you can be sure that we will take this into account when discussing the overall issue.
I do understand that everyone is very nervous and there is a lot of people worried and all of this is augmented with time limitations, but as we are all volunteers that want the best for our users and the community so let's all be empathetic to each other, and try to work together towards the best solution.
Thanks for your patience and for your understanding.

Hi folks.
Let me state up front that I'm a very satisfied user of pydantic and FastAPI, and I'm very thankful for the work and contributions the maintainers and community around them have put in. :sunny:
I also appreciate the volunteer efforts of the Python core team, Steering Council, and many contributors to Python.
I'm optimistic that we can find a win-win for pydantic / FastAPI and Python. I believe this is possible if we try not to polarize the solution prematurely to "all or nothing" or "accept or reject 649". To accomplish that we need to look at this through the lens of "what is possible", balance the tradeoffs, and work toward a "good but perhaps not ideal" solution.
Thank you and I'm looking forward to seeing us move forward together.

There is an API proposal over adding typing.get_annotations(object) that could help and seems to have similar motivation : https://bugs.python.org/issue43817
Not sure if it helps but an issue were KeyError was raised for typing.get_type_hints was resolved : https://bugs.python.org/issue41515

I will have the same issue in the future.
I wrote my own implemation wich works for the most of my tests (~90%)
https://github.com/FelixTheC/strongtyping/blob/%2347_in_py10_globals_from_func_must_be_known/strongtyping/utils.py#L43
maybe we can try together to solve this problem.

First of all, I really appreciate what the python core developers in general, and the steering council in particular do for us all.
I only heard about the debate about PEP 649 on Wednesday night, and given the extremely narrow window to get my point heard, I banged the drum in every way I could think of. The lesson here is "careful what you wish for", my message has been amplified more than I expected.
I'm sorry if that has wasted anyone's time, even more so if it reduces the chance my request gets a positive reception.
You're also right that i should have engaged with python-dev long ago about this, I'm sorry I did not. Lesson (hopefully) learnt.
I never meant to generate an "us vs. them" atmosphere, and re-reading what I wrote, I don't think that's what I did. I think the most contentious think i said was that python's core might value "principle over pragmatism" - that would be a perfectly justifiable thing for you do in some scenarios, even though I disagree in this case.
A few points in my defence:

I had already (before Larry emailed me about PEP 649) requested to talk about this at the python developer summit in May, it was on my radar and I didn't realise how little time we had
The situation has changed in the last week as Guido himself points out "... the hypothetical future where annotations are not always syntactically expressions (which did not even exist before this week)..." - this change really would be a problem for pydantic and other such libraries
I specifically avoided telling people to email a particular address or contact particular people to avoid individuals being spammed, I also specifically suggested people didn't reply to the python-dev thread with "me too"s.

overall: Thank you for taking this seriously and I'm sorry to have caused a fuss.

I'm now going to spend some time doing my best to produce a coherent explanation of the short-comings of PEP 563 for pydantic to post both here and on python-dev (I assume just posting a link to here on the mailing list will annoy people?)

Would it be possible to use eval to bypass this in some way?
Like eval what you can and make the rest internally a ForwardRef or something

@cheesycod Unfortunately eval() is slow, not always implemented and doesn't always have the necessary context to work. If you define a model or a type inside a function, there is no easy way to retrieve it and make it work (even when using the stack trace)

How important is performance here? I hear "eval is slow", but when looking at that PEP, it looks like while PEP 649 is a little better than Python 3.9, Python 3.10/PEP 563 is much better, as it makes all annotation strings, pretty much leaving you with 0 cost for type annotations - which is affects all code, not just these ~three libraries. If pedantic is a few percent slower but all other code is half a percent faster, you might even win for pydantic libraries, and a big win for all other libraries. There might be some caching tricks to be used here (most of the PEP 563 improvements are for startup / memory usage)?
Also, being able to put invalid objects or usage into type annotations is huge; it's the main reason using Python 3.7 is so much better than 3.6 for typed code, since you can use Python 3.9 and 3.10 constructs in your type annotations, even though they had not been developed yet, you just need a recent MyPy or other type checker. To me it looks like PEP 649 nullifies some of that benefit, as accessing an annotation can crash if invalid objects are used, and invalid objects (which might be used eventually, like Guido's --flag example) might also open up new library ideas. Also, under "other uses" in the description, you have documentation - that usage is certainly not better with PEP 649 than PEP 563!
I'm not very familiar with pydantic, just a bit with typer; to me pydantic is a better dataclasses/attrs with serialization (I actually just heard about it a couple of days ago from a happy user). A MWE for something hard to do would be helpful. "Good" usages of runtime attributes still look like typing, and Pydantic falls into that, so seeing what is hard might really help. Is making a local type a good idea? Is it for a loop or something like that? This seems like it might be very specific to Pydantic, as I can't think of a good reason to make a local type for Typer.

It appears that there are ways to make eval faster if you are using that route using lambdas: https://stackoverflow.com/questions/12467570/python-way-to-speed-up-a-repeatedly-executed-eval-statement
Could also use a optional C module to speed things up as well in performance

I'm now going to spend some time doing my best to produce a coherent explanation of the short-comings of PEP 563 for pydantic to post both here and on python-dev

Well luckily, I was saved the effort. Łukasz Langa (the instigator of PEP 563) has explained it very clearly in another thread on python-dev "PEP 563 in light of PEP 649".
There's another very interesting description of the subtle changes in scope associated with PEP 563, in PEP 649.
I won't go over everything here, but I'll just give one taster of challenges presented by PEP 563:
This works fine:
from __future__ import annotations
from pydantic import BaseModel, PositiveInt

class TestModel(BaseModel):
    foo: PositiveInt

debug(TestModel(foo=1))

But this fails:
from __future__ import annotations

def main():
    from pydantic import BaseModel, PositiveInt

    class TestModel(BaseModel):
        foo: PositiveInt

    debug(TestModel(foo=1))

main()

Why? Because PositiveInt is not defined in the module scope.
Even adding TestModel.update_forward_refs() doesn't help, you need to add TestModel.update_forward_refs(PositiveInt=PositiveInt) (update_forwards_refs() uses logic similar to get_type_hints(), though it doesn't call it).
Note: this isn't a limitation of pydantic or even a true "bug" in python, it is defined (albeit in passing) in PEP 563 "Annotations can only use names present in the module scope..."
This means if pydantic implemented some magic to get the second example to work, it would be contradicting PEP 563.
You might say "well imports like this should always be module level anyway", perhaps true, but what about custom data types which are extensively used with pydantic? There are many good reasons why you might want to use non-module level custom types, currently with python 3.10 that would require complex .update_forward_refs(...) calls.

If you value pydantic or FastAPI or other libraries that use annotations at runtime, please (constructively and respectfully) let the python steering council know that you would like PEP 649 to be accepted.

Considering that PEP 563 is from around four years ago, and that the plan to change this on Python 3.10 has been set in stone for so long, it doesn't see right to unleash a mass of complaints to the core Python developers three weeks before the release happens.
The plan to implement and release these changes on Python were clear even before FastAPI was even written. It was clear from that moment that it was not going to be future-proof. Yet here we are, with a last minute campaign trying to push back on this.

@samuelcolvin I think it would be helpful for you to update the root post for this issue thread to reflect the current state of things. For example, it seems like the Pydantic project has gotten sufficient attention and engagement from the Python Steering Council, so the stuff in "You can help" is probably no longer needed. Strikethrough (~text~) might be a good way to indicate an update to that information but also preserving its history for transparency.

Thanks @jayqi, done.
I'll update the issue again when we hear from the steering council.

Note: this isn't a limitation of pydantic or even a true "bug" in python, it is defined (albeit in passing) in PEP 563 "Annotations can only use names present in the module scope..."
This means if pydantic implemented some magic to get the second example to work, it would be contradicting PEP 563.

Is the reasoning behind this bit of the PEP explained somewhere in more detail than "postponed evaluation using local names is not reliable"? What would be wrong with pydantic getting frame.f_locals from where the class is defined (main in your example) and passing that to get_type_hints or update_forward_refs?

You might say "well imports like this should always be module level anyway", perhaps true, but what about custom data types which are extensively used with pydantic? There are many good reasons why you might want to use non-module level custom types, currently with python 3.10 that would require complex .update_forward_refs(...) calls.

I think using locally defined types (including imports) is perfectly fine and useful and allowing it is important. However I can't see how your custom data types link supports your argument, what are you referring to?

Is the reasoning behind this bit of the PEP explained somewhere in more detail than "postponed evaluation using local names is not reliable"?

The best explanation I can find is here in PEP 649, if you want more detail you would probably need to dig into the PR or python-dev mailing from the time.

However I can't see how your custom data types link supports your argument, what are you referring to?

Because custom types (class definitions which are used as annotations) as well as nested models, might often be defined outside the module scope.

Note: this isn't a limitation of pydantic or even a true "bug" in python, it is defined (albeit in passing) in PEP 563 "Annotations can only use names present in the module scope..."
This means if pydantic implemented some magic to get the second example to work, it would be contradicting PEP 563.

I'm still not sure about this. Again, here's what it says:

Annotations can only use names present in the module scope as postponed evaluation using local names is not reliable (with the sole exception of class-level names resolved by typing.get_type_hints()).

This is under the "Implementation" section. To me this means that implementations are not required to handle "postponed evaluation using local names" automatically for the user. Which makes sense, it'd be impossible, get_type_hints can't guess where to look and by the time it's called the scope may not exist any more. But it still allows passing a local namespace if you want. So I don't think the PEP is forbidding using local names or magically resolving them, just saying that the Python implementation is not required to resolve them for you. Besides, given the confusion about scopes when PEP 563 was accepted, is this still binding?
For pydantic, resolving these is more achievable than for get_type_hints, essentially by resolving the names ASAP (before the scope is deleted) either at class creation time or when update_forward_refs is called. Getting the correct frame isn't trivial but it's not too hard. Users should only notice problems if they write their own helpers wrapping pydantic, especially model class creation, and you can provide an API for such helpers. Here's how I'd solve this:
from __future__ import annotations
import inspect
from pydantic import BaseModel

no_locals_codes = set()

def no_locals_here(f):
    """
    Decorate a function with @no_locals_here
    if it's a wrapper around model class creation
    or update_forward_refs to indicate that the function
    is not the place to look for local type names.
    """
    no_locals_codes.add(f.__code__)
    return f

class AutoBaseModelMeta(type(BaseModel)):
    @no_locals_here
    def __new__(cls, *args, **kwargs):
        result = super().__new__(cls, *args, **kwargs)
        try:
            result.update_forward_refs()
        except NameError:
            pass  # user will have to call update_forward_refs themselves
        return result

class AutoBaseModel(BaseModel, metaclass=AutoBaseModelMeta):
    @classmethod
    def update_forward_refs(cls, **localns):
        frame = inspect.currentframe().f_back
        while frame.f_code in no_locals_codes:
            frame = frame.f_back
        localns = {**frame.f_locals, **localns}
        super().update_forward_refs(**localns)

def main():
    from pydantic import PositiveInt

    class TestModel(AutoBaseModel):
        # An actual forward ref, will raise a NameError in
        # AutoBaseModelMeta.__new__ when calling update_forward_refs,
        # so user still needs to call update_forward_refs
        # but they don't have to specify Sub=Sub
        sub: Sub = None

    class Sub(AutoBaseModel):
        # update_forward_refs fixes this field automatically
        # when the class is defined
        x: PositiveInt

    TestModel.update_forward_refs()

    print(repr(TestModel(sub=Sub(x=2))))

main()

It's not ideal, I prefer PEP 649, but it's a start and it'd be helpful even for current users.

@samuelcolvin @tiangolo and others following this thread,
Thanks for your patience. The Python Steering Council has posted the plans for 3.10 on python-dev mailing list.

Thanks a lot to the SC! ❤ I'm sorry some people felt wronged with this matter and that some extra work is needed to roll back PEP 563 as default. But for what it's worth I'm really glad it turned out well. Postponing the breaking change seems indeed like the most reasonable solution for now.

This is amazing news! 🎉
On behalf of the FastAPI community, thank you @willingc, the steering council, and everyone involved! 🙇
I understand the big effort and commitment the steering council is making with this decision to support the pydantic and FastAPI communities (especially with our short notice), and the last-minute extra work involved in reverting the default for 3.10 (sorry @pablogsal 😬😅). Thanks for that.
This will give us all the time to figure out how to handle it taking into account pydantic/FastAPI and the use cases that would benefit from PEP 563 and PEP 649. 🚀

and the last-minute extra work involved in reverting the default for 3.10 (sorry @pablogsal 😬😅). Thanks for that.

Well, I voted for it so I cannot complain 😉Sometimes being in the SC and being the release manager at the same time ends with me voting to give myself more work 😅

Thank you all so much for listening to us and making a hard decision! 🙇
I'm really sorry we wasted so much of your time @pablogsal by not bringing this up earlier. 🙏
This has been a really positive experience for me and I feel much more positive about talking to the python-dev community.
In future I will engage with the release process and treat it like something I'm a (very small) part of, rather than something that happens to me.

From linke above, python-dev mailing list (emphasis mine):

In the Steering Council’s unanimous opinion, rolling back the default flip
for stringified annotations in Python 3.10 is the least disruptive of all
the options.

I choked when I read "unanimous".  Others' well-wishes have already been better articulated than I could, but I wanted to highlight this.
My sincere thanks to everyone involved!

Hello, I wanted to ask about the current state of the issue. Will python 3.11 ship with additional functions that will enable Pydantic to function correctly?

Hello, I wanted to ask about the current state of the issue. Will python 3.11 ship with additional functions that will enable Pydantic to function correctly?

The steering council has decided to delay the decision on PEP 563 and PEP 649<EMAIL_ADDRESS>Therefore Pydantic will continue to work as normal if you are not using from __future__ import annotations.
Since this thread was originally opened an added function has been added "inspect.get_annotations" which is an official way to get annotations for an object instead of type hints. Not sure it's powerful enough to help pydantic but it at least exists for others.

Yup, I know about the delay of PEP-s. I was more thinking if there are plans for permanent long term solution.

Yup, I know about the delay of PEP-s. I was more thinking if there are plans for permanent long term solution.

I can't speak for authors of pydantic but in other projects I work on that could be affected by this we can not plan a long term solution until a decision on the PEPs and further the future of from __future__ import annotations is decided by the Python Steering Council. As it is unclear what the long term options available will be until then.

It would be helpful if there was a brief summary of the problem that PEP 563 causes for pydantic (and perhaps other projects). I looked at the top of this thread but it is very long and emotional and did not give me a clear understanding of the problem. I clicked on one of the referenced issues (#248) but IIUC it actually describes a problem in pydantic that could be solved with the help of PEP 563 (or with PEP 649).
Note that I am not a user of pydantic; from the README it looks like a nice tool but the example there doesn't look like it would be affected adversely by PEP 563 (assuming pydantic calls get_type_hints() when it sees a string annotation).
Am I to understand that the real problem (i.e. something where working code would break due to PEP 563) is that it is common to define pydantic classes in a function's local scope that are then used in another scope, in a way that causes get_type_hints() to fail?

I'll answer in more detail tomorrow, but yes, in summary the really hard problem is scope. It's not about whether the class is used right there or elsewhere, if the class is not defined at module level, things start breaking.
Larry went through this (and the fact that it was just a side note in the PEP) in great detail, in the many discussions that happened about this at the time. Maybe he would be worth asking?

See https://github.com/samuelcolvin/pydantic/issues/2678#issuecomment-821437650 above.

I am light user of FastAPI and pydantic, but I never used non-module level types with it. You wrote:

You might say "well imports like this should always be module level anyway", perhaps true, but what about custom data types which are extensively used with pydantic? There are many good reasons why you might want to use non-module level custom types, currently with python 3.10 that would require complex .update_forward_refs(...) calls.

I read the custom data types. But there are no examples using non-module level types for good reason.
Would you explain some of "many good reason" for using non-module level custom types?
--
I am not interested in PEP 649 vs PEP 563 (enable by default), but PEP 649 vs current behavior (opt-in PEP 563).  PEP 649 has still some behavior different from current behavior:
types = [int, str, float]

def make_constructors():
    cs = []
    for t in types:
        def c(t=t) -> t:
            return t()
        cs.append(c)
    return cs

cs = make_constructors()
for c in cs:
    print(c.__annotations__["return"])

# Current: int, str, float
# PEP 649: float, float, float

Is there no good reason for this?

A fuller answer to @gvanrossum's question:
There were 3 things that worried me back when I created this issue in April:
1. Performance
Time taken to construct pydantic models - effectively startup time.
typing.get_type_hints(Foo) is much slower than Foo.__annotations__, however that's only a small part of what pydantic does and from some unscientific experiments, it seems that creating models is only around 7-10% slower with from __future__ import annotations than without.
On the other hand, from __future__ import annotations/PEP-563/string-annotations provide a performance improvement in general, so there's two counteracting effects.
I've therefore also tested this in practice by running
[p.write_text(f'from __future__ import annotations\n{p.read_text()}') for p in Path('src').glob('**/*.py')]

on a ~10k fastapi/pydantic web application (a few files caused errors so I had to remove the __future__ declaration), overall this caused starting the app to slow down by 6% - roughly matching what I would expect from experiments on a single model.
Overall, while this isn't ideal, I can live with the performance impact. (If anyone else has seen a wildly different performance impact, feel free to bring that up)
2. Potential change of syntax
There were discussions back in April about a "hypothetical future where annotations are not always syntactically expressions".
Clearly if this became commonplace, it could be a problem for pydantic. However this seems unlikely, and even if it were the case, typing.get_type_hints() or some other function (inside or outside the stdlib) would need to be able to parse those expressions.
And anyway, blocking progress because it would open the door to something in future that could be a problem would be a mistake.
Overall, I think we can ignore this.
3. Scope
This is the only real concern I have with from __future__ import annotations/PEP-563.
Please note: this is not an issue with the concept of annotations being left as strings. It's a problem with the specifics of how PEP-563 defined annotation evaluations, from PEP 563:

Annotations can only use names present in the module scope as postponed evaluation using local names is not reliable (with the sole exception of class-level names resolved by typing.get_type_hints())

Because of that, the following breaks:
from __future__ import annotations
from pydantic import BaseModel

def main():
    from pydantic import PositiveInt

    class TestModel(BaseModel):
        foo: PositiveInt

    debug(TestModel(foo=1))

main()

Of course, I've used from pydantic import PositiveInt here, but it could be any import or a custom type defined within the function, including another pydantic model. It could even be simple type alias like Method = Literal['GET', 'POST']
Pydantic might be able to do some clever stuff to recognise locally defined annotations, but that would:

Require some magic that might be hard to maintain and unreliable
Would mean pydantic was being intentionally un-pythonic by not following PEP 563

Would you explain some of "many good reason" for using non-module level custom types?

Well, all the same reasons people do local imports in any other code (import performance, circular references, name clashes) and the same reasons people define classes or variables locally (performance, name clashes, readability, accessing local variables).

[...] pydantic classes in a function's local scope that are then used in another scope

It's not about where the class is used, in the above example PEP 563 explicitly states that PositiveInt or anything else defined within the function cannot be used a a valid annotation. The code will fail when the class is defined, not when it's used.
Overall, while there could be workarounds, pydantic and other code that relies on runtime use of annotations would be uglier with PEP 563 style annotations scope than without

I hope that helps. I'll add a note to the top of this issue to help any future travellers who pass this way.
Overall, even if PEP 563 is accept as-is, I don't think pydantic is doomed (just a little damaged) - I was mistaken to suggest that originally. Sorry.

Thanks for the clear explanation.
Let's focus on (3). (The example is missing from devtools import debug BTW, and in addition I get an error urging me to call TestModel.update_forward_refs() with the future import enabled. Adding that will indeed give the error about PositiveInt.)
I have a feeling you are reading a bit too much into the sentence you quote from the PEP. What the PEP meant was just that you have to pass the locals containing needed definitions into tying.get_type_hints() or inspect.get_annotations(), which is a bit of a pain when it is being called deep inside a library. Hence the PEP suggests to keep definitions global. The PEP doesn't imply that using local definitions is invalid or unpythonic!
Of the reasons for doing local imports:

performance: totally legit, if it's in a function/method that might never be called and a module that would otherwise not be imported (but I'm guessing importing anything from pydantic does not qualify)
circular references: I wouldn't solve those with local imports (PEP 563 is meant to help here actually)
name clashes: that sounds silly; why not use import ... as ...?

In general, defining classes locally is not a great pattern, except in certain types of unit tests.
With all that said, I agree that the situation is not great for pydantic. But there's no reason for despair either. :-)
Now, I can't look in the future to know what the SC will decide, but it looks like for now they are kicking the can down the road and 3.11 will end up being the same as 3.10 in this respect. So from __future__ import annotations is here to stay for another few releases at least, and people will want to use it for other reasons (as you say, it makes certain things faster).
I see that update_forward_refs() already takes a local namespace, e.g. TestModel.update_forward_refs(**locals()). Why not just recommend that people use that? You could even have a helper that uses sys._getframe(1) to access the parent frame and takes the locals from there.

@gvanrossum what up Guido!? Have you tried pydantic? It’s the best python lib ever written (see #3633 for more info).

The error message is here:

@samuelcolvin @gvanrossum (including @tiangolo as well) I'm happy to see this discussion. Our team uses pydantic and fastapi in our product. Our engineers really enjoy using these libraries (they really rave about the productivity benefits), and there is a lot of positive impact that the libraries are having on Python adoption. If we do a Typing Summit at PyCon again, I encourage Samuel and Sebastian to join the discussion.
Samuel, thanks for the detailed response to Guido's questions and for being open to collaborating and investigating going forward. Let's keep the positive discussion going. Thanks :sunny:

The error message is here:
https://github.com/samuelcolvin/pydantic/blob/9d631a3429a66f30742c1a52c94ac18ec6ba848d/pydantic/fields.py#L832
It could have **locals() added to it.
And here's the update_forward_refs: https://github.com/samuelcolvin/pydantic/blob/87da9ac23f0917741c0536b474cf13cdbed50dfc/pydantic/main.py#L776 - this could have the sys._getframe(1) trick to try to pull the locals (I would assume this would be dependent on if sys contains _getframe, as it's not guaranteed to exist, from what I gather - so both changes together likely would be helpful)
The fact you can detect this and produce a nice suggestive error message is great to see, though! As far as I can tell, local class definitions is still an antipattern for Pydantic, but is just something that comes up occasionally due to the number of users? A minor, detectable inconvenience is not a bad tradeoff, perhaps?
Working example code, FWIW, since a few bits were missing:
# Install pydantic and devtools
from __future__ import annotations
from pydantic import BaseModel
from devtools import debug

def main():
    from pydantic import PositiveInt

    class TestModel(BaseModel):
        foo: PositiveInt

    TestModel.update_forward_refs(**locals()) # can remove this line with the patch below

    debug(TestModel(foo=1))

main()

PS: really dumb question - why can't update_forward_refs() be called in __new__ with this already? Specially, __try_update_forward_refs__ could include the locals. In fact, the following patch completely fixes your example for me - without the update_forward_refs call at all:
diff --git a/pydantic/main.py b/pydantic/main.py
index eea8abc..bc8afdf 100644
--- a/pydantic/main.py
+++ b/pydantic/main.py
@@ -1,4 +1,5 @@
 import warnings
+import sys
 from abc import ABCMeta
 from copy import deepcopy
 from enum import Enum
@@ -289,7 +290,9 @@ class ModelMetaclass(ABCMeta):
         cls = super().__new__(mcs, name, bases, new_namespace, **kwargs)
         # set __signature__ attr only for model class, but not for its instances
         cls.__signature__ = ClassAttribute('__signature__', generate_model_signature(cls.__init__, fields, config))
-        cls.__try_update_forward_refs__()
+
+        localns = sys._getframe(1).f_locals if hasattr(sys, "_getframe") else {}
+        cls.__try_update_forward_refs__(**localns)

         return cls

@@ -765,12 +768,12 @@ class BaseModel(Representation, metaclass=ModelMetaclass):
             return v

     @classmethod
-    def __try_update_forward_refs__(cls) -> None:
+    def __try_update_forward_refs__(cls, **localns: Any) -> None:
         """
         Same as update_forward_refs but will not raise exception
         when forward references are not defined.
         """
-        update_model_forward_refs(cls, cls.__fields__.values(), cls.__config__.json_encoders, {}, (NameError,))
+        update_model_forward_refs(cls, cls.__fields__.values(), cls.__config__.json_encoders, localns, (NameError,))

     @classmethod
     def update_forward_refs(cls, **localns: Any) -> None:

PS: I think __init_subclass__ might really simplify your two-stage metaclass code.

Thanks for everyone's input.
@gvanrossum, good point about assuming debug() is already shoehorned into builtins, I've switched to print().
I noticed that there's a strong argument that annotations as strings doesn't kill pydantic:
It's hidden away in my comments about performance - I modified a medium sized application which makes extensive use of pydantic to insert from __future__ import annotations everywhere and .... it mostly just worked! (albeit, a bit slower to launch, and some issues with annotations-as-strings in FastAPI).
But still I think the FastAPI stuff could be fixed fairly easily. Overall, I think this is a fairly good indicator that in real life most pydantic usage won't be too adversely affected.
On circular imports, I need to think about this more but I'm not entirely convinced it won't be a problem - to avoid circular import problems while making my import global, I need to put the import behind if TYPE_CHECKING, if I do that, at runtime the import won't be available. Am I missing something?

@willingc I'll do my very best to get to pycon. If I can make it, I'll of course join a Typing Summit.

On circular imports, I need to think about this more but I'm not entirely convinced it won't be a problem - to avoid circular import problems while making my import global, I need to put the import behind if TYPE_CHECKING, if I do that, at runtime the import won't be available. Am I missing something?

It depends on how the modules are structured. The problem doesn't occur when each module has a function that uses something from the other module. But it may occur when two classes in different modules reference each other. The solution normally (when using annotations for static typing) is to put the annotations in "forward reference" string quotes -- or use PEP 563. However if each class does runtime introspection at (roughly) the point of definition then you will indeed be in trouble.

However if each class does runtime introspection at (roughly) the point of definition then you will indeed be in trouble.

tortoise-orm uses TYPE_CHECKING when working with tortoise.Models that reside in different modules. This is not an issue unless we try to generate a pydantic model from either of the models. tortoise has a function for this pydantic_model_creator.
I am guessing the best is to implement a workaround for now.


Remove hardcoded values from message

At the moment the current values are hardcoded in the message

We need to change this so it uses the correct values from the message.
The expected behavior is that is uses the sender/receiver role and ID received in the message instead of the hardcoded values we currently use.

Tasks:

[ ] Change hardcoded values to correct values received in message for sender and receiver.

Treated in #541


Fix permissioned pagination

Fixes an error when trying to apply permission checks using the HasRetvalPerm extension on a paginated list.

TypeError: Cannot filter a query once a slice has been taken.

Types of Changes

[ ] Core
[X] Bugfix
[ ] New feature
[ ] Enhancement/optimization
[ ] Documentation

Issues Fixed or Closed by This PR

https://github.com/strawberry-graphql/strawberry-django/issues/471

Checklist

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] I have tested the changes and verified that they work and don't break anything (as well as I can manage).

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (94830a1) 87.90% compared to head (ccb8d77) 87.90%.
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #480   +/-   ##
=======================================
  Coverage   87.90%   87.90%
=======================================
  Files          37       37
  Lines        3167     3167
=======================================
  Hits         2784     2784
  Misses        383      383

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


fix: fix template format of total

The total was displaying as a float with a single decimal which does not
make sense for a USD amount.
This uses format to make it a nice pretty dollar amount

Nice, i saw this yesterday, but didn't fix it <3


Reading EEGLAB requires pymatreader

Reading EEGLAB files requires pymatreader, which however is not a hard dependency. This makes it more difficult for other packages that depend on MNE to load such files, because they have to explicitly list pymatreader as a dependency.
In addition, most EEGLAB files I have seen do not use the v7.3 (HDF5) format, so they could be read with scipy.io.loadmat(), which is already a hard dependency.
Therefore, I see two options here:

Make pymatreader a hard dependency.
Fall back to scipy.io.loadmat() if pymatreader is not installed.

Option 2 implies that when trying to read v7.3 files with the fallback, users would get an error (after which they could still decide to install pymatreader). Furthermore, some tests need to be adapted to distinguish between v7.3 and <v7.3 files (currently 4 out of 17 tests fail).

See also https://eeglab.org/tutorials/03_Dataset_management/datasets.html#dataset-and-file-preferences, where they recommend storing to <v7.3 and only use v7.3 if you have files larger than 2GB.

You mean with just
Pip install mne ?
I would say no. Just install mne with the hdf5 variant

Yes, I mean pip install mne. Right now, this means that users cannot load EEGLAB files at all.
So I take it that you prefer option (2) then? Or do you think it's OK how it is and we shouldn't change anything?

Eeglab is not part of the 80/20 trade off for me.
Just use:
pip install mne[hdf5]
For packages who need hdf5 support

OK, but what about option (2) then? We could enable loading EEGLAB files without any additional dependency. I'd assume that >80% of files are non-HDF5. Currently, we cannot load these files.

For point 2 then if we can have a reader just using scipy that’s great but I suspect it’s a bit of work no? Ideally this effort should done in one place for the python community.

We can use the existing scipy.io.loadmat() reader. I've already implemented the changes, it's not much, the only thing that needs some work is our tests. We'd have to test HDF5 and non-HDF5 EEGLAB files separately.


Show commit messages as link to commits. (Take 2)

As the image above shows, this patch will display the individual commit messages as links to the actual commit in Gitlab. Future improvements would typically be to only show the first line of the commit messages and add a run time argument to disable commit message output.

I wrapped return text with str() to convert the unicode string to a regular string. No idea if this is enough.

Coverage increased (+0.2%) to 89.62% when pulling ecb3b208eb7943fa39b1e8c5316fe089dc883c85 on qvazzler:master into cc63c47a42878ad7a6511740947dc95d309f0e9e on NotSqrt:master.

Coverage increased (+0.2%) to 89.62% when pulling ecb3b208eb7943fa39b1e8c5316fe089dc883c85 on qvazzler:master into cc63c47a42878ad7a6511740947dc95d309f0e9e on NotSqrt:master.

@NotSqrt. So I know essentially very little about Python but I think the test data needed to reflect the new format, so I fixed the broken tests. There was also some unicode in there which I changed back to plain string.

Coverage increased (+0.2%) to 89.62% when pulling 3224dae2196d2e1520642f8358cc0803a4d6e4f7 on qvazzler:master into cc63c47a42878ad7a6511740947dc95d309f0e9e on NotSqrt:master.


copy zarr outputs into place in Assets dir

Currently not getting copied off temp dir into Assets dir.

Noting down for reference:
Assets folder are not removed if they are stale. This means, if the directory exists, the new workflow run will modify the Assets folder with new content, leaving old content if there's no interference. So, in cases of when some outputs are not copied, the old (and maybe wrong) contents might serve as the result.


fix resize problem of split view

Oh sorry, did not see this, I already added those fixes in https://github.com/zeit/hyper/pull/3830/commits/05358b9cf0dfd44eb0aca0a0e3c34e0f02fc35a2


DCOS-14844, DCOS-14756: refactor(services): remove applyPatch function

This is mostly the work of @nLight from #2081 Which has been rebased to current Master and squashed into one commit.
As this PR is introducing a MAYOR refactoring. Please give this PR some serious testing before approving to the changes.
This PR will introduce a mayor change in the way we parse / reduce application configurations in the service form. Now the ADD_ITEM transaction will hold null or a complete object of the item instead of the index. We do this so that we can keep unknown fields from the JSON. Right now this also is
creating redundancy because all the transactions after the ADD_ITEM are just reseting the same
value.
E.G. :
The following JSON is parsed:
{
  "array": [{
    "id": "1234",
    "unknownField": "withoutTransaction"
  }]
}

And will create the following Transactions:

ADD_ITEM, {id: "1234", unknownField: "withoutTransaction", ["array"]}
SET, ['array', 0, 'id'], "1234"

But in this case this redundancy is good,in some places we are depending on the separate Transactions.
Closes DCOS-14844, DCOS-14756
Checklist

[x] Did you add a JIRA issue in a commit message or as part of the branch name?
[x] Did you add new unit tests?
[ ] Did you add new integration tests?
[ ] If this is a regression, did you write a test to catch this in the future?

testses...

run integration tests

As discussed: Pod containers are multiplying as you jump between JSON and form 🐑  🐑  🐑

There seems to be an issue with the health checks json reducer...
HealthChecks.js?9d97:96 Uncaught TypeError: Cannot create property 'protocol' on number '1'

I tried for quite some time to break the UI and it all works great! I also couldn't seem to get the health check error that @orlandohohmeier found, so I'm going to approve.
Great work everyone!

@Poltergeist @nLight Great Job! ⚒ 💯  I'm ok with merging it as it, but would love it one of you could address @weblancaster comments to improve the maintainability.

Found some Bugs regarding the containers. Will add some tests for that.

Looks like there's something going on with the networking page for pods

The issue from @rcorral is fixed.

run integration tests


Migrate to Qwik ⚡

Big PR here 🔥
Description
Project is now organized in 2 main folders (frontend and backend), where frontend now contains an app built with Qwik ⚡
All information about how to build and run the project can be found in the updated README.md
Detailed explanation
The frontend is now built as SSG (Static Site Generation) that improves the overall website speed.
Instead of running a Qwik server by itself, the backend (NestJS) is loading the built solution (/dist folder) from the frontend and serving both apps (frontend and backend) in one.
It can be easily separated in the future by simply updating the API calls from the frontend and running the Qwik server, removing the static site loading from Nest will be required too.
In case of having any problem deploying to HerokuApp use the following commands with Heroku-CLI:
heroku config:set NPM_CONFIG_PRODUCTION=false -a url-shortener-live
heroku config:set NODE_MODULES_CACHE=false -a url-shortener-live
✍ Main tasks
[✔] Reorganize the project to keep frontend and backend in 1 solution
[✔] Migrate frontend to Qwik
[✔] Add scripts to build and run the whole project
[✔] Add scripts to build and run each app
[✔] Add a .code-workspace file to keep the project view clean.
[✔] Keep compatibility with HerokuApp
[✔] Update README.md
👣 Next steps

Follow best practices in frontend using Qwik
Update docker build scripts
Maybe implement a monorepo tool like nx
Optimize overall architecture

@GuillemPM Amazing!
Do you mind update your PR again?
There are a lot of changes that are not include in your PR (Cache implementation for example).

@GuillemPM Thank you so much for the PR!
You contributed a lot for this project, amazing work buddy!
I really liked the way you combined the front and the backend into one deployment.
Thanks again!

@GuillemPM Thank you so much for the PR!
You contributed a lot for this project, amazing work buddy! I really liked the way you combined the front and the backend into one deployment. Thanks again!

Thank you sm! I really appreciate your words.

Is there a way to contact you? 😄

Sure, you can find my LinkedIn in my profile

Sure, you can find my LinkedIn in my profile
I sent you an invitation 🥇


pull request for e2e test at 11/15/2016 18:56:10

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


FORMS-3225 Added fix for blank title getting replaced with component's name

Why does this PR have so many changes. Closing it, please open a new one with only relevant changes


Applying own data and pytorch

I have 500 edf files of ecg. and I would like to apply your model on my data.
They have a lot of electrode pop and bluetooth connections
Also I am familiar with pytorch.
Can you help me  applying Your model ?

Hi @alexxony
Our model and deep learning pipeline use Keras/Tensorflow. We don't have projects of recoding the project or release a PyTorch version any time soon.
However, you could use our code and pipeline to train our model then you can write a code for exporting the model + wights in ONNX format and import that in Pytorch to use it on your own signals.
Since your signals have different electrodes you can filter your signals with our model by filtering them each electrode at the time and merge the resulting signals at the end.
I wish you a good look at your project!!!

I am closing this since it's not really a code related issue


Question : taking advantage of docker caching while building MySqlContainter.

Issue:
I am having a little difficulty in figuring out how to take advantage of docker-caching to improve the speed of using test-containers.
Here is my simple use-case. When executing a test which uses MySQLContainer provided by TestContainers:

The container boots up and executes the init script init_mysql.sql every time for setting up database schema. This step can be cached.

MySQLContainer mySQLContainer = new MySQLContainer()
                .withDatabaseName("test_database")
                .withInitScript("mysql-container/init_mysql.sql")
                .withUsername("test")
                .withPassword("test")
                .start();

After the init script is executed and the container is ready, a java.sql.connection is created to the database and some of the tables are populated with data. This step can also be cached, by somehow included it during the creation of docker image.

All of this takes more ~1m30sec to 2m for the container to be ready for testing.
Question:
How can above code/process be restructured so that it can advantage of docker-caching feature to enable faster testing time ?

unstale


Settings screen

https://user-images.githubusercontent.com/956646/119075945-fc889b00-b9f1-11eb-9719-43048f8519c4.mp4

Resolves #28 by adding a settings  screen
Added an auto-hiding floating settings button on the main screen to show the settings
Added several new strings so localization update is needed, see #61
Added closer color variants to colors.xml and updated themes xml files with new colors, removed unused ones
Removed the splash screen background as it was causing issues with the settings screen, need to fix that, see #62
Removed hiding/showing watermark on tap since that is now used to hide/show settings button, instead you can now permanently hide/show the watermark from settings

When the Tutorial is merged we should add the link from the settings to the TutorialActivity so user can display it again (if they want to)

When the Tutorial is merged we should add the link from the settings to the TutorialActivity so user can display it again (if they want to)

Absolutely, will add any new stuff in separate PRs, or you can add it yourself in your PR if this is merged before.


Rewrite mcscf.addons.project_init_guess

The old project_init_guess function appears to have been designed to produce good initial orbitals for a calculation with a large basis set using an initial guess from a calculation on the same molecule, at the same geometry, carried out using a smaller basis set. Accordingly, only the active orbitals were actually taken from the initial guess; Hartree-Fock was used to first supply a set of inactive/closed/"core" orbitals and the guess active orbitals were then projected into their orthogonal complement space. This makes sense for the small -> large basis set projection, since HF carried out at a large basis would be able exploit higher quantum-number basis functions and are therefore probably closer to the optimized inactive orbitals.
However, this is not an ideal strategy for geometry scans or optimizations, especially when small steps are taken, in which case this strategy essentially throws away all of the effort the MCSCF solver expended in the previous iteration to relax the inactive orbitals in the presence of the electron correlation of the active space. It also does some violence to the active orbital guesses in order to make them orthogonal to the Hartree--Fock inactive orbital guesses, which is really not ideal if you have a large active space and a sensitive CI vector. It also essentially requires one to always converge the HF SCF calculation at every step of a geometry optimization, which shouldn't be necessary and sometimes might not be even possible. It would be better to simply take the initial orbital guesses as provided by the user, and make them orthonormal, giving priority first to the active orbitals (orthonormalizing among themselves), and then the inactive occupied orbitals (project into the orthogonal complement of the active space and then orthonormalize).
Therefore, I rewrote mcscf.addons.project_init_guess to do both using repeated SVDs, in an order controlled by a new optional kwarg. Another new optional kwarg allows one to toggle the substitution of HF inactive orbitals, and these two kwargs default to the old "increase basis set size" behavior of the first paragraph if "prev_mol" is passed and the new "geometry scan" behavior outlined in the second paragraph if "prev_mol" is not passed. The function remains compatible with point-group symmetry and UHF-CASSCF. I added some tests of the new functionality and logic in "test_addons.py", and revised and extended examples.mcscf.14-project_init_guess.py. The improved performance of the new function can be explored by looking at, for instance, examples/mcscf/30-hf_scan. For the singlet, if we compare by the number of MCSCF macro iterations required at each step, we have
> paste <(grep -ch "^macro iter" singlet/old/out_hf-*) <(grep -ch "^macro iter" singlet/new/out_hf-*)
14      6
9       3
9       3
6       3
6       3
5       3
4       3
4       3
3       3
3       3
3       3
3       3
3       3
3       3
3       2
2       2
2       2
2       2
2       2
2       2
2       2
2       2
2       2
2       2
2       2
2       2
2       2
2       2
2       2
2       2
2       2
2       2
2       2
1       1

and if we look at the wall clock (I have 8 cores), we have
> paste <(grep "CPU time for 1-step CASSCF" singlet/old/out_hf-* | awk '{ print $(NF-1) }') <(grep "CPU time for 1-step CASSCF" singlet/new/out_hf-* | awk '{ print $(NF-1) }')
2.33    0.99
1.51    0.51
1.45    0.47
0.96    0.48
0.94    0.49
0.79    0.49
0.67    0.48
0.61    0.52
0.44    0.54
0.43    0.46
0.43    0.45
0.42    0.46
0.40    0.63
0.39    0.46
0.39    0.42
0.28    0.30
0.29    0.32
0.29    0.39
0.29    0.33
0.30    0.35
0.31    0.37
0.29    0.34
0.29    0.29
0.29    0.29
0.29    0.27
0.29    0.28
0.29    0.28
0.28    0.28
0.28    0.28
0.29    0.29
0.28    0.27
0.28    0.42
0.29    0.30
0.16    0.16

For the triplet, the improvement is much more dramatic:
> paste <(grep -ch "^macro iter" triplet/old/out_hf-*) <(grep -ch "^macro iter" triplet/new/out_hf-*)
23      18
16      4
23      3
18      3
19      4
23      3
20      3
23      3
16      3
15      2
22      3
28      3
15      2
17      2
11      2
22      2
3       2
5       2
4       2
5       2
4       2
3       2
3       2
3       2
4       2
3       2
3       2
3       2
2       2
5       2
13      2
6       2
16      2
1       1
> paste <(grep "CPU time for 1-step CASSCF" triplet/old/out_hf-* | awk '{ print $(NF-1) }') <(grep "CPU time for 1-step CASSCF" triplet/new/out_hf-* | awk '{ print $(NF-1) }')
2.44    3.20
1.75    0.59
2.63    0.55
2.01    0.49
2.12    0.49
2.55    0.42
2.26    0.35
2.60    0.30
1.89    0.26
1.88    0.21
2.94    0.25
3.66    0.34
1.62    0.32
1.90    0.20
1.26    0.24
2.02    0.20
0.45    0.24
0.49    0.27
0.47    0.19
0.47    0.17
0.39    0.17
0.28    0.18
0.28    0.21
0.28    0.18
0.33    0.16
0.27    0.16
0.26    0.16
0.26    0.16
0.20    0.16
0.55    0.17
1.57    0.15
0.64    0.16
1.92    0.16
0.07    0.07

@MatthewRHermes thanks for finding this, I assume this affects geometries optimizations too since project_init_guess appears in CASSCF_scanner, is that correct?
https://github.com/pyscf/pyscf/blob/3872975e1a47c942335e5cd3b742246f66c5efd0/pyscf/mcscf/mc1step.py#L533
Have you checked to see if the improvements are similar using the geomopt module?

Yes, in fact anecdotally, that was my whole motivation for doing this.  It's hard to provide MWE of this since the optimization just overcomes the problem if the basis set is small enough, but I think I do see a small effect in examples.geomopt.12-mcscf_excited_states.py:
> grep "1-step CASSCF converged in" 12-mcscf_excited_states.old.out
1-step CASSCF converged in 3 macro (20 JK 7 micro) steps
1-step CASSCF converged in 3 macro (16 JK 6 micro) steps
1-step CASSCF converged in 2 macro (9 JK 4 micro) steps
1-step CASSCF converged in 2 macro (10 JK 5 micro) steps
1-step CASSCF converged in 2 macro (11 JK 4 micro) steps
1-step CASSCF converged in 2 macro (10 JK 5 micro) steps
1-step CASSCF converged in 3 macro (21 JK 7 micro) steps
1-step CASSCF converged in 3 macro (13 JK 5 micro) steps
1-step CASSCF converged in 2 macro (9 JK 4 micro) steps
1-step CASSCF converged in 2 macro (9 JK 4 micro) steps
1-step CASSCF converged in 2 macro (11 JK 4 micro) steps
1-step CASSCF converged in 2 macro (9 JK 4 micro) steps
1-step CASSCF converged in 3 macro (22 JK 6 micro) steps
1-step CASSCF converged in 3 macro (13 JK 5 micro) steps
1-step CASSCF converged in 2 macro (9 JK 4 micro) steps
1-step CASSCF converged in 2 macro (9 JK 4 micro) steps
1-step CASSCF converged in 2 macro (9 JK 4 micro) steps
1-step CASSCF converged in 3 macro (21 JK 7 micro) steps
1-step CASSCF converged in 3 macro (13 JK 6 micro) steps
1-step CASSCF converged in 2 macro (12 JK 5 micro) steps
1-step CASSCF converged in 2 macro (10 JK 5 micro) steps
1-step CASSCF converged in 2 macro (11 JK 4 micro) steps
1-step CASSCF converged in 2 macro (9 JK 4 micro) steps
1-step CASSCF converged in 3 macro (22 JK 6 micro) steps
1-step CASSCF converged in 3 macro (15 JK 7 micro) steps
1-step CASSCF converged in 2 macro (11 JK 4 micro) steps
1-step CASSCF converged in 2 macro (9 JK 4 micro) steps
1-step CASSCF converged in 2 macro (9 JK 4 micro) steps
1-step CASSCF converged in 2 macro (9 JK 4 micro) steps
1-step CASSCF converged in 2 macro (9 JK 4 micro) steps
> grep "1-step CASSCF converged in" 12-mcscf_excited_states.new.out
1-step CASSCF converged in 3 macro (20 JK 7 micro) steps
1-step CASSCF converged in 3 macro (13 JK 5 micro) steps
1-step CASSCF converged in 2 macro (7 JK 3 micro) steps
1-step CASSCF converged in 2 macro (7 JK 3 micro) steps
1-step CASSCF converged in 2 macro (7 JK 3 micro) steps
1-step CASSCF converged in 2 macro (4 JK 2 micro) steps
1-step CASSCF converged in 3 macro (21 JK 7 micro) steps
1-step CASSCF converged in 3 macro (13 JK 6 micro) steps
1-step CASSCF converged in 2 macro (7 JK 3 micro) steps
1-step CASSCF converged in 2 macro (7 JK 3 micro) steps
1-step CASSCF converged in 2 macro (4 JK 2 micro) steps
1-step CASSCF converged in 1 macro (3 JK 1 micro) steps
1-step CASSCF converged in 3 macro (22 JK 6 micro) steps
1-step CASSCF converged in 3 macro (13 JK 5 micro) steps
1-step CASSCF converged in 2 macro (7 JK 3 micro) steps
1-step CASSCF converged in 2 macro (7 JK 3 micro) steps
1-step CASSCF converged in 2 macro (4 JK 2 micro) steps
1-step CASSCF converged in 3 macro (21 JK 7 micro) steps
1-step CASSCF converged in 3 macro (14 JK 5 micro) steps
1-step CASSCF converged in 2 macro (9 JK 3 micro) steps
1-step CASSCF converged in 2 macro (7 JK 3 micro) steps
1-step CASSCF converged in 2 macro (6 JK 2 micro) steps
1-step CASSCF converged in 2 macro (4 JK 2 micro) steps
1-step CASSCF converged in 3 macro (20 JK 6 micro) steps
1-step CASSCF converged in 3 macro (15 JK 6 micro) steps
1-step CASSCF converged in 2 macro (9 JK 3 micro) steps
1-step CASSCF converged in 2 macro (9 JK 3 micro) steps
1-step CASSCF converged in 2 macro (7 JK 3 micro) steps
1-step CASSCF converged in 2 macro (4 JK 2 micro) steps
1-step CASSCF converged in 1 macro (3 JK 1 micro) steps

The numbers of JK and micro steps are different.

That looks great! We've also run into some problems with geometry optimizations of large active spaces so I hope this will solve our problems too. I'm not sure if I'm allowed to merge this, but this PR looks good to me.

LGTM. If no more comments next week, I think it can be merged


Stock Problem on Leetcode

https://blog.1pwnch.com/posts/stock-problem-on-leetcode/

So for 121, if I use the concept of prefix_min, which means the minimum prices so far
the answer would be more organized
 if(prices.size() == 0) return 0;
 int profit = 0, prefix_min = prices[0];
 for(int i = 1; i < prices.size(); i++) {
       prefix_min = min(prefix_min, prices[i]);
       if(prices[i] > prefix_min) profit = max(profit, prices[i] - prefix_min);
  }
return profit;


[OctoAI model provider] Llama3 update

What does this PR do?

This PR does the following:

It updates all OctoAI (Llama model API provider) examples to use the latest Llama3 model instead of Llama2
Some function signatures changed in Langchain around OctoAIEndpoint so code was updated to stay up to date with latest langchain package (langchain==0.1.19)
Examples involving live search and video summarization were updated to reflect the changes in their Replicate counterpart
CC @HamidShojanazeri

Feature/Issue validation/testing
All notebooks modified were tested on a Macbook M1 machine with a fresh python venv.
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,
Pull Request section?
[ ] Was this discussed/approved via a Github issue? Please add a link
to it if that's the case.
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

Thanks for contributing 🎉!

CC @HamidShojanazeri


Doesn't Update When Script Changes

zoton2 was developing his component and using ASLVarViewer for introspection on his script.
Detecting the script changing out underneath us is hard, but LiveSplit 1.7 will have an event for this now, so we can hook into that.  This is for tracking, but can add support for this in the future when 1.7 gets more widely adopted after its release.

I suppose you noticed by now, but the event is in the latest release.

Thanks @Dalet, I've been pretty swamped lately, and I haven't had a chance to get back into speed running or revving a version for this fix. I'll be sure to sync to the latest version and take a look when I do.  Hoping my schedule will get back to normal in the next month or two.


Claim Authentication (windows authentication)

I am trying to build a webapi service using windows authentication with a custom role provider using Claim authentication. Unfortunately I am not able to find a way to use ClaimTransformationAppBuilderExtension since it is only exposed has part of Microsoft.Aspnet.Http.Abstractions. How can we reference or use from Microsoft.AspNet.Builder.Internal.ApplicationBuilder the Microsoft.AspNet.Builder.ClaimsTransformationAppBuilderExtensions.UseClaimsTransformation?
example: app.UseClaimsTransformation(o => o.Transformer = new MyClaimsTransformer());

I don't quite understand the question. UseClaimsTransformation is defined here: https://github.com/aspnet/Security/blob/dev/src/Microsoft.AspNet.Authentication/ClaimsTransformationAppBuilderExtensions.cs#L45
It's an extension method on Microsoft.AspNet.Builder.IApplicationBuilder. You need to include the Microsoft.AspNet.Authentication package and the Microsoft.AspNet.Builder namespace.
It's also not clear what any of this has to do with windows auth.

I am trying to use UseClaimsTransformation but it is not exposed from configure (startup method). Yes I included Authentication and Builder as reference but [[Internal]].ApplicationBuilder does not have the UseClaimsTransformation exposed so it can used as app.UseClaimsTransformation(o => o.Transformer = new ProspectClaimsTransformer());

Before:
using Microsoft.AspNet.Builder;
using Microsoft.AspNet.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json.Serialization;
using Microsoft.AspNet.Authorization;
…
public void Configure(Microsoft.AspNet.Builder.IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
            loggerFactory.AddConsole(Configuration.GetSection("Logging"));
            loggerFactory.AddDebug();
            Microsoft.AspNet.Builder.Internal.ApplicationBuilder appBuild = (Microsoft.AspNet.Builder.Internal.ApplicationBuilder)app;
            app.UseClaimsTransformation(o => o.Transformer = new ProspectClaimsTransformer());

After:
using Microsoft.AspNet.Builder;
using Microsoft.AspNet.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json.Serialization;
using Microsoft.AspNet.Authorization;
…
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
            loggerFactory.AddConsole(Configuration.GetSection("Logging"));
            loggerFactory.AddDebug();
            app.UseClaimsTransformation(o => o.Transformer = new ProspectClaimsTransformer());

Hi Chris,
Yes it is working…
Thanks very much.
Regards,
Vincent Jourdain
Architect/Lead Developer
D: 0800 408 1451
From: Chris R<EMAIL_ADDRESS>Sent: 11 January 2016 15:23
To: aspnet/Security<EMAIL_ADDRESS>Cc: Vincent Jourdain<EMAIL_ADDRESS>Subject: Re: [Security] Claim Authentication (windows authentication) (#649)
Before:
using Microsoft.AspNet.Builder;
using Microsoft.AspNet.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json.Serialization;
using Microsoft.AspNet.Authorization;
…
public void Configure(Microsoft.AspNet.Builder.IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
        loggerFactory.AddConsole(Configuration.GetSection("Logging"));

        loggerFactory.AddDebug();

        Microsoft.AspNet.Builder.Internal.ApplicationBuilder appBuild = (Microsoft.AspNet.Builder.Internal.ApplicationBuilder)app;

        app.UseClaimsTransformation(o => o.Transformer = new ProspectClaimsTransformer());

After:
using Microsoft.AspNet.Builder;
using Microsoft.AspNet.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json.Serialization;
using Microsoft.AspNet.Authorization;
…
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
        loggerFactory.AddConsole(Configuration.GetSection("Logging"));

        loggerFactory.AddDebug();

        app.UseClaimsTransformation(o => o.Transformer = new ProspectClaimsTransformer());

—
Reply to this email directly or view it on GitHubhttps://github.com/aspnet/Security/issues/649#issuecomment-170585653.

Yes it is working


Allocate rightmost available dim in iaranges by default

The current dim allocation behavior for iarange is to allocate the dim to the left of the last allocated dim (and start with the right-most dim). By default, this allocation looks like -1, -2, -3, etc.. If the user manually allocates a dim, the next allocated dim is to the left of this. So for instance, if the outermost dim starts at -3, the dim allocation scheme will be -3, -4, -5 where the first two dims will continue to remain empty.
This proposes a minor change to our default dim allocation scheme so that we allocate the rightmost available dims by default instead of to the left of the last allocated one. So for the previous example, the dim allocation will be -3, -1, -2. As such, default allocation is from the rightmost available and manual placement can be used to deviate from this.
with pyro.iarange("outer", 100, dim=-3):    # dim allocated: earlier = -3, now = -3
    with pyro.iarange("inner", 10):         # dim allocated: earlier = -4, now = -1
        with pyro.iarange("innermost", 5):  # dim allocated: earlier = -5, now = -2
        ...

This will make it easier to wrap existing models inside an outermost num_particles iarange in an automatic way, for instance, without the user having to manually specify dim placement.
Impact:
This will only impact models where the users were relying on this behavior of default allocation to the left of the last assigned dim, i.e. have an outer iarange with a dim arg, but not specifying the dim arg in the inner iaranges, as in the code above. In almost all such cases, we have been using the dim arg in the inner iaranges as well so as not to have empty iarange dims, so this seems like a low impact change, even if it deviates from our existing behavior.
The following examples would work before this PR but would fail after:
@poutine.broadcast
def model_1():
    with pyro.iarange("foo", 2, dim=-2):
        with pyro.iarange("bar", 3):
            x = pyro.sample("x", dist.Normal(0., 1.))
            assert x.shape == (3,2,1)  # after this PR, shape == (2,3)

def model_2():
    foo = pyro.iarange("foo", 2, dim=-2)
    bar = pyro.iarange("bar", 3)
    baz = pyro.iarange("baz", 4, dim=-1)  # errors after this PR
    assert bar.dim == -3  # after this PR, bar.dim == -1

Please do not merge this breaking change. Let's chat on Monday.

Please do not merge this breaking change. Let's chat on Monday.

Sounds good, this isn't urgent.

This should have no impact on existing models since in all of these cases, we were anyways manually allocating all the dims

Why is the conclusion here different from the conclusion in #1115, where we decided not to make this breaking change?  If the goal is to implement parallel ELBO evaluation (#791), is there a way to address that more specifically?

I think it's fine to make a breaking change as long as there is negligible probability that users have written a breakable model. But let's at least find the simplest example of a model that would be broken.

Good idea. I added an Impact section - will be adding a few more failure scenarios for illustration.

Why is the conclusion here different from the conclusion in #1115, where we decided not to make this breaking change?

This preserves the existing allocation scheme, just that it changes where to start the dim allocation from, in a way that should have minimal impact to existing models. I am having a hard time thinking of working models that would break with this change. I have added the discussion tag to this PR for now, until we are all convinced that this is safe. I will update the Impact section too with a few examples. Let me know if you can foresee models where this would be an issue.

If the goal is to implement parallel ELBO evaluation (#791), is there a way to address that more specifically?

To be clear, #791 can be implemented without this change, if the user takes care to augment every iarange with a dim arg. Ideally, a user should be able to wrap their existing model/guide and call SVI with parallel num particles, so asking them to make unrelated changes to their model to use this feature, seems like a leaky abstraction which this PR tries to fix. This seemed like a fix with minimal changes, and reasonable default behavior. But I'm open to other suggestions too!

I will update the Impact section too with a few examples. Let me know if you can foresee models where this would be an issue.

Hmm, you're right, after looking at the examples it doesn't seem so invasive.  I should be able to adapt my vectorization stuff pretty easily to the new semantics.

Thanks for the review, @fritzo! I would just like to test the updated PR with the num particles change. Will update when this is ready to merge.

that will give us an excuse to cut a minor release and post a breaking change notice to forum.pyro.ai. We can pull the pre-1.0 card for this PR, but let's try to avoid doing this often.

Agreed, I don't foresee any further breaking changes with the broadcasting semantics at least.

Any other objections before merging this?

There was one issue that I wanted to bring up. As this stands, if we have an outer iarange with dim < -1, the inner iaranges will start getting allocated from -1 but once the inner dims are expended, we will start allocating to the left. This can have issues when say the user specifies a lower value for max_iarange_nesting. e.g. in the code snippet below, if the outer iarange corresponds to num_particles, we'll have that enmeshed midway, and we'll have a wrong value for max_iarange_nesting for parallel enumeration.
with pyro.iarange("outer", dim=-2):     # dim=-2
    with pyro.iarange("inner_1"):       # dim = -1
        with pyro.iarange("innder_2"):  # dim = -3

I'm not sure if that would be an issue, and we can address that in #1176  itself. Fine to merge otherwise.

we'll have a wrong value for max_iarange_nesting for parallel enumeration.

We could just set a reasonable default for this in our ELBO implementations

Good to merge, pending any further concerns / comments.


Add model class

This PR:

[ ] closes #18
[ ] closes #22
[ ] refactor forcing_io and soil_io and their tests
[ ] add stemmus_scope class and its tests
[ ] updates notebooks

Looks great! This will make it a lot easier to run the model. I just had some small remarks for you to consider, but I will already mark it as approved.

thanks for reviewing. Please see my replies. Then I will merge it.


Allow manual ordering of Solvers for a SolverHandler

Describe the problem
As noted in the "Known Issues", we currently have to set the Solvers' order of execution in case of a problem: it is neither practical nor does it allow it to be done per-instance.
Describe the solution you'd like
Currently, the list of solvers is obtained GetComponents<Solver>(): exposing the list so that it can be manually ordered within the inspector and from the code would solve this issue.
Additional context
Solvers

The list should be returned in the order they're positioned in the Inspector (higher = earlier), so I don't think script execution order needs to be changed. I'll update the known issue to clarify that.
Though! There are some solver improvements planned, so making this list more explicit definitely sounds like something that should be considered.

Oh, alright. Documentation seemed to talk about the script execution order! I apologize, I should've checked :), I didn't know the order of elements returned by GetComponents depended of the order in which they are added.

making this list more explicit definitely sounds like something that should be considered.

Indeed! I can see use cases where the list of Solvers should be determined at runtime, after the initialization of the SolverHandler.


Aded documentation about messagebox instance

@acn-dgopa  should we add documentation about the messageboxInstance model or is that not necessary?

@acn-dgopa should we add documentation about the messageboxInstance model or is that not necessary?

If instance model is documented, then this should also be documented


[SPARK-32748][SQL] Revert "Support local property propagation in SubqueryBroadcastExec"

What changes were proposed in this pull request?

This reverts commit 04f7f6dac0b9177e11482cca4e7ebf7b7564e45f due to the discussion in comment.
Why are the changes needed?

Based on  the discussion in comment, propagation for thread local properties in SubqueryBroadcastExec is not necessary, since they will be propagated by broadcast exchange threads anyway.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Also revert the added test.

Test build #128393 has started for PR 29674 at commit 77e29a8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33018/
Test PASSed.

cc @maropu @cloud-fan

GitHub Actions passed, so merged to master.

Test build #128393 has finished for PR 29674 at commit 77e29a8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128393/
Test PASSed.


Training instruction

Is there any guide to launch training? I tried to run run.sh, but nothing is printed(

Hi, the parameters in run.sh, "stage" and "stop_stage" are both -1 in default, which means running nothing.
As shown below, you can modify them to run some specific stages.

For example, if you want to run stage 1~7, you can run "bash run.sh --stage 1 --stop_stage 7"

Hi, thank you for your answer and this repository!

It's a pleasure. BTW, all you need is to follow the run.sh and run it stage by stage. If everything goes well, you can get a similar result as ours.


update ISCAN Tutorial 1 for current interface

whole ISCAN Tutorial 1 needs to be updated for current ISCAN version, soon.
don't close this without also raising an issue that Tutorial 1 needs to -> readthedocs site, but this is less urgent.
@vannawillerton do you have time soon (like next 24 hours). else I'll do.
(addresses #132 )

Are we transferring to rtd? I don't quite understand your 'do not close' command?
On wordpress this is done and could be closed.

Transferred to RTD and updated.  Still needs higher res pics, but I'll make a separate issue for that


[B16] Inventory blocks can be scanned again and again with different inventories

At least I explain it to myself with the different Inventories part.
So far successfully tested with Chests, Hungry Chests and Immersive Engineering crates.
Didn't work for Furnaces or Essentia Smelters though.

This is a feature so you can easily scan all the items in the chests. Its not a bug

huh. TIL. thanks


Removing last remnants of the old API

I forgot about this when we were removing the role field. Better to take care of it now than never.

recheck


API query endpoint doesn't use the rp query string parameter

Bug report
Ubuntu 16.04, version 1.2.0 (and 1.1.1)
Steps to reproduce:
1. Write some data to a non-default retention policy
~# influx
Connected to http://localhost:8086 version 1.2.0
InfluxDB shell version: 1.2.0
> create database test
> use test
Using database test
> create retention policy alternate on test duration 1d replication 1
> show retention policies on test
name      duration shardGroupDuration replicaN default
----      -------- ------------------ -------- -------
autogen   0s       168h0m0s           1        true
alternate 24h0m0s  1h0m0s             1        false

> insert into alternate mymeas value=1
> select * from test.alternate.mymeas
name: mymeas
time                value
----                -----
1486587715372417917 1

2.  Query the data in the alternate retention policy using the rp query string parameter
~# curl -G "http://localhost:8086/query?db=test&rp=alternate" --data-urlencode "q=SELECT * FROM mymeas"
{"results":[{"statement_id":0}]}

3. Query the data in the alternate retention policy by fully qualifying the measurement
~# curl -G "http://localhost:8086/query?db=test&rp=alternate" --data-urlencode "q=SELECT * FROM test.alternate.mymeas"
{"results":[{"statement_id":0,"series":[{"name":"mymeas","columns":["time","value"],"values":[["2017-02-08T21:01:55.372417917Z",1]]}]}]}

Expected behavior:
I'd expect the request in step two to return the results that I got in step three.
Actual behavior:
The query in step two returns no results.
Additional info: [Include gist of relevant config, logs, etc.]
If I'm reading the logs correctly, the query in step 2 seems to target autogen instead of alternate.

AFAICT the /query endpoint was never coded to accept the rp query parameter (I looked through the history of services/httpd/handler.go through May 2015). The docs say you can use the rp query parameter for /query but the docs are wrong in this case.
I'm re-tagging this issue as a feature request.

Oh, OK! Thanks. I'll get the docs updated.

If you specify an rp -- it is simply ignored and the default rp is used.  So, we should either add the rp into the query API and/or return an error if a user submits a query parameter which is unsupported.

Proposal:  User can specify rp as a query parameter to write to that rp; If the  user does both - provides a fully qualified named and an rp parameter that mis-match - then error.

Tangentially, when (HTTP url param or POST form) db=foo and the queried measurement is bar.rp.m, bar wins. 😬

In my last comment, I was worried that a user could access a database they didn't have access to by providing conflicting db values to the HTTP query and InfluxQL, but have tested and found this is not the case.
@rbetts after playing with this for a couple of hours, I think it feels natural to leave the current behavior. The code refers to the HTTP db param as "defaultDB" here and here, and I'm in favor of continuing this style.

@jacobmarble @rbetts does #9737 satisfy this ticket's requirements, or is there more to do?

I’d like to discuss - respecting rp on /query but not /write is what’s
proposed?
On Thu, Apr 19, 2018 at 06:37 Edd Robinson<EMAIL_ADDRESS>wrote:

@jacobmarble https://github.com/jacobmarble @rbetts
https://github.com/rbetts does #9737
https://github.com/influxdata/influxdb/pull/9737 satisfy this ticket's
requirements, or is there more to do?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/influxdata/influxdb/issues/7973#issuecomment-382689233,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJpvdN3wWUmaFeoR3qLQYhZQIrKWIwkks5tqGjTgaJpZM4L7Yly
.

@rbetts Looking closer at /write behavior.

For documentation change:
rp is now supported on the /query endpoint. A query that specifies a fully qualified series with db.rp.measurement will override the HTTP db and rp HTTP parameters (db override was already true, rp override is new behavior because rp HTTP parameter is new).


test docs updates for react 0.14

This has a couple updates for react 0.14 (not sure if you're updating docs for 0.14 yet though).

We'll update after 0.14 is out, thank you.
Let's keep this hanging.

@mjw56 Any chance you can resolve the merge conflicts (a rebase is fine)? Also, any other updates needed now that we've got a final 0.14?

I'm pretty sure these are updated now.
Thanks for the PR though!

:+1: sweet, thanks!


UNOMI-616 - Update GraphQL Playground to the latest version or switch to GraphiQL

UNOMI-616 - Update GraphQL Playground to the latest version or switch to GraphiQL
Switched from GraphQL Playground to GraphiQL Playground v3.0.6

I think that before merging we should analyze what is making the tests fail. @anatol-sialitski could you look into it ? Thanks.

I tried running the tests locally and it seems there is an error while trying to install the GraphQL feature :
Here's the log from the console :
Fri May 17 14:45:59 CEST 2024: Finish execution of: {version=2.4.0, name='viewEventPagePath', bundle=org.apache.unomi.shell-commands}
Fri May 17 14:45:59 CEST 2024: Migration step: migrationStatus reach: STARTED
Fri May 17 14:45:59 CEST 2024: Migration step: migrationStatus reach: COMPLETED
unomi:start
[2024-05-17T14:46:01,492][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [shuber-macbook.local] updating index lifecycle policy [context-unomi-rollover-policy]
Installing feature cdp-graphql-feature
and from the logs :
2024-05-17T14:47:18,897 | ERROR | pool-13-thread-1 | BundleWatcherImpl                | 218 - org.apache.unomi.lifecycle-watcher - 2.5.0.SNAPSHOT |  Error when installing cdp-graphql-feature feature
java.lang.InterruptedException: null
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:418) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:190) ~[?:?]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvisionInThread(FeaturesServiceImpl.java:1002) ~[?:?]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvisionInThread(FeaturesServiceImpl.java:972) ~[?:?]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:820) ~[?:?]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:740) ~[?:?]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.installFeature(FeaturesServiceImpl.java:725) ~[?:?]
at Proxy39f04bd5_537b_4211_9f8f_6b3888df90c2.installFeature(Unknown Source) ~[?:?]
at org.apache.unomi.lifecycle.BundleWatcherImpl.lambda$installFeatures$3(BundleWatcherImpl.java:290) [!/:?]
at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4698) [?:?]
at org.apache.unomi.lifecycle.BundleWatcherImpl.installFeatures(BundleWatcherImpl.java:285) [!/:?]
at org.apache.unomi.lifecycle.BundleWatcherImpl.access$500(BundleWatcherImpl.java:58) [!/:?]
at org.apache.unomi.lifecycle.BundleWatcherImpl$2.run(BundleWatcherImpl.java:319) [!/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2024-05-17T14:47:18,903 | INFO  | features-3-thread-1 | FeaturesServiceImpl              | 13 - org.apache.karaf.features.core - 4.2.15 |    wrap_file__Users_loom_.m2_repository_org_codehaus_mojo_animal-sniffer-annotations_1.14_animal-sniffer-annotations-1.14.jar/0.0.0
2024-05-17T14:47:18,903 | WARN  | pool-13-thread-1 | BundleWatcherImpl                | 218 - org.apache.unomi.lifecycle-watcher - 2.5.0.SNAPSHOT |  The bundle org.apache.unomi.cdp-graphql-api-impl is in not active, some errors could happen when using the application
2024-05-17T14:47:18,904 | WARN  | pool-13-thread-1 | BundleWatcherImpl                | 218 - org.apache.unomi.lifecycle-watcher - 2.5.0.SNAPSHOT |  The bundle org.apache.unomi.unomi-graphql-ui is in not active, some errors could happen when using the application

This has been done in the new PR https://github.com/apache/unomi/pull/681


OVA Link Point to Server 2016 Version

The link in the article for the OVA template points to a Windows Server 2016 version that is very out of date.  The text indicates that the link should point to a Windows Server 2022 based template.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 7a4a21a9-fcda-355f-6668-21cd3adf25af
Version Independent ID: 659ad43a-882f-1510-36df-dd1d994b6a09
Content: Azure Migrate appliance - Azure Migrate
Content Source: articles/migrate/migrate-appliance.md
Service: azure-migrate
GitHub Login: @Vikram1988
Microsoft Alias: vibansa

@JeremyWx Thanks for your comment! We will review and provide an update as appropriate.

Hi @JeremyWx,
Thank you for your suggestion. The link has been updated and is available at https://learn.microsoft.com/en-us/azure/migrate/migrate-appliance#appliance---vmware.
Hence, closing this issue.


ZIP Projects

Add better support for uploading ZIP files to Hangar.
What this should do:

Go though jar files in zip and match them with a platform or list of platforms
Extract the various jars so each can be downloaded individually
Add platform selection when user clicks to download the version.

I've also seen people upload extra assets, or even text files with additional instructions into zip files, tho that's a bit of a non-issue imo, since assets should go into the resource/plugin folder, and text on the page itself... but I would expect some people thinking you can do that, so it should be made clear what the zip file upload does and does not do.

Would there be a possibility of supporting "platforms" or standalone jars by using this system?
I would love to provide standalone jars or special versions for different cloud systems via the link and a button or selection of some sort instead. This would help people who don't read the instructions...

additional platforms (other than paper, velocity and waterfall) is unrelated to this issue


Page 2678 "Redistribute Acc. Allocations" - Procedures SetParentSystemId and SetParentTableId from internal to global

Describe the request
Can the procedures please be set from internal to global.
page 2678 "Redistribute Acc. Allocations"
internal procedure SetParentSystemId(NewParentSystemId: Guid)
internal procedure SetParentTableId(NewParentTableId: Integer)

Additional context
We would like to use this functionality in our document pages.
Internal work item: AB#527527

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 21326.


Baseline模型对比模式结果错误

使用https://evalscope.readthedocs.io/zh-cn/latest/user_guides/arena.html#id8
因为没有baseline，我先跑了两个模型的预测，得到了两个jsonl预测。然后将一个作为baseline，一个作为target_answers，用openai评估完后得到registry/data/arena/reviews/review_gpt4_pair_baseline.jsonl，如下：
应该是tie吧，但是每次都是model_a赢。
请问baseline_file的示例和跑一次模型预测一样吗？
{"model_a": "2.2", "model_b": "self", "win_1": "model_a", "win_2": "model_a", "anony": true, "tstamp":<PHONE_NUMBER>.5014713, "language": "NA", "question_id": 1, "category": "generic", "question": "How can I improve my time management skills?", "review_text_1": "Both assistants provide relevant and helpful advice on improving time management skills. However, Assistant A provides a more detailed and structured response, with each point elaborated upon. Assistant A also includes additional tips such as eliminating distractions, practicing saying no, and learning to delegate, which are not mentioned by Assistant B. Furthermore, Assistant A ends with a note of encouragement, emphasizing that improving time management skills takes practice and patience. Therefore, Assistant A's response is more comprehensive and helpful. \n\nFinal Verdict: [[A]]", "review_text_2": "Both assistants provide relevant and helpful advice on improving time management skills. They both suggest prioritizing tasks, creating a schedule, breaking down large tasks, and taking breaks. However, Assistant B provides a more detailed and structured response, including additional tips such as eliminating distractions, learning to say no, and delegating tasks. Assistant B also emphasizes the importance of patience and practice in improving time management skills. Therefore, Assistant B's response is more comprehensive and detailed. \n\nFinal Verdict: [[B]]"}
{"model_a": "2.2", "model_b": "self", "win_1": "model_a", "win_2": "model_a", "anony": true, "tstamp":<PHONE_NUMBER>.6124618, "language": "NA", "question_id": 2, "category": "generic", "question": "What are the most effective ways to deal with stress?", "review_text_1": "Both assistants provide relevant and accurate responses to the user's question about dealing with stress. However, Assistant A's response is more detailed and comprehensive, offering a wider range of strategies and explaining how each one can help manage stress. Assistant A also mentions the importance of trial and error in finding what works best for the individual, and suggests seeking professional help if needed. While Assistant B's response is accurate and relevant, it lacks the depth and detail of Assistant A's response. Therefore, Assistant A provides a better answer to the user's question. \n\nFinal Verdict: [[A]]", "review_text_2": "Both assistants provided relevant and helpful responses to the user's question about dealing with stress. They both mentioned similar strategies such as exercise, relaxation techniques, social support, self-care, and mindfulness. However, Assistant B provided a more detailed and organized response, breaking down the strategies into numbered points and elaborating on each one. Assistant B also mentioned cognitive-behavioral therapy, which Assistant A did not, and concluded by reminding the user that everyone deals with stress differently and that it's okay to seek professional help if needed. Therefore, Assistant B's response is more comprehensive and detailed. \n\nFinal Verdict: [[B]]"}

感谢，我们复习一下

就是不太理解第一个问题的的win_1": "model_a", "win_2": "model_a"，明明后面的评价一个是A，一个是B啊。怎么得出来两个win都是model_a？
而且后续在计算指标的时报错：
Traceback (most recent call last):
  File "/root/lhd/eval/evalscope/evalscope/run_arena.py", line 206, in <module>
    main()
  File "/root/lhd/eval/evalscope/evalscope/run_arena.py", line 202, in main
    arena_workflow.run(dry_run=args.dry_run)
  File "/root/lhd/eval/evalscope/evalscope/run_arena.py", line 187, in run
    self.get_rating_results()
  File "/root/lhd/eval/evalscope/evalscope/run_arena.py", line 169, in get_rating_results
    res_list = ae.run(self.review_file)
  File "/root/lhd/py_env/swift/lib/python3.10/site-packages/evalscope/evaluator/rating_eval.py", line 176, in run
    res_list = self.eval_samples([data_df])
  File "/root/lhd/py_env/swift/lib/python3.10/site-packages/evalscope/evaluator/rating_eval.py", line 157, in eval_samples
    res = self.compute_pairwise_rating(raw_data)
  File "/root/lhd/py_env/swift/lib/python3.10/site-packages/evalscope/evaluator/rating_eval.py", line 124, in compute_pairwise_rating
    df = df.groupby(['model']).sum()
  File "/root/lhd/py_env/swift/lib/python3.10/site-packages/pandas/core/frame.py", line 9183, in groupby
    return DataFrameGroupBy(
  File "/root/lhd/py_env/swift/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1329, in __init__
    grouper, exclusions, obj = get_grouper(
  File "/root/lhd/py_env/swift/lib/python3.10/site-packages/pandas/core/groupby/grouper.py", line 1043, in get_grouper
    raise KeyError(gpr)
KeyError: 'model'

其他的两两对比和单独打分没有报错，正常运行