Projection middleware excluding getter-only properties — use anonymous types instead?

Is there an existing issue for this?

[X] I have searched the existing issues

Product
Hot Chocolate
Describe the bug
This problem, if fixed, would also basically resolve both #2365 and #2373; without virtually any substantial changes to Hot Chocolate.
There's a library called EntityFrameworkCore.Projectables that basically adds the ability to declare computed properties on EF Core entity types:
using EntityFrameworkCore.Projectables;

public class Author
{
    public required int Id { get; set; }
    public required string FirstName { get; set; }
    public required string LastName { get; set; }

    [Projectable]
    public string FullName => FirstName + " " + LastName;
}

All you need to do is define a getter-only property like above and you're good to go (it will then replace every occurrence of that property in any expression passed to EF Core with its equivalent expression, which it gathers at compile-time with the help of source generators — you can read more about how it works on its README).
This means that you could now write something like this:
db.Authors.Select(a => new
{
    a.FullName,
});

And it would be translated to the right SQL — as if you had used the expression a.FirstName + " " + a.LastName directly inside the projection:
SELECT (a."FirstName" || ' ') || a."LastName" AS "FullName"
      FROM "Authors" AS a

However, there is ONE thing that stops this from being the perfect solution to the computed property problem in Hot Chocolate+EF:

The problem is that the projection middleware refuses to include any getter-only properties in the projection expression that it generates. So, for example, if you execute the following query against the server:
{
  author {
    fullName
  }
}

The expression it generates and passes to .Select is the following:
_s1 => new Author() {}

It does not include the FullName property in the projection that it sends to EF, and seems to then invoke the property on the application side afterwards.
It would've been perfect if it instead generated this:
_s1 => new { _s1.FullName }

Would it be possible to slightly modify the implementation of the projection middleware such that it uses anonymous objects instead, and also actually includes properties like this in the projection? Or at least provides an option that lets the developer enable this?
As I said in the beginning, this relatively minor adjustment to the projection middleware would effectively once and for all resolve age-old problems such as those described in #2365 and #2373, and any adjacent issues like #4995.
Could you please consider this? Tell me what you think. Thank you.
Steps to reproduce
Explained above.
Relevant log output
No response
Additional Context?
No response
Version
13.0.0
cc: @kresimirlesic

cc: @kresimirlesic

I'm actually interested in whether there's a simple way to achieve this. I am going to go through the docs again for clues, but in my head I'm thinking something like:
descriptor.Field("FullName").Expression(x => x.FirstName + " " + x.LastName);

With a definition like this, the framework can figure out which other fields were referenced by inspecting the provided expression tree, and generate the selects accordingly.

@AntonC9018 I actually proposed that exact thing in #2365 about three years ago, but that issue is still open, and so it was never implemented.
The solution I'm proposing now, which takes advantage of EntityFrameworkCore.Projectables is very much superior in my opinion; because you could still use those same computed properties elsewhere in your code, and they wouldn't be tied to your GraphQL API, and also that Hot Chocolate wouldn't have to do a lot, and the projection middleware, etc. can all be kept pretty simple and straightfroward.

I don't necessarily agree. Forcing it to be a member property is limiting more than anything. Integration with that library could be useful, but the implementation should not be tied to it.

It won't be "tied" to it. As far as HC is concerned, EntityFrameworkCore.Projectables isn't a thing.
Either way, I don't think the change I'm proposing here in unreasonable from HC's point of view, if the projection didn't refuse to include getter-only properties in the projection, this would be working.

We also I think looked at that library. We would not use it ... but we have looked at it and also experimented with it. The next big epic we are working on is AOT and all features that we have now need to become AOT compliant. From this angle we are also looking at projection. We will completely overhaul it but not yet.

I actually talked about this with pascal and shay some time back, we also have this as experimental code somewhere. Essentially instead of projecting into the proper object ... new Author() {} ef can project into an array of object ... we will get boxing in this case but can use that to store anything we cannot store on the object ... but its also stuck in the projection overhaul at the moment.

Good to know. I was about to ask why you'd use an array instead of an anonymous type but I just realized that you actually can't build an expression with an arbitrary anonymous type at runtime; so yeah, it basically has to be either an array or a dictionary (I checked and a dictionary also seems to work in a projection expression).
This flexibility could also potentially open the door to new possibilities. For instance, it would be very useful if another middleware could "inject" an additional field into the projection if needed, so that it could then use that result after the data has been fetched to do something based on it. A use case for this would be when you need some sort of authorization middleware on a Book.DownloadLink field and you want to make sure the current user has actually purchased the associated book. This is a pretty raw idea so I'm not sure if something like this will be achievable with the architecture you guys have in mind.

We also I think looked at that library. We would not use it ... but we have looked at it and also experimented with it.

Do you mean you did not use it, or would not use it somehow because you find it problematic? The wording there is a bit confusing :P

The next big epic we are working on is AOT and all features that we have now need to become AOT compliant. From this angle we are also looking at projection. We will completely overhaul it but not yet.

That's great. Is AOT compatibility planned for v14?

Interesting. I was about to ask why you'd use an array instead of an anonymous type but I just realized that you actually can't build an expression with an arbitrary anonymous type at runtime (unless you use things like TypeBuilder which would be too much overhead); so yeah, it basically has to be either an array or a dictionary.

Well, if you do know all of the properties you can have beforehand, which you should normally, you can just make one anonymous type with all of them on initialization. The question is whether it's actually faster than a dictionary.

Well, if you do know all of the properties you can have beforehand, which you should normally, you can just make one anonymous type with all of them on initialization. The question is whether it's actually faster than a dictionary.

Sure, as I mentioned you use things like TypeBuilder to achieve that. It just seems like it would be less efficient than just using an array, though I'm not sure, it needs to be benckmarked.

BTW ... descriptor.Field("FullName").Expression(x => x.FirstName + " " + x.LastName);
this exists already for a couple of years ...

BTW ... descriptor.Field("FullName").Expression(x => x.FirstName + " " + x.LastName);
this exists already for a couple of years ...

Does it add includes too?

@AntonC9018  So, I have built it into the type system ... but it seems we never fully connected it to the projections engine ... do you want to help get that fully working ... I can guide you on this and we get it into 13.1?

@AntonC9018  So, I have built it into the type system ... but it seems we never fully connected it to the projections engine ... do you want to help get that fully working ... I can guide you on this and we get it into 13.1?

Sure, when I have some time for this.

I will start on version 13.1 end of next week ... give me a ping, it should not be to much to get this working ... I also have an idea for annotation-based. I move that to the 13.1 milestone.

Pascal also had most of the new projection engine done ... I thought its farther off ... there are apparently 3 issues open to get it in... if you guys help we can make it work.

BTW ... descriptor.Field("FullName").Expression(x => x.FirstName + " " + x.LastName);
this exists already for a couple of years ...

Am I missing something?

I also don't see it, and I'm wondering how it relates to #2575.
I think that it's more powerful to allow anything to be projected, and to use the resolver to combine/construct the return value.
descriptor.Field("fullName")
    .Projects(u => new { u.FirstName, u.LastName })
    .Resolve(ctx => ctx.Parent<User>().FirstName + " " + ctx.Parent<User>().LastName);

Maybe there could be an overload of Resolve that gives you the parent automatically?:
descriptor.Field("fullName")
    .Projects(u => new { u.FirstName, u.LastName })
    .Resolve(u => $"{u.FirstName} {u.LastName}");

Pascal also had most of the new projection engine done ... I thought its farther off

Thats great news being able to derive/calculate even simple fields like descriptor.Field("FullName").Expression(x => x.FirstName + " " + x.LastName) and have it working with sorting/filtering/projections would be killer feature imho and something I have been struggling a lot and it seems other people too as there are many issues basically describing this.

@AntonC9018 So, I have built it into the type system ... but it seems we never fully connected it to the projections engine ... do you want to help get that fully working ... I can guide you on this and we get it into 13.1?

Hey, I'm planning to implement this in the next week or two. Should I start by checking out the new projection engine? Is it in a separate branch?

cc @PascalSenn

@michaelstaib @AntonC9018
There is more to this as it may seem at first. The expressions that we have on the field (or any custom expression that you add) are not projectable. We really need to completely rework the projection engine for this to work. If we want to project custom expressions, we can no longer use the projection into "classes".
I did a lot of work around this already, but we stalled as we had a couple of issue and we were busy with other things. The idea of the new projection engine basically is that we want to project into object arrays instead of the actual clr type. We rewrite all the resolvers during the operation compilation and then access the index of the object.
There are a bunch of challenges with this. and we already solved a bunch. I cannot remember which ones are still open.
I think we stopped working on it because we anyway want to rework how data works in 14? or 15?.
To make sure that we do not loose the progress on this:
This PR is JUST a PROOF OF CONCEPT at the moment
Most of this is quickly hacked together to see if the concept works:
https://github.com/ChilliCream/graphql-platform/pull/6156/files

There is more to this as it may seem at first. The expressions that we have on the field (or any custom expression that you add) are not projectable. We really need to completely rework the projection engine for this to work. If we want to project custom expressions, we can no longer use the projection into "classes".

I thought I'd just add some metadata with the expression in way, and have it be used in projection (if we're doing that projection on the db side, which should be supported). I haven't thought about doing the calculation on the client side yet.
At type initialization, if we end up using anonymous types for projection, we'd just add a property for such types.
Do other systems rely on the output type being the same as the queried type? Right now I know for sure that the projection system uses the property info of the same type as the associated type to do member inits. What else breaks?

I did a lot of work around this already, but we stalled as we had a couple of issue and we were busy with other things. The idea of the new projection engine basically is that we want to project into object arrays instead of the actual clr type. We rewrite all the resolvers during the operation compilation and then access the index of the object.

How did you settle on arrays of objects? Have you profiled it against an anonymous type, created once, with all of the properties?

I feel really skeptical about arrays of objects, since they will allocate a lot of garbage on boxes value types. That might really become a problem

Just to give you guys some ideas, I built an in-house solution for this a couple weeks ago and it basically goes as follows:

There are two sets of classes: The EF Core (entity) classes, and the so-called "DTO" classes which are basically the GraphQL object types.
There's a bunch of extension methods that you can use to configure a DTO class as a "mapped" type that corresponds to an entity type. You can then say that property X on the DTO class should be projected to [some expression]. Here's an example:

public class ProductType : ObjectType<ProductDto>
{
    protected override void Configure(IObjectTypeDescriptor<ProductDto> descriptor)
    {
        descriptor.Mapped().To<Product>(d =>
        {
            d.Property(p => p.PurchasesCount)
                .MapTo(p => p.Purchases.Count(p => p.IsFinalized));
        });
    }
}

By default, a property on a DTO type will be implicitly mapped to a property on the corresponding entity type with the same name and type. If there are any properties that are neither implicitly nor explicitly mapped, an exception will be thrown on startup.

Finally, there's a middleware that generates the right projection expression whenever a query comes in, and its implementation is actually relatively simple. Along with an attribute ([UseProjector}) which registers that middleware which you use on any query type field that returns e.g. IQueryable<Product>.

I'm really satisfied with how this turned out, it manages to solve nearly all the problems I previously had, and does so without suffering from the boxing problem, which is a big pro, I think.
You might want to consider something along these lines, so that the whole object array problem and the complexity it introduces is eliminated. I initially wanted to use an object array as well but quickly found out that it makes the implementation considerably more tricky, with little to no benefit, at least in my case.

 Is the mapping expression incorporated into the query, or is it evaluated on the client? If it's incorporated in the query, how did you manage to do that in a middleware?

When working with dtos with automapper, I've come across other issues. Sometimes the queries would randomly overfetch, it doesn't work on the latest version of automapper, it generates joins for ids, because by projecting it loses information about keys. There might be others, I didn't document them. So I'm not willing to use dtos at this point.
It just makes so many other things way more awkward. Like the fact that now the filters are applied to dtos, and not the entities, which makes EF Core lose some context and generate worse queries; while translating queries in terms of dtos to the db entity is a whole slew of complexity and extra work. (Although to be real the new projection engine will also suffer from this).

Is the mapping expression incorporated into the query, or is it evaluated on the client?

It is incorporated into the query, of course.

So I'm not willing to use dtos at this point.

Yes, and the approach I'm talking about solves all of those problems. I'm just borrowing the term "DTO".

It just makes so many other things way more awkward. Like the fact that now the filters are applied to dtos, and not the entities, which makes EF Core lose some context and generate worse queries;

Yeah, exactly. Things like inheritance would also be practically impossible to properly handle with the AutoMapper approach — although this isn't limited to AutoMapper, you could just write projection expressions by hand, what we're really talking about here is intermediate projections, which is when you do a Select on your DbSet and convert it to IQueryable<SomeDto> and then feed that to Hot Chocolate's projection middleware.
In my approach, however, the queries are done against the entities themselves, (i.e. on IQueryable<SomeEntity>) as opposed to the DTOs (e.g. IQueryable<SomeDto>.

Sure, I'll see if I can post some code.

I will really have to understand what you did there, because I made my own thing for automatically applying some filters to queries based on ownership (see the repo and an example). I had to override QueryableProjectionHandler and so that will also probably need adjustments to keep working.

@aradalvand another problem with DTOs as opposed to automatically generated anonymous types is that it's a huge commitment. What I mean is the fact that you can't for example introduce it for a single type in your scheme. If you have e.g. a Company entity and you introduce a DTO for it, all other classes which have a navigation to Company will also have to be replaced with a DTO, where the reference is to CompanyDto instead of Company. This in effect means typing out DTOs for all entities, because they all end up having references to each other in one way or another. The generation of an anonymous type on the other hand can happen seamlessly. All of the dependent types could then just as seamlessly cascade.

I don't think you can generate anonymous types dynamically, you'd have to use System.Reflection.Emit if you want to create a new type at runtime, which doesn't work in AOT; and given that Hot Chocolate plans to support AOT I would presume that's out of the picture.
The only other possibilities left would be using an object array, which has the boxing problem, and using user-defined DTOs.

What I mean is the fact that you can't for example introduce it for a single type in your scheme. If you have e.g. a Company entity and you introduce a DTO for it, all other classes which have a navigation to Company will also have to be replaced with a DTO, where the reference is to CompanyDto instead of Company

I think you might have slightly misunderstood the approach I'm proposing. You have to define DTOs corresponding to all of your entity class upfront. If you have Book, Author, Publisher, etc. as your entity classes, you'd define BookDto, AuthorDto, PublisherDto, etc.. Think of the DTOs as your GraphQL types, they may or may not be mostly identical to your entity classes.

Yeah, that's exactly what I mean. You have to define dtos for all of your entities.

I don't see how that's a problem.

Source gening DTOs instead feels doable. But again, either approach suffers from the db entity information loss problem. You claim yours doesn't, which I'm curios as to how.

It does not. As I said, the queries will be run against an IQueryable of entities, not of DTOs, so there's no "entity information loss".

I don't see how that's a problem. If you want to use this approach it means your GraphQL types differ from your entity classes anyway.

They are the same, I just need a few computed properties.

It does not. As I said, the queries will be run against an IQueryable of entities, not of DTOs, so there's no "entity information loss".

Looking forward to the code, because I don't understand how that's possible

They are the same, I just need a few computed properties.

Don't you also want to hide your foreign key properties as well, for example? Your entity classes contain database-level implementation details like that which shouldn't be part of API schema. You don't want type Book { authorId: Int!; author: Author! } in your GraphQL schema, you just want type Book { author: Author! }.
The point is there are usually more differences between your entity classes and your GraphQL types than "just a few computed properties", at least in my experience, so you might as well separate the two from the very beginning.

Looking forward to the code, because I don't understand how that's possible

I had hacked together a basic (although working) prototype at https://github.com/aradalvand/hc-ef-custom, you check it out. I've refactored things a bit in my main application, so the file structure you see there isn't great, it was just meant to be a quick proof-of-concept.
The bulk of it is the middleware.

@AntonC9018
I understand your concerns, but I'd like to provide some rationale behind our decision to use object arrays for projections.
Regarding the alternative options you proposed, here are some thoughts:
DTOs:

Manually implementing this would be quite demanding and we believe this extra step to be unnecessary as the type itself serves as a DTO. We want to keep the Developer Experience (DX) streamlined.
Even if you auto-generate the DTO, it would come with its own set of problems. For instance, features like aliasing won't work well. Projecting a field with backing properties twice, each with different selection sets, wouldn't be possible.
There's also the overhead of auto-mapper which adds performance cost.

Anonymous Types:

This approach isn't feasible for several reasons. Each unique query would lead to the generation of a new type. As a result, we'd have to load and create these types at runtime, which can be quite slow and resource-intensive. Moreover, once the runtime generates a type, it cannot unload it, which could lead to security issues.
This approach requires reflection to compile the resolvers at runtime, which is slow and not AOT (Ahead-of-Time) compliant.
While you can generate a type once and reuse it, aliases present the same challenges as with DTOs.
Anonymous types also don't work well with AOT.

Arrays:

While using arrays does involve boxing, this is also the case in the resolver pipeline.
The resolvers can potentially be reused, making this a more flexible solution.
Arrays allow us to project any shape needed for the response.
EF Core can project directly into arrays, which is efficient.
I've run benchmarks in the past and arrays performed reasonably well. I've even attempted to pool them and, in the context of EF Core's memory usage, the impact was negligible.

I hope this helps understand why we settled on arrays after all. Let's keep the discussion going

Thanks for the clarification, @PascalSenn , I really appreciate it!

@PascalSenn would you be able to rebase the projection engine PR branch onto master? It's got to have come a long way in the last year, the reworking should probably be done against the latest version.

I also ran into this issue. As a workaround you can do this.
  [NotMapped]
  [Projectable(UseMemberBody = nameof(SqlUserCount))]
  public int UserCount { get; set; }
  private static Expression<Func<Project, int>> SqlUserCount => project => project.Users.Count;

+1 I would like to see something that's done about this!

this seems to work as well:
    [NotMapped]
    [Projectable(UseMemberBody = nameof(SqlBooksCount))]
    public int BooksCount { get; set; }
    private int SqlBooksCount => Books.Count;

[NotMapped]
[Projectable(UseMemberBody = nameof(SqlBooksCount))]
public int BooksCount { get; set; }
private int SqlBooksCount => Books.Count;

I tried to mimic this for a computed property I need:
[NotMapped]
[Projectable(UseMemberBody = nameof(IsInUse))]
public bool ReadOnly { get; set; }
private bool IsInUse => this.WorkflowExecutions != null && this.WorkflowExecutions
    .Any(workflowExecution => workflowExecution.Workflow != null && workflowExecution.Workflow.Id == this.Id);

public ICollection<WorkflowExecution>? WorkflowExecutions { get; set; }

query GetWorkflowReadOnly {
  getWorkflowById (workflowId: 1) {
    id
    readOnly
  }
}

Unfortunately this is not working for me. It compiles, but it does not behave as expected. In which ReadOnly is always false in cases where it should be true.
Am I missing something?

In version 14 when using the new projectable DataLoader we support requirements ....
[ExtendObjectType<Brand>]
public class BrandExtensionsWithRequirement
{
    [BindMember(nameof(Brand.Details))]
    public string GetDetails(
        [Parent("Name")] Brand brand)
        => "Brand Name:" + brand.Name;
}

The parent attribute allows for a selection set syntax describing the resolver requirement for the parent, the new experimental projection engine will ensure that the required data is available.


Add result.cr to Algorithms & Data Structures

:wave:
I've written an implementation of the Result type. I think it will be useful to developers, so I'm adding it here  :)

Aah, I hadn't noticed they were sorted. My apologies, fixed @veelenga

Typically in crystal, this problem would be solved by either exceptions, or returning a union A | B instead of Result(A, B). But Result does have some use in determining which of A or B is the failure or error condition. There's existing patterns that work fantastically when B is Nil (i.e. A | Nil), but when your error type is a little more complex I can see structure is needed.
That being said, there are a few problems with this implementation:

The whole language uses underscore_case for method names, but andThen and orElse aren't?
In T? we use .try to run a block unless the value is nil, so naming andThen as try would make a lot of sense.
All the result implementations I've seen convert the return value from try to be a Result too. They often provide a flat_map which allows you to return another Result.

Designing a class like this is a difficult problem, and I think it requires a little bit more thought than this to truly be awesome.

[Thanks @RX14 for your thoughtful feedback - I wondered if you might show up! :laughing:
There's enough in what you've said for me to go away and chew on, so I'll close this and come back later :+1:


Resolve chicken-vs-egg scenario of initial Heroku deployment vs initial DANDI Infrastructure deployment

Girder is the basis for the DANDI Archive API -- https://github.com/dandi/dandi-infrastructure/blob/master/terraform/api.tf#L5-L6.
A chicken vs. egg scenario occurs when using Girder for Heroku deployments of DANDI Archive, as DANDI Archive must be deployed at least once, applying its relevant Procfile to Heroku, as noted here in the Girder docs, prior to Heroku dyno configuration being recognized in corresponding dandi-infrastructure Terraform code.
However, when initially deploying the DANDI Archive, certain environment variables must be set -- in the Django settings file here -- these values are set by the Terraform values set here -- thus, it is impossible to provide a clean, initial terraform apply, causing a chicken vs. egg scenario where DANDI Archive needs to be deployed before dandi-infrastructure, but can't because of missing env vars set via Girder.
This ticket captures this issue -- there are be step-wise mechanisms to get around this (e.g. LINC Archive/infrastructure's prior trial-and-error), albeit messy and confusing to a new user.
Cc @satra @kabilar @yarikoptic @waxlamp

Yikes, I was not aware of this intricacy. Thank you @aaronkanzer for filing the issue. For starter -- do we need that Girder for Heroku (source = "girder/girder4/heroku") or could it be some more plain django for Heroku ? I am yet to fully grasp the situation here

In this case, Girder provisions much of the foundational resources that DANDI Archive relies on -- for starters, see here. We can always take the parts of Girder we like upstream and remove the reference to the module, but that could introduce different complexities.
The core concept to keep in mind here (with Terraform modules in general), is that Terraform is designed to not have modules and resources defined upstream overwritten downstream -- upstream is immutable. Thus, any extension of Terraform is purely extension, not overwriting.
I don't think this is a blocker by any means (we were able to deploy LINC Archive well enough); however, it depends on our appetite for following docs / step-wise approach vs. more automation -- food for thought....

Thanks @aaronkanzer.  This is good to know.  For LINC, what manual steps did you have to take to get around this circular dependency?  I may have missed it in https://github.com/dandi/handbook/pull/104.

Thanks @aaronkanzer. This is good to know. For LINC, what manual steps did you have to take to get around this circular dependency? I may have missed it in dandi/handbook#104.

@kabilar -- definitely going to add handbook docs regarding this -- for LINC, I went into the Heroku app and populated the env vars manually in the Heroku console -- just a quick fix, but another manual step (see screenshot below -- red square is just hiding the env vars)

@kabilar -- definitely going to add handbook docs regarding this -- for LINC, I went into the Heroku app and populated the env vars manually in the Heroku console -- just a quick fix, but another manual step (see screenshot below -- red square is just hiding the env vars)

Thanks.  I would suggest that we keep this as a manual step for now.  To resolve this circular dependency, I agree that we should update the Girder Readme and DANDI Handbook.

Girder Readme

@kabilar this isnt so much a problem with Girder, rather, it is the environ=True kwarg passed in the settings.py of DANDI Archive (https://github.com/dandi/dandi-archive/blob/master/dandiapi/settings.py#L88-L112)

@kabilar this isnt so much a problem with Girder, rather, it is the environ_required=True kwarg passed in the settings.py of DANDI Archive (https://github.com/dandi/dandi-archive/blob/master/dandiapi/settings.py#L88-L112)
If environ_required was set to False, than this chicken vs. egg scenario would be resolved

Thanks.  Is it possible to not hardcode this argument?  I am not sure where mutate_configuration gets called.

Is this problem addressed by the documentation note in the official docs for the Terraform Resonant module?


SharePoint JSLink file create/update operations do not work properly when not root site collection

Hey,
Like the title says, the error occurs while trying to create/update  JSLink file when site collection uses managed path.
Potentially the issue is related with the method SP.Web.getFolderByServerRelativeUrl, it seems path variable does not take into account site relative url in SPActions.getCode_createFileInSharePoint method since it's value comes from FilesList.filespath property as '/Style Library/':
var files = context.get_site().get_rootWeb().getFolderByServerRelativeUrl(path).get_files();
After applying the following fix:
var files = context.get_site().get_rootWeb().getFolderByServerRelativeUrl(_spPageContextInfo.siteServerRelativeUrl + path).get_files();
the file could be saved/updated properly.
Best regards,
Vadim the Great!

Should work now


Cannot build on python 3.7.2, ‘PyThreadState’ has no member named ‘exc_type’ and similar errors

Environment:
CPython 3.7.2
Cython 0.29.13 installed from PyPI
Attempting to install gil_load 0.3.5 from PyPI fails. See errors below.
$ pip3 install gil_load
Collecting gil_load
  Using cached https://files.pythonhosted.org/packages/2e/32/be5dbb75fe4ec3308578f5e3164248bd5509ebe59591fa72d9adf244ba9f/gil_load-0.3.5.tar.gz
Building wheels for collected packages: gil-load
  Building wheel for gil-load (setup.py) ... error
  Complete output from command /home/vagrant/venvs/3.7.2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-q2p2qbsg/gil-load/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-0s_x6skj --python-tag cp37:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/gil_load
  copying gil_load/__version__.py -> build/lib.linux-x86_64-3.7/gil_load
  copying gil_load/__init__.py -> build/lib.linux-x86_64-3.7/gil_load
  running build_ext
  building 'gil_load.gil_load' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/gil_load
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/vagrant/.pyenv/versions/3.7.2/include/python3.7m -c gil_load/gil_load.c -o build/temp.linux-x86_64-3.7/gil_load/gil_load.o
  gil_load/gil_load.c: In function ‘__Pyx__ExceptionSave’:
  gil_load/gil_load.c:6848:19: error: ‘PyThreadState’ has no member named ‘exc_type’
       *type = tstate->exc_type;
                     ^
  gil_load/gil_load.c:6849:20: error: ‘PyThreadState’ has no member named ‘exc_value’
       *value = tstate->exc_value;
                      ^
  gil_load/gil_load.c:6850:17: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       *tb = tstate->exc_traceback;
                   ^
  gil_load/gil_load.c: In function ‘__Pyx__ExceptionReset’:
  gil_load/gil_load.c:6857:22: error: ‘PyThreadState’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                        ^
  gil_load/gil_load.c:6858:23: error: ‘PyThreadState’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                         ^
  gil_load/gil_load.c:6859:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                      ^
  gil_load/gil_load.c:6860:11: error: ‘PyThreadState’ has no member named ‘exc_type’
       tstate->exc_type = type;
             ^
  gil_load/gil_load.c:6861:11: error: ‘PyThreadState’ has no member named ‘exc_value’
       tstate->exc_value = value;
             ^
  gil_load/gil_load.c:6862:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tstate->exc_traceback = tb;
             ^
  gil_load/gil_load.c: In function ‘__Pyx__GetException’:
  gil_load/gil_load.c:6907:22: error: ‘PyThreadState’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                        ^
  gil_load/gil_load.c:6908:23: error: ‘PyThreadState’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                         ^
  gil_load/gil_load.c:6909:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                      ^
  gil_load/gil_load.c:6910:11: error: ‘PyThreadState’ has no member named ‘exc_type’
       tstate->exc_type = local_type;
             ^
  gil_load/gil_load.c:6911:11: error: ‘PyThreadState’ has no member named ‘exc_value’
       tstate->exc_value = local_value;
             ^
  gil_load/gil_load.c:6912:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tstate->exc_traceback = local_tb;
             ^
  gil_load/gil_load.c: In function ‘__Pyx__ExceptionSwap’:
  gil_load/gil_load.c:8083:22: error: ‘PyThreadState’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                        ^
  gil_load/gil_load.c:8084:23: error: ‘PyThreadState’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                         ^
  gil_load/gil_load.c:8085:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                      ^
  gil_load/gil_load.c:8086:11: error: ‘PyThreadState’ has no member named ‘exc_type’
       tstate->exc_type = *type;
             ^
  gil_load/gil_load.c:8087:11: error: ‘PyThreadState’ has no member named ‘exc_value’
       tstate->exc_value = *value;
             ^
  gil_load/gil_load.c:8088:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tstate->exc_traceback = *tb;
             ^
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for gil-load
  Running setup.py clean for gil-load
Failed to build gil-load
Installing collected packages: gil-load
  Running setup.py install for gil-load ... error
    Complete output from command /home/vagrant/venvs/3.7.2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-q2p2qbsg/gil-load/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-cs0rw4he/install-record.txt --single-version-externally-managed --compile --install-headers /home/vagrant/venvs/3.7.2/include/site/python3.7/gil-load:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/gil_load
    copying gil_load/__version__.py -> build/lib.linux-x86_64-3.7/gil_load
    copying gil_load/__init__.py -> build/lib.linux-x86_64-3.7/gil_load
    running build_ext
    building 'gil_load.gil_load' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/gil_load
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/vagrant/.pyenv/versions/3.7.2/include/python3.7m -c gil_load/gil_load.c -o build/temp.linux-x86_64-3.7/gil_load/gil_load.o
    gil_load/gil_load.c: In function ‘__Pyx__ExceptionSave’:
    gil_load/gil_load.c:6848:19: error: ‘PyThreadState’ has no member named ‘exc_type’
         *type = tstate->exc_type;
                       ^
    gil_load/gil_load.c:6849:20: error: ‘PyThreadState’ has no member named ‘exc_value’
         *value = tstate->exc_value;
                        ^
    gil_load/gil_load.c:6850:17: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         *tb = tstate->exc_traceback;
                     ^
    gil_load/gil_load.c: In function ‘__Pyx__ExceptionReset’:
    gil_load/gil_load.c:6857:22: error: ‘PyThreadState’ has no member named ‘exc_type’
         tmp_type = tstate->exc_type;
                          ^
    gil_load/gil_load.c:6858:23: error: ‘PyThreadState’ has no member named ‘exc_value’
         tmp_value = tstate->exc_value;
                           ^
    gil_load/gil_load.c:6859:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tmp_tb = tstate->exc_traceback;
                        ^
    gil_load/gil_load.c:6860:11: error: ‘PyThreadState’ has no member named ‘exc_type’
         tstate->exc_type = type;
               ^
    gil_load/gil_load.c:6861:11: error: ‘PyThreadState’ has no member named ‘exc_value’
         tstate->exc_value = value;
               ^
    gil_load/gil_load.c:6862:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tstate->exc_traceback = tb;
               ^
    gil_load/gil_load.c: In function ‘__Pyx__GetException’:
    gil_load/gil_load.c:6907:22: error: ‘PyThreadState’ has no member named ‘exc_type’
         tmp_type = tstate->exc_type;
                          ^
    gil_load/gil_load.c:6908:23: error: ‘PyThreadState’ has no member named ‘exc_value’
         tmp_value = tstate->exc_value;
                           ^
    gil_load/gil_load.c:6909:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tmp_tb = tstate->exc_traceback;
                        ^
    gil_load/gil_load.c:6910:11: error: ‘PyThreadState’ has no member named ‘exc_type’
         tstate->exc_type = local_type;
               ^
    gil_load/gil_load.c:6911:11: error: ‘PyThreadState’ has no member named ‘exc_value’
         tstate->exc_value = local_value;
               ^
    gil_load/gil_load.c:6912:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tstate->exc_traceback = local_tb;
               ^
    gil_load/gil_load.c: In function ‘__Pyx__ExceptionSwap’:
    gil_load/gil_load.c:8083:22: error: ‘PyThreadState’ has no member named ‘exc_type’
         tmp_type = tstate->exc_type;
                          ^
    gil_load/gil_load.c:8084:23: error: ‘PyThreadState’ has no member named ‘exc_value’
         tmp_value = tstate->exc_value;
                           ^
    gil_load/gil_load.c:8085:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tmp_tb = tstate->exc_traceback;
                        ^
    gil_load/gil_load.c:8086:11: error: ‘PyThreadState’ has no member named ‘exc_type’
         tstate->exc_type = *type;
               ^
    gil_load/gil_load.c:8087:11: error: ‘PyThreadState’ has no member named ‘exc_value’
         tstate->exc_value = *value;
               ^
    gil_load/gil_load.c:8088:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
         tstate->exc_traceback = *tb;
               ^
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/home/vagrant/venvs/3.7.2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-q2p2qbsg/gil-load/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-cs0rw4he/install-record.txt --single-version-externally-managed --compile --install-headers /home/vagrant/venvs/3.7.2/include/site/python3.7/gil-load" failed with error code 1 in /tmp/pip-install-q2p2qbsg/gil-load/
You are using pip version 19.0.1, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$

I know that gil_load is not been tested on Python 3.7, so this isn't a regression bug - just missing support.
It looks like other cython based modules are having similar issues:

https://github.com/kundajelab/genomelake/issues/21
https://github.com/mcfletch/pyopengl/issues/11
etc..

Alternatively, when I install gil_load directly from the source code like so (I had to figure out that 0.3.5 was commit 36de61d8f08d01d9e2c858d3ee24a1381c1558e7 based on browsing the git history as the repo has no tags), I can compile and install.
$ python setup.py install --use-cython
Compiling gil_load/gil_load.pyx because it changed.
[1/1] Cythonizing gil_load/gil_load.pyx
/home/vagrant/venvs/3.7.2/lib/python3.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /vagrant/apps/gil_load/gil_load/gil_load.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
running install
running bdist_egg
running egg_info
creating gil_load.egg-info
writing gil_load.egg-info/PKG-INFO
writing dependency_links to gil_load.egg-info/dependency_links.txt
writing top-level names to gil_load.egg-info/top_level.txt
writing manifest file 'gil_load.egg-info/SOURCES.txt'
reading manifest file 'gil_load.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'gil_load.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/gil_load
copying gil_load/__init__.py -> build/lib.linux-x86_64-3.7/gil_load
copying gil_load/__version__.py -> build/lib.linux-x86_64-3.7/gil_load
running build_ext
building 'gil_load.gil_load' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/gil_load
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/vagrant/.pyenv/versions/3.7.2/include/python3.7m -c gil_load/gil_load.c -o build/temp.linux-x86_64-3.7/gil_load/gil_load.o
gil_load/gil_load.c: In function ‘__pyx_pw_8gil_load_8gil_load_6_run’:
gil_load/gil_load.c:3691:17: warning: ‘__pyx_v_f’ may be used uninitialized in this function [-Wmaybe-uninitialized]
                 (void)(fprintf(__pyx_v_f, ((char const *)"%s  GIL load: %.2f (%.2f, %.2f, %.2f)\n"), __pyx_v_timestamp, __pyx_v_8gil_load_8gil_load_gil_load, __pyx_v_8gil_load_8gil_load_gil_load_1m, __pyx_v_8gil_load_8gil_load_gil_load_5m, __pyx_v_8gil_load_8gil_load_gil_load_15m));
                 ^
gil_load/gil_load.c:3279:9: note: ‘__pyx_v_f’ was declared here
   FILE *__pyx_v_f;
         ^
gcc -pthread -shared -L/home/vagrant/.pyenv/versions/3.7.2/lib -Wl,-rpath=/home/vagrant/.pyenv/versions/3.7.2/lib -L/home/vagrant/.pyenv/versions/3.7.2/lib -Wl,-rpath=/home/vagrant/.pyenv/versions/3.7.2/lib build/temp.linux-x86_64-3.7/gil_load/gil_load.o -L/home/vagrant/.pyenv/versions/3.7.2/lib -lpython3.7m -o build/lib.linux-x86_64-3.7/gil_load/gil_load.cpython-37m-x86_64-linux-gnu.so -lrt
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/gil_load
copying build/lib.linux-x86_64-3.7/gil_load/__init__.py -> build/bdist.linux-x86_64/egg/gil_load
copying build/lib.linux-x86_64-3.7/gil_load/__version__.py -> build/bdist.linux-x86_64/egg/gil_load
copying build/lib.linux-x86_64-3.7/gil_load/gil_load.cpython-37m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/gil_load
byte-compiling build/bdist.linux-x86_64/egg/gil_load/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/gil_load/__version__.py to __version__.cpython-37.pyc
creating stub loader for gil_load/gil_load.cpython-37m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/gil_load/gil_load.py to gil_load.cpython-37.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying gil_load.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying gil_load.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying gil_load.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying gil_load.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
gil_load.__pycache__.gil_load.cpython-37: module references __file__
creating dist
creating 'dist/gil_load-0.3.5-py3.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing gil_load-0.3.5-py3.7-linux-x86_64.egg
creating /home/vagrant/venvs/3.7.2/lib/python3.7/site-packages/gil_load-0.3.5-py3.7-linux-x86_64.egg
Extracting gil_load-0.3.5-py3.7-linux-x86_64.egg to /home/vagrant/venvs/3.7.2/lib/python3.7/site-packages
Adding gil-load 0.3.5 to easy-install.pth file

Installed /home/vagrant/venvs/3.7.2/lib/python3.7/site-packages/gil_load-0.3.5-py3.7-linux-x86_64.egg
Processing dependencies for gil-load==0.3.5
Finished processing dependencies for gil-load==0.3.5

There are a few things from the output that I wonder might be causing problems of some kind, but I don't know:
Cython directive 'language_level' not set, using 2 for now (Py2).

gil_load/gil_load.c: In function ‘__pyx_pw_8gil_load_8gil_load_6_run’:
gil_load/gil_load.c:3691:17: warning: ‘__pyx_v_f’ may be used uninitialized in this function [-Wmaybe-uninitialized]
                 (void)(fprintf(__pyx_v_f, ((char const *)"%s  GIL load: %.2f (%.2f, %.2f, %.2f)\n"), __pyx_v_timestamp, __pyx_v_8gil_load_8gil_load_gil_load, __pyx_v_8gil_load_8gil_load_gil_load_1m, __pyx_v_8gil_load_8gil_load_gil_load_5m, __pyx_v_8gil_load_8gil_load_gil_load_15m));
                 ^
gil_load/gil_load.c:3279:9: note: ‘__pyx_v_f’ was declared here
   FILE *__pyx_v_f;
         ^

Is this an easy fix for gil_load?

Thanks for the report! I'm looking into it, and I think the problem is simply that I distribute C files instead of cython files. If I allow cython to generate the C code at build time instead of distributing pre-generated C, then things ought to be less fragile to version changes. Things are working for me locally with Pytohn 3.7, and I'm just fixing up some issues with this package (it was actually in the middle of a pretty big change and so git-master isn't functional right now) and will release a new version soon!

I'm not sure about the warning you're seeing "note: ‘__pyx_v_f’ was declared here", I don't see that one - but then again I'm working on git-master. The Cython language level warning I do see - I guess I'll declare language level = 3 and confirm everything still works with the new version.

That makes sense. Can you git tag the already-released versions on the repo, as well? Right now anyone trying to collaborate/test will have a bit of a harder time.

Ah, sure thing, have pushed some tags!
Version 0.4 should be out today or tomorrow once I finish updating the README.

I am standing by to test! :)

@chrisjbillington let me know if there is anything I can do to help you out!

Is there any way you could create a 0.3.6 release for just the build issues, without mixing it in with the larger effort to track different threads separately in 0.4.x?
Are the changes simple to just fix the build? Can you explain what is involved?

Sorry for the delay, there was not much work getting 0.4 released, I just didn't get to it last week. But, I have now released it. It works a little bit differently (requires you to run your script with python -m gil_load prefixed on the command line), so you should check the readme for instructions.
Yes, I could make a 0.3.x release that fixed the build issue, but I am not inclined to do this unless there is a good reason, since 0.4 supersedes 0.3. Let me know if this is something important to you, otherwise hopefully 0.4 works for you!

Ah. This does not help me particularly much, as I'm running mod_wsgi.  It does not look like this file is intended to be used in such an environment. Any tips on how to make this work in a nice way?
The gil_load/__main__.py file contains (as of 0.4):
import gil_load
import os
import sys

preload_path = gil_load.preload_path

LD_PRELOAD = os.getenv('LD_PRELOAD')
if LD_PRELOAD is not None:
    preload_path += ':' + LD_PRELOAD

os.environ['LD_PRELOAD'] = preload_path

os.execv(sys.executable, [sys.executable] + sys.argv[1:])

You could manually set LD_PRELOAD in your environment prior to starting the web server. Or, is there a way to set environment variables for the Python process(es) spawned by mod_wsgi? In either case, setting LD_PRELOAD to gil_load.preload_path (which is just the path to preload.so within the gil_load install directory) will do the trick. That's all gil_load cares about, and __main__.py is just a wrapper to set the environment variable as you can see.
It would be preferable to set it only for the Python process, since the patched libc functions will slow down your web server slightly and I think running LD_PRELOAD in a web server is probably not a good idea security-wise since it increases the attack surface to include bugs in my code (though of course that is true to a lesser extent regardless of how you use my code!).
The only thing is that the environment variable must be set prior to Python process startup.
If this is still a pain in the neck, I can modify the code so that if LD_PRELOAD has not been set it can still monitor overall GIL load, but not the contention or individual thread statistics.

According to the mod_wsgi docs: "The SetEnv directive has no effect on ‘os.environ’ and there is no way through Apache configuration directives to affect what is in the process environment."
It sounds like the Python interpreter for mod_wsgi runs in the same process as Apache. So I think this leaves the only option as setting LD_PRELOAD for the Apache process prior to running it.
Tricky. I'll have a bit of a think as to whether there is anything I can do to gil_load to make it work in contexts like this! I suspect there isn't, other than dropping the per-thread statistics.

mod_wsgi does not always run in the same process as apache. I'm using the WSGIDaemonProcess option for example.

Any further thoughts on this @chrisjbillington ?

A few thoughts:
Environment variables are inherited, so if you set LD_PRELOAD in a way that affected Apache as a whole, it would affect the Python interpreter too whether it's a child process or otherwise. The drawback is that it would slow the web server down slightly by overriding some C stdlib functions, and possibly this might confuse some debuggers or other tools. I wouldn't expect the slowdown to be major though.
I don't know how mod_wsgi works, but can you control the Python process shortly after startup, at all? If so, you could set LD_PRELOAD (this would have no effect on the current process), and then fork, and LD_PRELOAD would take effect in the child process resulting from the fork. Though forking might not be compatible with assumptions made by mod_wsgi, I'm not sure.
Other tools can track the GIL, such as py-spy: https://github.com/benfred/py-spy. Does that suit your needs? You can attach it to a running process by PID.

Thanks, I'll give all that a look/think.

Hi Chris, I work w/ Eric & have been trying to instrument our mod_wsgi app with gil_load.
I'm often getting one of these three errors in our error logs after making first HTTP request against application (but app does occasionally start too, in which case it doesn't output any of these errors).
[Fri Mar 08 18:06:55.685970 2024] [wsgi:error] [pid 235] [remote <IP_ADDRESS>:53768] mod_wsgi (pid=235): Exception occurred processing WSGI script '/var/de-sandbox/apps/edde/.local/edde_learn.wsgi'.
[Fri Mar 08 18:06:55.687406 2024] [wsgi:error] [pid 235] [remote <IP_ADDRESS>:53768] Traceback (most recent call last):
[Fri Mar 08 18:06:55.687815 2024] [wsgi:error] [pid 235] [remote <IP_ADDRESS>:53768]   File "/var/de-sandbox/apps/edde/.local/edde_learn.wsgi", line 1, in <module>
[Fri Mar 08 18:06:55.688152 2024] [wsgi:error] [pid 235] [remote <IP_ADDRESS>:53768]     import gil_load; gil_load.init(); gil_load.test(); gil_load.start()  # provides data for gil-watcher
[Fri Mar 08 18:06:55.688487 2024] [wsgi:error] [pid 235] [remote <IP_ADDRESS>:53768]   File "gil_load/gil_load.pyx", line 732, in gil_load.gil_load.test
[Fri Mar 08 18:06:55.688860 2024] [wsgi:error] [pid 235] [remote <IP_ADDRESS>:53768] AssertionError: gil_held() returned 0 when we were holding the GIL

[Fri Mar 08 18:28:26.089274 2024] [wsgi:error] [pid 234] [remote <IP_ADDRESS>:46746] mod_wsgi (pid=234): Exception occurred processing WSGI script '/var/de-sandbox/apps/edde/.local/edde_learn.wsgi'.
[Fri Mar 08 18:28:26.089925 2024] [wsgi:error] [pid 234] [remote <IP_ADDRESS>:46746] Traceback (most recent call last):
[Fri Mar 08 18:28:26.090308 2024] [wsgi:error] [pid 234] [remote <IP_ADDRESS>:46746]   File "/var/de-sandbox/apps/edde/.local/edde_learn.wsgi", line 1, in <module>
[Fri Mar 08 18:28:26.090588 2024] [wsgi:error] [pid 234] [remote <IP_ADDRESS>:46746]     import gil_load; gil_load.init(); gil_load.test(); gil_load.start()  # provides data for gil-watcher
[Fri Mar 08 18:28:26.090865 2024] [wsgi:error] [pid 234] [remote <IP_ADDRESS>:46746]   File "gil_load/gil_load.pyx", line 736, in gil_load.gil_load.test
[Fri Mar 08 18:28:26.091154 2024] [wsgi:error] [pid 234] [remote <IP_ADDRESS>:46746] AssertionError: gil_held() returned 1 when we were not holding the GIL

[Fri Mar 08 18:11:35.523872 2024] [wsgi:error] [pid 234] [remote <IP_ADDRESS>:46516] mod_wsgi (pid=234): Exception occurred processing WSGI script '/var/de-sandbox/apps/edde/.local/edde_learn.wsgi'.
[Fri Mar 08 18:11:35.549002 2024] [wsgi:error] [pid 234] [remote <IP_ADDRESS>:46516] Traceback (most recent call last):
[Fri Mar 08 18:11:35.549617 2024] [wsgi:error] [pid 234] [remote <IP_ADDRESS>:46516]   File "/var/de-sandbox/apps/edde/.local/edde_learn.wsgi", line 1, in <module>
[Fri Mar 08 18:11:35.550240 2024] [wsgi:error] [pid 234] [remote <IP_ADDRESS>:46516]     import gil_load; gil_load.init(); gil_load.test(); gil_load.start()  # provides data for gil-watcher
[Fri Mar 08 18:11:35.550763 2024] [wsgi:error] [pid 234] [remote <IP_ADDRESS>:46516]   File "gil_load/gil_load.pyx", line 483, in gil_load.gil_load.init
[Fri Mar 08 18:11:35.551224 2024] [wsgi:error] [pid 234] [remote <IP_ADDRESS>:46516] RuntimeError: gil_load.init() must be called prior to other threads being started

Here's the changes I've made to our application.
Added this to /etc/httpd/bin/apachectl below the section mentioning /etc/httpd/bin/envvars
(and before $HTTP is executed)
# Needed for gil_load
export LD_PRELOAD="/var/venvs/3.9.16/lib/python3.9/site-packages/gil_load/preload.cpython-39-x86_64-linux-gnu.so"

Put this at the top of the file referred to by the WSGIScriptAlias in apache's configs, before the wsgi application is imported & setup.
import gil_load; gil_load.init(); gil_load.test(); gil_load.start()

Here's the relevant section from apache's configuration.
  WSGIApplicationGroup %{GLOBAL}
  WSGIDaemonProcess learn display-name=w:learn group=apache user=some_user
  WSGIProcessGroup learn
  WSGIPassAuthorization On
  WSGIScriptReloading On
  WSGIScriptAlias / "/path/to/edde_learn.wsgi"

I believe this will get us one process with 15 threads according to the docs for the WSGIDaemonProcess directive here:
https://modwsgi.readthedocs.io/en/master/configuration-directives/WSGIDaemonProcess.html
We're using mod_wsgi==4.9.0.

It seems to me that gil_load.init() may be getting called multiple times in different threads. It's important it only gets called once at interpreter startup. Is there a place you can put Python code that runs before mod_wsgi creates its 15 threads?

Not to my knowledge, but I'll do a little more spelunking in the docs for mod_wsgi.


makeimage.sh fails: cannot retrieve file from server (libiconv)

I was trying today makeimage.sh for target=linux64, variant=nonfree. It built the image for about 40 minutes, then stalled at:
=> [layer-20-libiconv 1/1] RUN --mount=src=scripts.d/20-libiconv.sh,dst=/stage.sh run_stage /stage.sh
=> => # cd045e114e96a9f00483
=> => # Retry 2...
=> => # error: Server does not allow request for unadvertised object e46dee2f581c1167137bcd045e114e96a9f00483
=> => # ++ shuf -i 5-90 -n 1
=> => # + sleep 80
=> => # + echo 'Retry 6...'
=> => # + [[ 6 -lt 10 ]]
=> => # + timeout 480 git -C iconv fetch --depth=1 origin e46dee2f581c1167137bcd045e114e96a9f00483
=> => # error: Server does not allow request for unadvertised object e46dee2f581c1167137bcd045e114e96a9f00483
=> => # ++ shuf -i 5-90 -n 1
=> => # + sleep 85
=> => # + echo 'Retry 7...'
=> => # + [[ 7 -lt 10 ]]
=> => # + timeout 540 git -C iconv fetch --depth=1 origin e46dee2f581c1167137bcd045e114e96a9f00483
=> => # error: Server does not allow request for unadvertised object e46dee2f581c1167137bcd045e114e96a9f00483
=> => # ++ shuf -i 5-90 -n 1
=> => # + sleep 41
eventually it fails with
#0 619.8 Retry 10...
#0 619.8 + [[ 10 -ge 10 ]]
#0 619.8 + echo 'Max retry count exceeded.'
#0 619.8 Max retry count exceeded.
#0 619.8 + exit 1

Their git server is ridiculously broken. Every time they update, it forgets about the history somehow...

Yes, thank you, got over that step now. :)

Still broken, see:
#0 422.2 + timeout 600 git -C iconv fetch --depth=1 origin 0a05ca75c08ae899f6fca5f79254491e13ffb500
#0 424.0 error: Server does not allow request for unadvertised object 0a05ca75c08ae899f6fca5f79254491e13ffb500
#0 424.0 ++ shuf -i 5-90 -n 1
#0 424.0 + sleep 7
#0 431.0 Retry 9...
#0 431.0 + RETRY_COUNTER=9
#0 431.0 + CUR_TIMEOUT=660
#0 431.0 + echo 'Retry 9...'
#0 431.0 + [[ 9 -lt 10 ]]
#0 431.0 + timeout 660 git -C iconv fetch --depth=1 origin 0a05ca75c08ae899f6fca5f79254491e13ffb500
#0 432.7 error: Server does not allow request for unadvertised object 0a05ca75c08ae899f6fca5f79254491e13ffb500
#0 432.7 ++ shuf -i 5-90 -n 1
#0 432.7 + sleep 39
#0 471.7 + RETRY_COUNTER=10
#0 471.7 + CUR_TIMEOUT=720
#0 471.7 + echo 'Retry 10...'
#0 471.7 + [[ 10 -lt 10 ]]
#0 471.7 + [[ 10 -ge 10 ]]
#0 471.7 + echo 'Max retry count exceeded.'
#0 471.7 + exit 1
#0 471.7 Retry 10...
#0 471.7 Max retry count exceeded.
------
Dockerfile:11
--------------------
   9 |     COPY --from=layer-10-xorg-macros $FFBUILD_PREFIX/share/aclocal/. /usr/share/aclocal
  10 |     FROM layer-10 AS layer-20-libiconv
  11 | >>> RUN --mount=src=scripts.d/20-libiconv.sh,dst=/stage.sh run_stage /stage.sh
  12 |     FROM layer-10 AS layer-20-libxml2
  13 |     RUN --mount=src=scripts.d/20-libxml2.sh,dst=/stage.sh run_stage /stage.sh
--------------------
ERROR: failed to solve: process "/bin/sh -c run_stage /stage.sh" did not complete successfully: exit code: 1
+ rm -f /tmp/tmp.VfacDGwL2z.toml
+ docker container stop ffbuildreg
ffbuildreg

is there another mirror that hosts the libiconv package?

None that I'm aware of.
Their official git is the only source, and it's super broken. You can only fetch the latest commit, not any older one...


Question

Hi
I have a question. If it is possible:
How can I use
"https://github.com/mesmotronic/air-ane-androidid" inside Adobe flash?

If you mean as part of a SWF, then no, ANEs can only be used with Adobe AIR.
If you're asking how to add an ANE to an Adobe Flash/Animate project, see https://www.youtube.com/watch?v=ik-vfeqzToY.

OK, I will see it but in this link: https://github.com/mesmotronic/air-ane-androidid there are several folders and files, Do we need them? and I can't see the ANE file

https://github.com/mesmotronic/air-ane-androidid/releases

Special thanks for your support. I don't know where is the problem! Can you add .fla Source that contains all code? I will add ANE library.

This is my fla file
yyaaa3.zip

Please refer to the code code example in the README.md file.

I work with adobe flash and adobe animate:
Please check these steps and tell me that is something wrong:
---->For this line:
"Enables Adobe AIR 3.1+ for Android to access ANDROID_ID. That's it."
I can choose Air 23 for target.
---->For this:
Adds about 2KB to your app. Released under BSD license.
app.xml
com.mesmotronic.ane.androidid
I created a XML file and added above content inside of it. In adobe flash "purplish setting" ---> "General" tab I included my XML file.
---->Also I add ANE file as you showed in your video (YouTube)
-----> Last step I added following code:
import com.mesmotronic.ane.AndroidID;
trace(AndroidID.ANDROID_ID);
--------------------------------------------------------That 's it---------------------------------

Neilrackett
I 'm waiting for your response....
Thanks in advance.

Please ensure that you are using version 1.0.0 of the ANE and not the prerelease version 1.0.64, unless you are already using Adobe AIR 33 SDK.
Beyond that, we are not able to offer any further technical support regarding integration with Adobe Flash / Animate.

Thanks a lot for your response.
Where can I read version  of the ANE?


Vkontakte provider and example of usage

Vkontakte.  vk.com  is a most popular social network in Russia and countries of post USSR.
More than 340 millions users.

What do think about my pull request?

Hi! thanks, its a awesome works :)

fixed

Did you check my fix ?

Hello, I need this changes  for my site. I want to delete providers package from my code and use it from your lib. Where can you accept them and merge?

Hi, what about my pull request?


Add AspectRatioContainer class

3.2 version of #43807.

CI fails https://github.com/godotengine/godot/pull/45129/checks?check_run_id=1688580627#step:10:955, not sure how to fix because those come from GDCLASS macro expansion...

CI fails https://github.com/godotengine/godot/pull/45129/checks?check_run_id=1688580627#step:10:955, not sure how to fix because those come from GDCLASS macro expansion...

not sure how to fix because those come from GDCLASS macro expansion...

Remove override from virtual Size2 get_minimum_size() const override; (aspect_ratio_container.h line 42), it won't complain when there are no overrides at all.

not sure how to fix because those come from GDCLASS macro expansion...

Remove override from virtual Size2 get_minimum_size() const override; (aspect_ratio_container.h line 42), it won't complain when there are no overrides at all.

Thanks!

Thanks!


[FEATURE REQUEST] Add Optional Trend Line to Balance Over Time

Is your feature request related to a problem? Please describe.
I would like to see the trend over my balance over time. Example, I have 2 CCs with debit. I am concentrating on one CC while the other is getting little attention. I also cheating the system a little and did a balance transfer on the cards to get zero interest for some time. It is hard to visualize the debt progress since the lines are a little all over the place.
Describe the solution you'd like
I would like a trend line (that can be turned on and off) to help visualize the overall progress.
Describe alternatives you've considered
I am not sure I have one.
Additional context
N/A

This would be a nice addition.
Perhaps trend lines could be turned on and off on a per-account basis with checkboxes placed in the account legend. A "check all" checkbox of some sort could also be implemented if the user wants all accounts to have trend lines.

This has been implemented!

Looks great, thank you!


middleware/file: do additional section processing for SRV request

We don't add the IP address we can find to the additional section.

fixed.


[Feature]Microwave oven mode cluster implementation

Microwave oven mode cluster implementation

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.midea seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


chore: support icon class component

closing #39

hi @efec89, could you test this PR if it fixes your issue.
yarn add ssh://git@github.com:gorhom/react-native-animated-tabbar#feature/support-icon-class-component

hi @efec89 , could post your implementation here, i'll try to reproduce it

@gorhom  I refresh project and works. I think cached. You can confirm this build to others.

@efec89 good to hear 👍


[admin]merge to live

Docs Build status updates of commit 322d21c:
:warning: Validation status: warnings

File
Status
Preview URL
Details

Skype/SkypeURIs/SkypeURItutorial_Webpages.md
:warning:Warning

Details

Skype/UCWA/SkypeForBusinessOnlineScopePermissions.md
:warning:Warning

Details

Skype/SkypeURIs/SkypeURItutorial_Webpages.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.
Line 45, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```).

Skype/UCWA/SkypeForBusinessOnlineScopePermissions.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 40, Column 1: [Warning-hard-coded-locale] Link 'https://msdn.microsoft.com/en-us/skype/websdk/docs/developwebsdkappsforsfbonline#tenant-administrator-consent-flow' contains locale code 'en-us'. For localizability, remove 'en-us' from links to most Microsoft sites.
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 322d21c:
:warning: Validation status: warnings

File
Status
Preview URL
Details

Skype/SkypeURIs/SkypeURItutorial_Webpages.md
:warning:Warning

Details

Skype/UCWA/SkypeForBusinessOnlineScopePermissions.md
:warning:Warning

Details

Skype/SkypeURIs/SkypeURItutorial_Webpages.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.
Line 45, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```).

Skype/UCWA/SkypeForBusinessOnlineScopePermissions.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 40, Column 1: [Warning-hard-coded-locale] Link 'https://msdn.microsoft.com/en-us/skype/websdk/docs/developwebsdkappsforsfbonline#tenant-administrator-consent-flow' contains locale code 'en-us'. For localizability, remove 'en-us' from links to most Microsoft sites.
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 6e0a7dc:
:warning: Validation status: warnings

File
Status
Preview URL
Details

Skype/SkypeURIs/SkypeURItutorial_Webpages.md
:warning:Warning

Details

Skype/UCWA/SkypeForBusinessOnlineScopePermissions.md
:warning:Warning

Details

Skype/SkypeURIs/SkypeURItutorial_Webpages.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.
Line 45, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```).

Skype/UCWA/SkypeForBusinessOnlineScopePermissions.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 40, Column 1: [Warning-hard-coded-locale] Link 'https://msdn.microsoft.com/en-us/skype/websdk/docs/developwebsdkappsforsfbonline#tenant-administrator-consent-flow' contains locale code 'en-us'. For localizability, remove 'en-us' from links to most Microsoft sites.
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 6e0a7dc:
:warning: Validation status: warnings

File
Status
Preview URL
Details

Skype/SkypeURIs/SkypeURItutorial_Webpages.md
:warning:Warning

Details

Skype/UCWA/SkypeForBusinessOnlineScopePermissions.md
:warning:Warning

Details

Skype/SkypeURIs/SkypeURItutorial_Webpages.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.
Line 45, Column 1: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```).

Skype/UCWA/SkypeForBusinessOnlineScopePermissions.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 40, Column 1: [Warning-hard-coded-locale] Link 'https://msdn.microsoft.com/en-us/skype/websdk/docs/developwebsdkappsforsfbonline#tenant-administrator-consent-flow' contains locale code 'en-us'. For localizability, remove 'en-us' from links to most Microsoft sites.
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 1a9aada:
:warning: Validation status: warnings

File
Status
Preview URL
Details

Skype/SkypeURIs/SkypeURItutorial_Webpages.md
:warning:Warning

Details

Skype/UCWA/SkypeForBusinessOnlineScopePermissions.md
:warning:Warning

Details

Skype/AppSDK/SkypeAppSDK.md
:bulb:Suggestion

Details

Skype/SkypeURIs/SkypeURItutorial_Webpages.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 45, Column 1: [Warning-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```).
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.

Skype/UCWA/SkypeForBusinessOnlineScopePermissions.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 40, Column 220: [Warning-hard-coded-locale] Link 'https://msdn.microsoft.com/en-us/skype/websdk/docs/developwebsdkappsforsfbonline#tenant-administrator-consent-flow' contains locale code 'en-us'. For localizability, remove 'en-us' from links to most Microsoft sites.
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.

Skype/AppSDK/SkypeAppSDK.md

Line 38, Column 2: [Suggestion-column-header-missing] Column headers are required for tables. Add appropriate header text, or remove any extra table cells that cause an extra column.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 5d72460:
:warning: Validation status: warnings

File
Status
Preview URL
Details

Skype/SkypeURIs/SkypeURITutorial_iOSApps.md
:warning:Warning

Details

Skype/SkypeURIs/SkypeURItutorial_Webpages.md
:warning:Warning

Details

Skype/UCWA/SkypeForBusinessOnlineScopePermissions.md
:warning:Warning

Details

Skype/AppSDK/SkypeAppSDK.md
:bulb:Suggestion

Details

Skype/SkypeURIs/SkypeURITutorial_iOSApps.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.

Skype/SkypeURIs/SkypeURItutorial_Webpages.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 45, Column 1: [Warning-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```).
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.

Skype/UCWA/SkypeForBusinessOnlineScopePermissions.md

Line 1, Column 1: [Warning-title-missing] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 40, Column 220: [Warning-hard-coded-locale] Link 'https://msdn.microsoft.com/en-us/skype/websdk/docs/developwebsdkappsforsfbonline#tenant-administrator-consent-flow' contains locale code 'en-us'. For localizability, remove 'en-us' from links to most Microsoft sites.
Line 1, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.

Skype/AppSDK/SkypeAppSDK.md

Line 38, Column 2: [Suggestion-column-header-missing] Column headers are required for tables. Add appropriate header text, or remove any extra table cells that cause an extra column.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Bugfix: Scala 3 enums (#356)

Adds support for scala 3 enumerations which reads/writes them as their canonical string representations (used by the builtin "valueOf" method).

@jodersky not sure if I could bother you to look at this one as well :)

Moved to #378. @BoopBoopBeepBoop, feel free to open a new PR if you have more amendments to the updated PR.


handle ocp console disabled scenario [2.6]

Signed-off-by: Erin Murphy<EMAIL_ADDRESS>issue: https://github.com/stolostron/backlog/issues/25561

/lgtm


Bump treesitter and rules_nodejs

DISCLAIMER: I know nothing about bazel. This was just trial and error.

Thanks so much for doing this upgrade! I'll have a look later today :)

It looks like the tests are failing. You should be able to run them locally with:
bazelisk test --strategy=standalone --build_tests_only //tests

Hi, I'd love to see this merged since rules_tree_sitter currently don't work on darwin-arm64 or linux-arm64. What would be required to do that?
I have verified that reverting changes in the WORKSPACE file, but setting node_version to be at least 16.0.0 makes tests pass. 16.0.0 or higher is required because that is the first nodejs version with a darwin-arm64 build.

Hi, I'd love to see this merged since rules_tree_sitter currently don't work on darwin-arm64 or linux-arm64. What would be required to do that? I have verified that reverting changes in the WORKSPACE file, but setting node_version to be at least 16.0.0 makes tests pass locally. 16.0.0 or higher is required because that is the first nodejs version with a darwin-arm64 build.

Thanks for checking this out! I'd be happy to merge this PR with fixes. Would you like to clone the branch and submit a new one? It would be great if we could retain @Maan2003's contributions as well.


Update QA Process UPDATE_ECMPS_STATUS_FOR_TEE_EVALUATION.

[ ] Use the existing UPDATE_ECMPS_STATUS_FOR_MP_EVALUATION (the MP Evaluation version) as a model.

[ ] Any code that relates to emissions will need to be noted in a comment, but left out until emissions are worked on.

Finished and ready for test.


Migrates tests from Boost.Test to Catch framework.

Move away from Boost.Test; use the much nicer Catch instead.

You have mentioned BDD... I have created a tool -- see https://github.com/pepr/BDDtool . Written in Python. I did not finish the second part of my goals -- it requires more work.
I was just learning what it really means, why other people created the tools like cucumber. In my opinion, some of the other tools are too much clever. My goal was just to convert the test description in features/ to the Catch skeletons in tests/. I did not used it for the "real" development; anyway, it seems to be nice approach. Similarly to TDD, it forces you to think more before you really implement.
The second goal (not implemented) was to synchronize changes from the test files backward to the .feature files.
The next goal was to rewrite the tool from Python to C++ -- I guess it would be much more acceptable by others.
If you find it interesting, feel free to ask. If you have friends who may be interested, tell them. Otherwise, the project is rather dormant.
Have a nice day!

:+1:

@lexicalunit Before releasing next nanodbc version, you may want to wait until Clara has been fixed (see my PR I submitted to Clara and auto-linked above).

Sounds good! :+1:


Travis: update Ruby versions

This is a maintenance PR.

Rubies matching ruby-build versions
Avoid deprecated should in RSpec
Rubygems update, to be able to install Rainbow
Pass lint

👋 @jkeiser, it's Friday!

Thanks


web: restore old path prefix behavior

Signed-off-by: Fabian Reinartz<EMAIL_ADDRESS>@brian-brazil
@beorn7 I believe your issue in #4004 was not a bug in that sense. --web.route-prefix sets a prefix for the internal HTTP router, while --web.external-url allows defining a prefix path that is exposed to the outside world.
The intention being that different kinds of reverse prefix may or may not strip path prefixes away. Thus we need support to configure both things independently.

I think the confusion is that if route prefix is unset and the external url path is set, then route prefix defaults to that path of the external url - I think that’s kind of confusing. I think the next time we do a major release we should change this behavior and not make one flag (sometimes) depend on the value of the other.

:+1:
The behaviour of these flags is a result of long long discussions over years, I don't think we should change them lightly.

Yes, I think the intent was to do the least surprising thing for the user. But what is surprising probably also depends on what kind of reverse proxy they are dealing with, so that was possibly not the right choice.

Perhaps the meaning of the flags can be explained better in their docstring then? People seem very confused about them in general.


The 'Timestamp' and custom property columns will display blank on the 'Column Options' dialog if delete all entities then manually refresh the editor

Storage Explorer Version: 1.25.0
Build Number: 20220803.8
Branch: rel/1.25.0
Platform/OS: Windows 10/Linux Ubuntu 22.04/MacOS Monterey 12.4 (Apple M1 Pro)
Architecture: ia32/x64
How Found: Ad-hoc testing
Regression From: Previous release (1.24.3)
Steps to Reproduce

Expand one storage account -> Tables.
Create a table -> Add one entity.

Delete the entity -> Click 'Refresh'.
Open the 'Column Options' dialog.
Check whether all columns display well.

Expected Experience
All columns display well.

Actual Experience
The 'Timestamp' column displays blank.

Additional Context
The 'Timestamp' column disappears when reopening the table. (The same behavior as the previous release)

Verified this issue on the build main/20220926.1. Fixed.


interface name start with I

all interfaces name start with I in demo

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: raitalharehman sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


golang: upgrade to golang 1.20

Description
upgrade golang 1.20 as the latest go-toolset is 1.20.
Type of change
What is it?

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[X] Documentation update
[ ] Tests update
[ ] Refactor

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[X] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[X] New and existing unit tests pass locally with my changes
[X] I run make pre-commit to check fmt/vet/lint/test-no-fdo

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (8dbb92b) 62.13% compared to head (c54ff6e) 62.11%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2367      +/-   ##
==========================================
- Coverage   62.13%   62.11%   -0.03%
==========================================
  Files          88       88
  Lines       13232    13232
==========================================
- Hits         8222     8219       -3
- Misses       4334     4336       +2
- Partials      676      677       +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Automatically generate typing stubs

This PR enables generating typing stubs for cmdc automatically.
This is still WIP and I'm not really happy with the organization of everything yet, I'm open to any ideas there.
To generate the stubs:

run mayapy scripts/generate_typing_stubs.py
cmdc needs to be available in the PYTHONPATH
pybind11-stubgen needs to be installed in mayapy

Some notes:

I've had to declare all the classes before they're used anywhere to Avoiding having C++ types in the docstrings, this caused syntax errors and messed up the stubs generations.
I've put that the forward declarations in ForwardDeclarations.inl. I don't know if there's a better way to do that.
I haven't automated the generation when building yet because I only have a Windows machine and don't know PowerShell at all.
The stubs are saved in the build directory, not sure if that's where they should be saved.
Some of the docstrings have typing information, I'm not sure if they're still relevant but I've left them for now in case they were.
pybind11-stubgen does take those into account and some weren't valid ((string, string, ...) instead of List[str] ) so I've updated those.

@mottosso I feel like we're getting there, I'm just not sure how to make the CICD work now

Maya isn't available to CI at the moment, only regular Python with the Maya devkit. Do we need it? We'll need it to run the tests and I've been meaning to add it anyway. Could get that underway this weekend.

Ah I assumed it was, that tracks.
Pybind11-stubgen needs to import cmdc and as far as I know that can only work from mayapy

Cool, I'll add it to CI this weekend, then you can merge that into here to test it out.

Done. It's all yours.

I've just added pybind11-stubgen in the "Setup Test Environment" part, I expect it to fail on everything but 2022, pybind11-stubgen seems to only be able to run from python3

I don't have a Linux machine at hand to set that up though

mayapy is only available to GitHub on Linux, so if that's required then it'll only ever work on Linux.
A Linux user could be forgiven for not having access to Windows, but Windows users these days have no excuse haha. xD You've got both WSL, Docker, Virtual Box and VMWare Player. All free options.
In my case, I do this on Windows.
# From PowerShell
cd cmdc
docker run -ti --rm -v ${pwd}:/workdir mottosso/maya:2020
cd /workdir

From there you could literally run what the CI is running to recreate it verbatim.
# Setup Build Environment
yum install centos-release-scl -y
yum install devtoolset-7 bc -y

# Build
export DEVKIT_LOCATION=$(pwd)/devkitBase
. scl_source enable devtoolset-7 || true
g++ --version
chmod +x ./build_linux.sh
./build_linux.sh 2020

When you exit, the environment is gone. To iterate, you could chuck all commands into a Dockerfile like this one and build it once.
cd mycontainer
echo "My Container" > Dockerfile
docker build -t mycontainer .
docker run -ti --rm -v ${pwd}/workdir mycontainer
$ # Same environment each time

Alternatively, you could keep updating that yml file for GitHub actions to run things. I expect the results to be the same across Windows and Linux, so you could keep testing on Windows and have it built on Linux.

I'm not familiar with github actions so if you have any recommendations on what should be done, I'm all ear

For completeness, anything under steps is just a series of commands run in bash one-by-one, in the order they are written. There are a few keywords like ${{ matrix.maya }} in there that should hopefully make sense, but if not don't worry about it I don't think you need to use or make any of those. The sections like "Setup Test Environment" is just to make it more readable in the web UI.

A Linux user could be forgiven for not having access to Windows, but Windows users these days have no excuse haha. xD You've got both WSL, Docker, Virtual Box and VMWare Player. All free options.

Haha I haven't thought of that. I've been meaning to learn Docker for a while, that's a good opportunity to do that
Thanks for all the info, that'll be helpful 👍

Once you pop, you can't stop. :) Docker is a game changer. That command for mayapy above can also be used to spawn any of the last releases of Maya of the past 8 years.
Try some more "tags" from here: https://github.com/mottosso/docker-maya

So I looked a bit more into this today, I'm not 100% when we should generate the stubs in the process.
I have two questions:

Is it ok to only build the stubs with Maya 2022?
pybind11-stubgen only works with python 3. If we don't expect cmdc to have any differences between Maya versions, I don't see a problem there.
If we do, we could modify pybind11-stubgen a bit to make it work with python 2 (at first glance, that seems doable)

Should the stubs generation be a separate job that's run after the build jobs?
That would give us the flexibility of generating the stubs with only some Maya versions and not all the ones listed in the build jobs.

One note on the generated stubs:
They look mostly good but a lot of arguments end up named arg0, arg1, etc. instead of their actual name.
This comes from pybind11, not the stubs generation itself. pybind11 inserts a function signature at the top of the docstring and that signature has the misnamed arguments.
I think only the arguments with a default value are named properly.
If that issue rings a bell let me know, otherwise I'll have a deeper look later.

Is it ok to only build the stubs with Maya 2022?

I should think so. Maya 2022 would have everything past versions have, so if anything 2022 is the ideal choice. Although, maybe it would also give you hints about arguments that only exists since 2022, such as MDagModifier.deleteNode(includeParents=True)?
If Python 3 is what it needs, then it is how it is. Unless it doesn't need mayapy at all?

Should the stubs generation be a separate job that's run after the build jobs?

I would have thought they could run right after the Python tests, like just another set of commands? The trouble with giving them their own job is that we then need to pass the build artifacts from one job to another along with setting up the dependency. Jobs are basically running on different machines.
How about just another block following this one?
https://github.com/mottosso/cmdc/blob/8345f7960c9c6d555c25de89651ccb42a63ac1cd/.github/workflows/main.yml#L165-L171

They look mostly good but a lot of arguments end up named arg0, arg1, etc. instead of their actual name.

Sounds bad, but maybe it's good enough for a first pass? This kind of thing doesn't seem to care much for backwards compatibility. Would like to merge this PR to take many small steps soon in favour of one large perfect step later.

If Python 3 is what it needs, then it is how it is. Unless it doesn't need mayapy at all?

pybind11-stubgen uses the inspect module to parse everything so it needs to be able to import cmdc. I don't think we can do that without mayapy?

I would have thought they could run right after the Python tests, like just another set of commands?

Wouldn't that make it run on every maya version in the maya-linux job?
If there's a way to conditionally run commands by checking the current {{ matrix.maya }} or something, I'm happy to do that yeah

Sounds bad, but maybe it's good enough for a first pass?

Yeah agreed. I should have something ready by next weekend I think.

If there's a way to conditionally run commands by checking the current {{ matrix.maya }} or something, I'm happy to do that yeah

There is yeah, there's another step with one you can look at for reference.

https://github.com/mottosso/cmdc/blob/master/.github/workflows/main.yml#L225

Next step is looking at the GitHub Actions docs.

pybind11-stubgen uses the inspect module to parse everything so it needs to be able to import cmdc. I don't think we can do that without mayapy?

I'm not sure. :S It's perfectly capable of building cmdc without mayapy, so it seems logical that we'd be able to inspect the results without it. But what it's asking for are some of the DLLs that ship with Maya..

So then at least, if necessary, we could just bundle these with this repo and avoid having to call into Maya at all.

Ah, looks like I didn't write the condition to generate the stubs properly haha
@mottosso do you know if there's a good way to test the CI configuration locally instead of having to push all the time to test things?

There's one very annoying with pybind11-stubgen, it can completely freeze in some cases
So far I suspect it comes from docstrings containing an invalid signature. The solution is to just remove the signature from the docstring
Some function stubs result in something like: my_func(*args, **kwargs) -> Any
This happens when pybind11 generates a docstring with a syntax error in the signature
pybind11 gives this warning:
Generated stubs signature is degraded to `(*args, **kwargs) -> typing.Any` for
def objectColorRGB(self: cmdc.FnDagNode) -> Autodesk::Maya::OpenMaya20220000::MColor: ...
                                                     ^-- Invalid syntax

In most this is because the return (or argument) type is not declared as a binding before. Declaring it in ForwardDeclarations.inl fixes this.
There's also this issue :
Generated stubs signature is degraded to `(*args, **kwargs) -> typing.Any` for
def create(self: cmdc.FnDagNode, type: cmdc.TypeId, parent: cmdc.Object = <cmdc.Object(kInvalid)>) -> cmdc.Object: ...
                                                                          ^-- Invalid syntax

In this case parent has a default value of MObject::kNullObj. Pybind uses the value's __repr__ in the signature which isn't valid.
It's possible to override this manually by doing something like this
On a final note, specifying the arguments with py::arg("my_arg") seems to fix the issue with the arguments named arg0, arg1, etc. in the stubs.

Also, I have absolutely no idea why the tests failed on Maya 2018 there.

I'll have a look at the crash yeah
I'm surprised it doesn't crash in master though, I don't think I've touched the implementation of anything, unless I've messed up the merge somehow, will dig more tonight 👍

Cool so this seems to work so far, I've uploaded the stubs along with the Maya 2022 module, meaning in the release they'll only be in the linux-2022 folder.
I'm guessing that'll work but I don't really know how to test the release job

There's a lot that can be done to improve the quality of the stubs, mainly adding py::arg and py::arg_v everywhere to let pybind know of the name of all the arguments like you've started doing
That's probably better suited for a second PR though so I'm happy to merge this one now

That Maya 2018 crash is also just gone and I haven't managed to reproduce it on my end

I'm guessing that'll work but I don't really know how to test the release job

Where do they need to be? Is it also on PYTHONPATH? Simplest thing you can try is downloading the latest release and putting your stubs wherever they belong and make sure it works wherever it needs to work. I've never used stubs so I'm not too familiar with what goes into it.
Come to think of it, they should appear in the next release too so let me merge and then you can continue experimenting with the real thing. Do we want these in any way related to Maya? Like, in Maya's PYTHONPATH too? Or are they completely separate from the cmdc .pyds? :S The canonical way would be to put them under..
~/maya
  /modules
    /cmdc
      /scripts
        /stubs-here.stub

But that would only make sense if Maya is consuming them. Anyway, I'll leave this to you!

Looks like they made it. https://github.com/mottosso/cmdc/releases/tag/v0.1.4
One option is simply copying the single pyi file to the other folders at release time, like I'm copying the .mod file currently, just before zipping them up for final upload.
https://github.com/mottosso/cmdc/blob/86cc176e5a717487f7de696fa718c9160797a61e/.github/workflows/main.yml#L250-L253

The stubs are never really useful in maya, they're meant for your editor to give you nice autocompletion, typing information, etc.
In vscode you'd add the folder where the stubs live to python.analysis.extraPaths
Once generated the stubs don't need the .pyd files at all, I think some libraries even just publish their stubs independently on pypi.
In our case, I feel like something along those lines would make sense
~/maya
  /modules
    /cmdc
      /stubs
        /cmdc.pyi

I'll start another PR that tackles all the ugly arg0 stuff soon and will change the location of the stub there as well. 👍

Sounds good!