pagination

Currently we don't have anything implemented to help users with pagination.   While we have nextlink users are still required to hack there way into it.
Example of my recent creative attempt:
var results = DataGaSample.get(service, "ga:78110423", "yesterday", "today", "ga:sessions");
        while (results != null)
        {
            foreach (var row in results.Rows) {
                Console.WriteLine(row[0] + " - " + row[1]);
            }
            results = results.getNextRows(service);
        }

public static class extensions
 {

    public static GaData getNextRows(this GaData data, AnalyticsService service)
    {
        Console.WriteLine(data);
        if (data != null)
        {
            if (data.NextLink == null)
            {
                return null;
            }
            var request = service.Data.Ga.Get(data.Query.Ids, data.Query.StartDate, data.Query.EndDate, data.Query.Metrics.ToString());
            request.Dimensions = data.Query.Dimensions;
            request.MaxResults = data.Query.MaxResults;
            request.StartIndex = data.Query.StartIndex + data.Query.MaxResults;
            return request.Execute();
        }
        return null;
    }
}

here is another example of the hacking pagination.
UrlShortener
I think it would be smarter to add an interface to the library.

I agree something as common as pagination would be a great thing to put in the code gen. I know that we (Google) are currently looking into specific client libraries on a per-API basis to make APIs easier to consume. But that won't help you if you aren't using that specific library.
Let's use this bug to track the codegen request.
/cc @jskeet

It's definitely my intention to write some general pagination support in a library which can be used from both these client libraries and ones using GRPC. That will help make it easy to write specific libraries (my immediate goal) but won't help on other libraries at the moment, as Chris said. We could certainly look at either duplicating that code within the existing base client library, or adding a dependency to the new client library, and then tweaking the existing codegen. I wouldn't like to promise anything at this stage, other than to say that we're aware there's a gap here, and that we're keen to make the experience of using Google APIs from C# as delightful as we can.

I am willing to add it to the Client library.  I consulted @peleyal about this before I posted this issue, and he suggested I create an issue so we can decide how to best go about it.
@jskeet  If you are willing to consult on the best way of going about it.  I would be thrilled to work with you on it.

Hi Linda, sorry for not getting back to you before this.
I've hardly looked at the code generator or runtime support for this at all. I suspect it makes sense to work on this entirely separately from the other efforts I'm looking at at the moment (and definitely not gate either one on the other) but try to make the public APIs exposed at least similar.
Are you happy to take the next steps in terms of proposing a strawman of what the extra generated code should look like, and we can take it from there?

As far as i know the code generator is Python, I don't know python nor do i have the time to learn python.
I was hoping this was something we could add to the client library directly.  I had the crazy idea that response could return a new request object.   that you could just call execute on.   I haven't been though the deep dark depths of the core library really.

We'll certainly need to add some code within the client library - but I'd expect to need to modify the code generator as well to expose new code to use it.
Rather than the application having to call GetNextRows or similar, I'd envisage having a lazy evaluator which just returned resources, making extra requests where necessary. So you could really just do something like:
foreach (var result in service.Books.ListAll().Execute())
{
    // This would get all the books the service knows about.
    // It could require multiple requests, but the caller doesn't care.
}

The caller could affect the pagination by making changes to the object returned by ListAll(), e.g. changing the results per page, the query etc... but after that, it would be transparent.
Do you think that's a reasonable goal, or do you think the pagination still needs to be explicit, just a bit easier?

oh i like your idea and that's exactly what I was looking for. I think it should just be easier.   IMO my start index hacks are messy and well hacks.  Being able to loop the requests to get all of the data available would be valuable.
My only concern would be someone doing that and returning 1m rows from an API.  However the question would be is it our problem if they blow out there quota or crash there connection due to requesting to much data.
Related to that how do we deal with say throttling.   That could run to fast and the user will get User Rate Limit Exceeded should we add exponential back off?   However if we do implement exponential back-off as recommend  if we add that they we are auto re-sending it and eating there quota.
Then there is my personal favorite 500: Backend Error. I normally just resend the request a few times waiting for the server hiccup to clear up.   Again if we do this we are eating there quota.
Eating there quota on a free API is one thing but users who have to pay for these extra requests should be aware of what the library is going to be doing.    (I wonder if they get charged for request that resulted in errors.)

I think we could add retry options, request limits, back-off options and a rate limit to the object returned by the ListAll method, potentially. I think it would be worth starting off with something dirt simple though :)

Point taken :)
I will get a simple version working.  We can decide what to do from there.

Just a little status:  I have never tried a lazy evaluator before.  I think this is the way to go.  I am still trying to get it working.

Good timing - I meant to update this issue to point at https://github.com/GoogleCloudPlatform/gcloud-dotnet/tree/master/Google.Apis.Common which now includes common pagination code.
I'm not sure whether we'd want to take a dependency on that right now (there's no NuGet package for it, to start with...) but if we expose APIs that are compatible with it, we can use it later.

OH thank you! There isn't a lot of documentation on this.   Having your code to reference will help a lot.

Oh, and look in the GoogleApiWrappers project for uses of the code :)

Any special reason we have a separate project for Google Cloud .net client?

Well, the idea is to host manually-written code for various APIs - have a look at the gcloud-node and gcloud-python peer projects for an idea. Final locations for various bits of code are still up in the air, but basically I'm committed to providing the best API client libraries I can :)

We should work well together then:
I'm committed to providing the best examples, samples and tutorials for Google API client libraries I can.

That's great to hear - we'll certainly want as much good documentation as we can. It's probably a bit early yet to go into too much detail, partly as there's still quite a lot of planning to do, but I'll definitely be asking for help in the future.

If you have time you are welcome to join the code review on my Generated samples project.
One pre-genrated sample was uploaded:  https://github.com/LindaLawton/Google-Dotnet-Samples/tree/GeneratedSample/Analyticsv3Sample   If you want another API let me know i have 152 of them generated.

I probably won't have time to do much in the way of code reviewing right now, but I love the aim of the repository :)

Note: I've renamed Paginator to PageStreamer as that makes more sense in terms of what it does. (It streams items that are already paginated, rather than breaking down a stream into pages...)
I'm still not 100% sure on what the project organization should end up looking like - we may want to basically clone this code into the Google .NET Client Library core package.

I have been playing with it.  Actually I have it working in my own library which works with Google Analytics only and supports .net 3.5.     I must admit that I have never made a lazy evaluator before when you see it working its pure genius.   Thanks for the idea :)
I am wondering if this is something that like you said should be added to the .dll generation application rather than to the core library.
If memory services not all of the APIs have Nextlink some have NextPage and some just have a start index.   (Again if memory services i will have to dig around in Discovery service a bit to see if i can verify this.)
Assuming that my research into proper application design is paying off.  If its not the same then the code should probably be placed inside of the dlls for each of the APIs, or we should have separate methods in core for the different APIs the code in the API dll detect which one it needs to use.
If possible i think cloning your code which we know works is probably the best way to go.  No reason to reinvent the wheel.
I haven't had the time to touch the code for this library.  I have a list of projects that need looking into I was hoping to do some of it over Vacation.   That didn't happen ... at all.

I'd rather not have the same code in lots of places - it's deliberately generic to avoid that :) Three options:

Include the PageStreamer in the core library, but not actually use it directly from any of the generated code - instead, just provide an example of how it can be used, where appropriate.
Make the code generator spot when it could make use of it, and expose a PageStreamer property or something similar within the generated resource code so that client code can use that
Make the code generator spot when it could make use of it, and expose different methods that return IEnumerable<Foo> appropriately.

The first is clearly simplest, as it doesn't require any codegen changes. With the second and third options we'd need to write the code to spot where the pattern is applicable, which is likely to be significantly trickier. (Generating the code to use it probably isn't too hard, but the "spotting" is trickier.)

I was thinking of the first option as well.   I can have the sample projects generate examples of how to use it.      We can look at the other options when we have more time.

Sounds good to me. No extra dependency necessary, adding convenience but without too much complexity.
Are you happy for me to create a pull request for this then? Basically it'll be a copy of the code I've already got, unless you have any suggested changes. (The code gcloud-dotnet will stay there but that project will probably become more GRPC-specific.)

be my guest.  I can add the example to the sample generation and test it.

Humbug - have just seen that we don't currently have a dependency on Ix-Async, which is where IAsyncEnumerable<T> (and supporting code) comes from.
We could just include the synchronous part for the moment, potentially. I'd rather not add a new dependency for this. Hmm.

// cc @chrsmith @mmdriley @ivannaranjo for thoughts on what to do in terms of async with the core library. (We could fairly easily provide a method returning a Task<List<T>>, as a starting point...)

Nothing stands out as obviously superior. But let me see if I understand the things correctly:

Problem
Pagination in the generated libraries is a pain. We should make this easier.
Solution
Jon proposed three options, each with their tradeoffs.

1.) Include the PageStreamer in the core library, but not actually use it directly from any of the generated code - instead, just provide an example of how it can be used, where appropriate.

This would unblock developers building against the code gen.

2.) Make the code generator spot when it could make use of it, and expose a PageStreamer property or something similar within the generated resource code so that client code can use that

Seems better, but we don't know how difficult it would be to detect when this would be the best spot to emit the code, right?

3.) Make the code generator spot when it could make use of it, and expose different methods that return IEnumerable appropriately.

This is the best option, since pagination is completely transparent and "just works". However, requires the biggest code gen work.

I personally don't like any of these. But if I had to choose one, option 1 seems reasonable. The idea of putting the code in a Google.Apis.Util namespace might help, and conceivably there are other cross-API functionality we would want to put there in the future.
If we live in a world where we have both the code gen and the hand-crafted wrapper, I don't think we would ever want 3.) in the code gen library however. That seems like it hides a few too many details. i.e. if the code gen is targeting low level bits and the hand-crafted wrapper provides abstractions, then we should double down on that conceptual layering.
Something we might want to discuss in person is what we want the long-long term solution for the code gen to be. Do we want to eventually have it produce code on par with the hand-crafted library? And obviate the need for it?

Seems to me that 1) is also the most complete solution, I am not sure that we can accomplish 2) or 3) without some attribute or another way of definitely marking when something is using pagination and how to integrate it with the PageStreamer class.
This can also be an incremental approach we start with 1) and can move to 2) or 3) later on, it seems to me that we can generate compatible changes that add the async properties/methods since they will have new names.

Sounds like everyone's at least somewhat comfortable with option 1.
What about the async aspect? Here there are options (labelled alphabetically to avoid colliding with the above!) of:
a. Don't include any async methods (ick)
b. Include a Task<IList<TResource>> FetchAllAsync method, and don't add any new dependencies
c. Add a dependency on Ix-Async and stick with the IAsyncEnumerable<TResource> FetchAsync method in the current library
I think we should rule out option a immediately - I only listed it for completeness, really.
I'm somewhat dependency-phobic, which is why I was suggesting option b. But if others are happy with us adding another dependency, option c would be the nicest for callers who actually use the PageStreamer. The downside is that it adds a dependency for everyone else as well.

I'd like the libraries to be as unopinionated as possible about their environment, so I also prefer avoiding a new dependency here.
FetchAllAsync is probably enough. If we're determined to be fancy, we can provide a method that returns an IObservable.

Right, the code is now in. I'd like to wait until we've got a good sample added to the class doc comment, and @LindaLawton has some samples committed before we close this issue though.
Linda, with your experience of the APIs, do you have any particularly good examples where this will be useful and easy to demonstrate?

@jskeet  off hand make a request to the Google Analytics API and set maxrows to 1.
Dimensions: Ga:date,ga:hour
metrics:   ga:users
start-date: 2015-01-01
end-date: 2015-01-01
maxrows: 1
Should result in a lot of data and a lot of next links.   Evil request really :)
If you want something thats public.   How about a search to YouTube.
var service = new YouTubeService(new BaseClientService.Initializer()
{
ApiKey = "A Key",
ApplicationName = "YouTube Authentication Sample",
});
           var request =  service.Search.List("snippet");
           request.Q = "teenage mutant ninja turtles";
           var x = request.Execute();

Both of those sound like good ideas - we'd probably want to see just how many results we get in each case if we want to show the async version as well (which will fetch everything...)

both responses contain totalResults.  easy enough to check that you have the correct number returned.
@jskeet Are you going to test this?  I don't have client library compile power from here. You could email me a dll (or nuget package) for Analytics and YouTube.  If you want me to test it for you.

Re checking: hmm... as the page streamer never gives you access to the immediate responses, that check could be tricky (unless you make the same request manually first...) And of course the number of results can change while you're paging through it.
You should be able to use the existing API-specific client library from Nuget - you'd only need a new version of the GoogleApis assembly. I'll see if I can get a Youtube example together myself (possibly not today) and we can go from there.

Ack Good point i was over thinking new dll for core would be enough.
We also need to be tested against an a request that doesn't contain next links.   Like a Get or delete request to ensure that it doesn't return an error if someone uses it incorrectly.  or at the very least returns a well formatted exception.   TBH should probably just do nothing but whatwhereyouthinking exception might work as well.

I think it would be hard to construct the PageStreamer in that case - or at least, you'd have to do so in a deliberately-weird way. Basically the code is agnostic to what you're really doing - it just knows how to take a request, modify it, execute it, extract resources, and create the next request from the page token.

We can write some good documentation to show how to use it then.   If it does return an error at least we can document the error and direct the users what the cause was.

My point is that you just wouldn't be able to write the wrong code easily, to construct the PageStreamer. Any errors are likely to be just normal compile-time errors.
If you can give an example of the kind of thing you mean, we can check that we're doing the best we can to prevent misuse.

Ok here is what i mean:
 var AnalyticsService = new AnalyticsService(new BaseClientService.Initializer()
        {
            ApiKey = "A key",
            ApplicationName = "Authentication Sample",
        });
var response = AnalyticsService.Metadata.Columns.List("ga").Execute();

the response body for that request does not contain nextlink
 {
   "kind": "analytics#columns",
   "etag": etag,
   "totalResults": integer,
   "attributeNames": [
     string
   ],
   "items": [
     metadata.columns Resource
   ]
 }

What is your PageStreamer going to do if i try and use it with that request?  If you try and read from next link it wont be there.   Are you going to get an error?

The PageStreamer doesn't do anything without being configured - so where in my example in the pull request I had:
private static readonly PageStreamer<Bucket, BucketsResource.ListRequest, Buckets, string> bucketPageStreamer =
    new PageStreamer<Bucket, BucketsResource.ListRequest, Buckets, string>(
        (request, token) => request.PageToken = token,
        buckets => buckets.NextPageToken,
        buckets => buckets.Items);

... you'd find it hard to do the equivalent with the AnalyticsService. Don't forget that PageStreamer isn't going to change the behaviour of existing code at all - you've got to opt into using it by creating an instance and then using that instance. If you find you haven't got the right properties when you try to construct the instance, that's a good indication that PageStreamer isn't suitable for you :)

Perfect. I look forward to seeing how it works and picking it apart I learn a lot from your code :)

Okay, here's a console example which works for me:
using System;

using Google.Apis.Requests;
using Google.Apis.YouTube.v3;
using Google.Apis.YouTube.v3.Data;
using Google.Apis.Services;

namespace YouTubeDemo
{
    class Program
    {
        static void Main(string[] args)
        {
            var pageStreamer = new PageStreamer<SearchResult, SearchResource.ListRequest, SearchListResponse, string>(
                (request, token) => request.PageToken = token,
                response => response.NextPageToken,
                response => response.Items);

            var service = new YouTubeService(new BaseClientService.Initializer()
            {
                ApplicationName = "YouTube Demo",
                ApiKey = API_KEY
            });
            var req = service.Search.List("snippet");
            req.Q = "using google apis";
            req.MaxResults = 5;

            foreach (var result in pageStreamer.Fetch(req))
            {
                Console.WriteLine(result.Snippet.Description);
            }
        }
    }
}

Will send out a PR to add the construction part into the XML doc comment for PageStreamer.

Is this part of the nuget packages?  I did a refresh and I didn't get it. I ended up downloading your PageStreamer and adding it to my project.    It works like a dream.
I am trying to figure out how to add it to the auto generated sample projects.
Here is an example for Drive V3 files.list
var pageStreamer2 = new PageStreamer<File, FilesResource.ListRequest, FileList, string>(
                                                    (request, token) => request.PageToken = token,
                                                    response => response.NextPageToken,
                                                    response => response.Files);
                foreach (var result in pageStreamer2.Fetch(service.Files.List()))
                {
                    Console.WriteLine(result.Name);
                }

It's not part of a public release yet, I'm afraid. Soon, very soon! :)

OK.   Do you have an opinion on how to add this to the sample projects?   We need a simple yet useful example.

Well, I wouldn't publish any new samples with it until the package is released. But then, I'd start off by looking at any samples which already have listing - I'd hope it would pretty much drop into place. I'd make the page streamer a static readonly field, to emphasize that it's not stateful. (That's the usage we expect in real code.)

I haven't actually released much of my generated sample projects yet.  They are still a work in progress.   I have had a few requests for Drive v3 so I have been tuning that.
Once I have a basic layout for the method I can genrate it on any method that has nextlinks in it without to much trouble.
Here is what i have so far:
/// <summary>
/// Name: File.List page streamer
/// Description: PageStreamer is helper method is used for looping over pages.  This helper method ///  will assist you in creating a PageStreamer for the File.list method
///
/// Usage example
/// var pageStreamer = GoogleSamples.Drive.v3.FilesSample.ListpageStreamer(service);
/// foreach (var result in pageStreamer.Fetch(service.Files.List()))
/// {
///     Console.WriteLine(result.Name);
/// }
/// </summary>
/// <returns>A page streamer for both synchronous and asynchronous page streaming of a listable or queryable resource.</returns>
public static PageStreamer<File, FilesResource.ListRequest, FileList, string> ListpageStreamer()
{
      try
      {
                return new PageStreamer<File, FilesResource.ListRequest, FileList, string>(
                                                    (request, token) => request.PageToken = token,
                                                    response => response.NextPageToken,
                                                    response => response.Files);
      }
      catch (Exception ex)
      {
                Console.WriteLine("Failed to create file.list pageStreamer " + ex.Message);
                throw new Exception("FilesListPageStreamerFailed", ex);
      }
}

I wouldn't bother with the method, to be honest - I'd just give an example of how to create it with the constructor, assigning it to a static field. You don't need to bother catching an exception, either. (Note that your method doesn't have a service parameter - which is deliberate and good. Otherwise it would be harder to construct statically.)

Just add usage at the top of my example for file.list maybe?

Possibly - I'm not sure how your samples are laid out at the moment. Any good examples that show your latest style?

I haven't released them because there are bugs and there are a few more things I want to add.  Like methods for usage of optional variables and enum support.
Email me I will send you what i built for Drive v3 or any other API you want i have 162 of them built just not verified.  This is all built using tt templates  I currently have one for resources and one for auth.
I would love your feed back.

Someone should close this.   I have a short tutorial on it.    Pagination Google .net Client library

Excellent - thanks Linda!

@jskeet Do we have a date for when this will be part of a nuget release?

@tstojecki: It's already in the latest NuGet release.
It's true that it only addresses certain use cases. As it happens, we've been thinking about this more broadly in the https://github.com/GoogleCloudPlatform/gcloud-dotnet world, designing an API which allows access to the underlying pages as well. It doesn't try to use the "retrieve a maximal number of records" approach you've described, however - the maximum page size varies by API, aside from anything else. Fundamentally a page token scheme isn't designed for arbitrarily going to "future" pages - but it can allow navigate forward/backward a page at a time by retaining page tokens.
I'm hoping to create a new project within this repo for page streaming for REST APIs, with an API to match the one for gRPC APIs; there's a prototype of that at https://github.com/GoogleCloudPlatform/gcloud-dotnet/pull/192
Note that this is a separate project so that it can depend on Ix-Async; we don't want to add a new dependency to Google.Apis or Google.Apis.Core. It also means we can version it separately, starting out with a prerelease version in NuGet so we can iterate. (As PageStreamer is now "in" Google.Apis, we can't make breaking changes to it. Adding to a released API is hard!)

the maximum page size varies by API

Clarification.   Max page size is varies by methods in APIs.  No real way of knowing what method has what max limit. Discovery does appear to tell us what the min limit is but there is no value for max unless you rip it out of some of the descriptions.

Maximum number of entries returned on one result page. By default the value is 100 entries. The page size can never be larger than 250 entries. Optional.

I have in the past hacked start-Index in the Google analytics API to allow for paging like you suggest going back and forth.   However i think it would be hard to get this to work for all of the APIs generically due to the fact not all of the APIs use start-index Google Calendar for example uses pagetoken.     For us to add something to the auto-generated API library's its going to have to be something that can be generated and used by all.
@jskeet  i look forward to checking out your idea. :+1:

Thanks @jskeet and @LindaLawton
Seems like for now it is going to be some sort hack along the lines of what I described. I too will keep on the work that is being done in google cloud platform.
I still think that this is something the rest api team could take into consideration. AFAIK the ability to specify the page and limit parameters is a fairly standard feature among the rest apis.


Add AD and Alerting to the manifest 2.0.0

Signed-off-by: Peter Zhu<EMAIL_ADDRESS>Description
Add AD and Alerting to the manifest 2.0.0
Issues Resolved
Part of #1624
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Hold this for a bit before merge.

Codecov Report

Merging #1878 (341670f) into main (15c4f5d) will increase coverage by 5.45%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##               main     #1878      +/-   ##
=============================================
+ Coverage     94.54%   100.00%   +5.45%
=============================================
  Files           178         6     -172
  Lines          3630       105    -3525
  Branches         27        19       -8
=============================================
- Hits           3432       105    -3327
+ Misses          194         0     -194
+ Partials          4         0       -4

Impacted Files
Coverage Δ

...RunBwcTestScript_OpenSearch_Dashboards_Jenkinsfile

src/test_workflow/perf_test/perf_test_suite.py

src/system/config_file.py

src/assemble_workflow/bundle_rpm.py

src/paths/tree_walker.py

src/test_workflow/test_cluster.py

...sts/jenkins/jobs/DetectTestDockerAgent_Jenkinsfile

src/manifests/input_manifests.py

src/system/process.py

src/ci_workflow/ci_check_gradle_dependencies.py

... and 162 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 15c4f5d...341670f. Read the comment docs.


No warn option for copy overwrite

Copy file to existing overwrites and there is no
"warn_overwrite_on_copy" option

I have just lost a file and i can't retrieve its content back


Add insert_with

Why not use the existing vacant_key or vacant_entry methods?

I don't have much experience with the slab crate. I just started using it and expected a insert_with to exist

Closures are generally more difficult to work with because they constrain control flow, and there's no widespread convention of a method with these semantics in Rust APIs. Do the existing APIs address your use case?

Yeah vacant entry would also do that in a more complicated way...

If you'd prefer simpler, vacant_key is also good. Sounds like this isn't needed.


mitmproxy: 7.0.4 -> 8.0.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://pypi.org/project/mitmproxy/.
meta.description for mitmproxy is: "Man-in-the-middle proxy"
meta.homepage for mitmproxy is: "https://mitmproxy.org/"
meta.changelog for mitmproxy is: ""
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
Warning: no invocation of /nix/store/wabq27br1y11rqvrmcl06nlwll0mszcr-python3.9-mitmproxy-8.0.0/bin/mitmdump had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/wabq27br1y11rqvrmcl06nlwll0mszcr-python3.9-mitmproxy-8.0.0/bin/.mitmdump-wrapped had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/wabq27br1y11rqvrmcl06nlwll0mszcr-python3.9-mitmproxy-8.0.0/bin/.mitmweb-wrapped had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/wabq27br1y11rqvrmcl06nlwll0mszcr-python3.9-mitmproxy-8.0.0/bin/mitmproxy had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/wabq27br1y11rqvrmcl06nlwll0mszcr-python3.9-mitmproxy-8.0.0/bin/mitmweb had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/wabq27br1y11rqvrmcl06nlwll0mszcr-python3.9-mitmproxy-8.0.0/bin/.mitmproxy-wrapped had a zero exit code or showed the expected version
0 of 6 passed binary check by having a zero exit code.
0 of 6 passed binary check by having the new version present in output.
found 8.0.0 with grep in /nix/store/wabq27br1y11rqvrmcl06nlwll0mszcr-python3.9-mitmproxy-8.0.0
found 8.0.0 in filename of file in /nix/store/wabq27br1y11rqvrmcl06nlwll0mszcr-python3.9-mitmproxy-8.0.0

Rebuild report (if merged into master) (click to expand)

6 total rebuild path(s)

3 package rebuild(s)

3 x86_64-linux rebuild(s)
3 x86_64-darwin rebuild(s)

First fifty rebuilds by attrpath
mitmproxy
python310Packages.mitmproxy
python39Packages.mitmproxy

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/wabq27br1y11rqvrmcl06nlwll0mszcr-python3.9-mitmproxy-8.0.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A mitmproxy https://github.com/r-ryantm/nixpkgs/archive/eebff7472936aecb1231632c73e2d5d43c2ad315.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/wabq27br1y11rqvrmcl06nlwll0mszcr-python3.9-mitmproxy-8.0.0
ls -la /nix/store/wabq27br1y11rqvrmcl06nlwll0mszcr-python3.9-mitmproxy-8.0.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

2 packages built:

mitmproxy (python39Packages.mitmproxy)
python310Packages.mitmproxy

Maintainer pings
cc @fpletz @kamilchm for testing.

Re-enabled tests on darwin as it WFM, macos 10.15.


Carry 19733: Respond with 401 when there is an unauthorized error from the registry.

Fixes #19523
Signed-off-by: David Calavera<EMAIL_ADDRESS>

LGTM

Thanks @stevvooe :heartbeat: LGTM too.

LGTM

Apparently @cpuguy83 and @icecrime have issues when testing this, so dont merge yet.

All tests I've been running against Hub for private images and missing images have been consistently returning a 200. What am I doing wrong?

Well, CI seems to agree here:
01:21:17 ----------------------------------------------------------------------
01:21:17 FAIL: docker_cli_images_test.go:294: DockerSuite.TestImagePullWrongCreds
01:21:17
01:21:17 docker_cli_images_test.go:310:
01:21:17     c.Assert(out, checker.Contains, "401 Unauthorized")
01:21:17 ... obtained string = "" +
01:21:17 ...     "Using default tag: latest\n" +
01:21:17 ...     "Pulling repository <IP_ADDRESS>:5000/dockercli/busybox\n" +
01:21:17 ...     "Error while pulling image: Get http://<IP_ADDRESS>:5000/v1/repositories/dockercli/busybox/images: dial tcp <IP_ADDRESS>:5000: getsockopt: connection refused\n"
01:21:17 ... substring string = "401 Unauthorized"
01:21:17
01:21:18
01:21:18 ----------------------------------------------------------------------

@aaronlehmann it looks like the authorization error is being masked by a fallback error, but we don't understand why that logic is there in the first place:
https://github.com/docker/docker/blob/master/registry/registry.go#L176
Can you take a look?

On 401, we allow fallback to the v1 protocol. I think one of the reasons for this is that the v2 registry returns 401 in all cases when the token doesn't grant access to a repository, including nonexistent repositories. So a public repository that's only available over the v1 protocol would get a 401 status code from the v2 registry. This is tracked in https://github.com/docker/distribution/issues/1291.
What complicates this is that the v1 registry returns a 404 when there are insufficient creds, so that's the error that's eventually seen by the logic in this PR.
Any idea how this worked in previous versions? I'm pretty sure we've always fallen back in the 401 case.

Okay, it looks like there are two distinct cases:

Pulling a repo that you don't have access to. I don't think this has ever prompted for creds. It invokes a v1 fallback, and the ultimate error code is a 404.
Trying to pull any repo, including a public one, with a bad login or password. This used to prompt, but doesn't in 1.10. So this should be considered the regression.

This PR should be able to fix (2), but (1) is outside the scope IMHO.
However, I just tested it, and I don't get a login prompt when I have a bad password:
# docker pull busybox
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/library/busybox/manifests/latest: unauthorized: incorrect username or password

I'm not sure this needs to be a high priority for 1.10-rc2. The only way I can think of to hit this is if the password is changed after using docker login (which doesn't accept invalid passwords).

I also tested the second case described by @aaronlehmann and output seems exactly the same as in master
HTTP/1.1 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
Server: Docker/1.10.0-dev (linux)
X-Content-Type-Options: nosniff
Date: Wed, 27 Jan 2016 03:39:34 GMT
Content-Length: 115

Get https://registry-1.docker.io/v2/library/busybox/manifests/latest: unauthorized: incorrect username or password

The error is wrapped with *url.Error because it comes from the token server.
Adding the following at the beginning of isAuthorizedError solves this:
       if urlError, ok := err.(*url.Error); ok {
               err = urlError.Err
       }

But this still doesn't behave correctly. I get prompted for a username, but not for a password.
# docker pull busybox
Using default tag: latest

Please login prior to pull:
Username (aaronlehmann): aaronlehmann
Error response from daemon: Wrong login/password, please try again

I suggest deferring this to 1.11 or a 1.10 point release since it seems like neither behavior is a regression from 1.9, and it looks like it will take some work to get this right.

Pulling a repo that you don't have access to. I don't think this has ever prompted for creds. It invokes a v1 fallback, and the ultimate error code is a 404.

This behavior is correct. Let's not deviate this PR. We need to fix the regression.


HTTPS breaks after changing environment variables

Hello there,
Thanks a lot for this repository, it's been a big help. A minor issue though is when I update the values of environment variables from the aws console, HTTPS stops working. I think whatever ELB does after updating environment variables overwrites the nginx.conf file. Is this the intended behavior? In ELB there are "confighooks" which are the same "hooks" but are tied to configuration changes rather than deployments. I tried duplicating the script there and it's working but I wanted to confirm if this is the way to do it or if I should expect changing environment variables to break HTTPS.
Thanks

@MahmoudKhaledAli It isn't the intended behavior. Duplicating the script in a configuration hook works, but comes at the small cost of rerunning Certbot at each configuration change. I think an ebextension nginx configuration override after initial deploy could work. Another solution would be to have the app deployment hook save the file somewhere and have the config hook just copy that over. You're welcome to implement a solution and make a pull request!

Added hook to save nginx.conf between configuration changes


update translation with latest changes

[ ] @mark800 Can you help us in updating zh-cn again with the latest changes?
[ ] @TeddyCr Can you update fr-fr  again
[ ] @pmbrull Need help in Spanish and Catalan

Refer steps to add new localisation file here: https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-ui/src/main/resources/ui#steps-to-run-openmetadata-ui

Spanish done


Allow setting additional command line arguments in cmake-tools-kits.json

I am trying to use Clang with libc++ rather than libstdc++ via CMake. The perfect solution would be to have "args" parameter in cmake-tools-kits.json so I could setup:
  {
    "name": "Clang 10.0.0",
    "compilers": {
      "C": "/bin/clang-10",
      "CXX": "/bin/clang++-10"
    },
    "args": "-stdlib=libstdc++"
  },
  {
    "name": "Clang 10.0.0 libc++",
    "compilers": {
      "C": "/bin/clang-10",
      "CXX": "/bin/clang++-10"
    },
    "args": "-stdlib=libc++"
  },

Is there any other easy way to do it currently?

This sounds like something you might want to create Toolchains for.  You can use the toolchainFile property to point to a toolchain file that configures your compilers for you (and remove the compilers property).
Another option is to try the cmakeSettings property.  Any argument you pass to cmake via the command line should be settable here.  I haven't used the CLI to set compiler flags before, but it might be possible.  I think the preferred method is to instead modify the CMakeLists.txt to support the addition of the flag based on some control variable you might create (which could be passed in via a toolchain or the cmakeSettings property).

Thanks for your answer. I am using toolchainFile already to setup warnings in CXX_FLAGS that should only be used during package development. By this, I mean that compilation warnings should not be embedded inside the library CMake file and should not be enforced for its customers.
As I wrote above I already use toolchain files for warnings and I would need another orthogonal toolchain set for setting a library. Unfortunately, it is not easy to externally stack toolchain files (i.e. one for standard library + another for warnings). In other words, I do not want to include warnings script from libc++ toolchain script. And I cannot provide a list of toolchain files to CMake...

I think the preferred method is to instead modify the CMakeLists.txt to support the addition of the flag

I do not agree. CMakeLists.txt file should only describe how to build a project and describe its constraints. It should not embed any additional logic (compiler warnings, static analysis steps, libc++ vs libstdc++ vs libstdc++11, etc.). Those are development environment properties rather than project-specific information. Thus it should not appear in the CMakeLists.txt file. You could also name them "development kits" ;-) This is why I am looking for a way to provide such information in CMake kit description (it would be trivial in tasks.json but with CMake we do not have easy access to it).

What does your cmake command look like?  (In other words... how do you switch between libraries when you're not using this extension?)  Perhaps this will help me make a better recommendation if you disagree with updating your CMakeLists.txt.

I am not using cmake command line at all outside of VSCode. If I have to build a project from a command line I use Conan that handles all of this by itself.

If you can figure out the command line that Conan generates, then we might be able to cook up a solution that will work for you. Unfortunately, our team doesn't have much experience with Conan.


added clarification in apply docs

Was using apply and was slightly confused on how the function should be called, this should clarify things for others.

thanks


fix(stubs): teleport stub children as a function

Fixes #1829
I'm not super convinced by this fix, but it indeed looks like @freakzlike intuition is right: we need to return the children as a function for Teleport to avoid the warning.
Maybe @xanf you have a better idea on how to handle that?

@freakzlike I added a test to check that the warning is not logged anymore, and added a check in other tests when it is logged, but when that we expect it.
Maybe we should do the same for the warnings logged by vue compact (either mock them or fix them if needed) to avoid polluting the test logs, and missing a regression like this one in the future
Anyway, this should fix the issue, unless someone comes up with a better idea. I'll let you give a last check and merge


Error while installing usd_from_gltf: DRACO: ERROR: [Error 2] The system cannot find the file specified

I'm at a loss for where to begin debugging this error, given that the missing file isn't specified. Hoping at first for suggestions.

C:\Users\neoke>python .\usd_from_gltf\tools\ufginstall\ufginstall.py .\usd_from_gltf_build .\usd-20-11-win64_py27_release --testdata
Install Settings:
  Source Directory:   C:\Users\neoke\usd_from_gltf
  Install Directory:  C:\Users\neoke\usd_from_gltf_build
  USD Directory:      C:\Users\neoke\usd-20-11-win64_py27_release
  Download Directory: C:\Users\neoke\usd_from_gltf_build\src
  Build Directory:    C:\Users\neoke\usd_from_gltf_build\build
  Build Config:       Release
  CMake Generator:    Default
  Already Installed:  <none>
  Installing:         DRACO, GIF, JPG, JSON, ZLIB, PNG, STB_IMAGE, TCLAP, TESTDATA_SAMPLES, TESTDATA_REFERENCE, USD_FROM_GLTF

-------- Installing DRACO --------
  DRACO: File already exists, skipping download: C:\Users\neoke\usd_from_gltf_build\src\draco.zip
  DRACO: Unpacking to: C:\Users\neoke\usd_from_gltf_build\src\draco-1.3.5
  DRACO: CWD: C:\Users\neoke\usd_from_gltf_build\build\draco-1.3.5
  DRACO: Run: cmake C:\Users\neoke\usd_from_gltf_build\src\draco-1.3.5 -DCMAKE_INSTALL_PREFIX=C:\Users\neoke\usd_from_gltf_build -DCMAKE_PREFIX_PATH=C:\Users\neoke\usd_from_gltf_build -DCMAKE_POSITION_INDEPENDENT_CODE=1 -DCMAKE_GENERATOR_PLATFORM=x64
  DRACO: ERROR: [Error 2] The system cannot find the file specified

If it helps, I'm using Windows 10 with prebuilt usd binaries from Nvidia.
After a a bit of digging, I've narrowed down the place throwing the error to a subprocess call, but still no progress with finding out which file is missing.
def run(args):
    ....
    status('got here y')
    process = subprocess.Popen(
        args,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        universal_newlines=True)
    status('got here z')

terminal output
C:\Users\neoke>python .\usd_from_gltf\tools\ufginstall\ufginstall.py .\usd_build .\Downloads\usd-20-11-win64_py27_release
Install Settings:
  Source Directory:   C:\Users\neoke\usd_from_gltf
  Install Directory:  C:\Users\neoke\usd_build
  USD Directory:      C:\Users\neoke\Downloads\usd-20-11-win64_py27_release
  Download Directory: C:\Users\neoke\usd_build\src
  Build Directory:    C:\Users\neoke\usd_build\build
  Build Config:       Release
  CMake Generator:    Default
  Already Installed:  <none>
  Installing:         DRACO, GIF, JPG, JSON, ZLIB, PNG, STB_IMAGE, TCLAP, USD_FROM_GLTF

-------- Installing DRACO --------
  DRACO: File already exists, skipping download: C:\Users\neoke\usd_build\src\draco.zip
  DRACO: Unpacking to: C:\Users\neoke\usd_build\src\draco-1.3.5
  DRACO: CWD: C:\Users\neoke\usd_build\build\draco-1.3.5
  DRACO: Run: cmake C:\Users\neoke\usd_build\src\draco-1.3.5 -DCMAKE_INSTALL_PREFIX=C:\Users\neoke\usd_build -DCMAKE_PREFIX_PATH=C:\Users\neoke\usd_build -DCMAKE_POSITION_INDEPENDENT_CODE=1 -DCMAKE_GENERATOR_PLATFORM=x64
  DRACO: log path: C:\Users\neoke\usd_build\build\draco-1.3.5\log.txt
  DRACO: got here 1
  DRACO: got here y
  DRACO: ERROR: [Error 2] The system cannot find the file specified

Update:
Turns out I was missing cmake.
Given I used prebuilt binaries from Nvidia, I skipped the USD build step, which would have required me to download cmake.
That said, I'm still getting build errors as shown below, but at this point I'm closing this issue and going to explore using the Docker builds discussed in #39
-------- Installing DRACO --------
  DRACO: File already exists, skipping download: C:\Users\neoke\usd_build\src\draco.zip
  DRACO: Unpacking to: C:\Users\neoke\usd_build\src\draco-1.3.5
  DRACO: CWD: C:\Users\neoke\usd_build\build\draco-1.3.5
  DRACO: Run: cmake C:\Users\neoke\usd_build\src\draco-1.3.5 -DCMAKE_INSTALL_PREFIX=C:\Users\neoke\usd_build -DCMAKE_PREFIX_PATH=C:\Users\neoke\usd_build -DCMAKE_POSITION_INDEPENDENT_CODE=1 -DCMAKE_GENERATOR_PLATFORM=x64
  DRACO: log path: C:\Users\neoke\usd_build\build\draco-1.3.5\log.txt
  DRACO: got here 1
  DRACO: got here y
  DRACO: got here z
  DRACO: got here 2
  DRACO: got here 3
-- Building for: Visual Studio 14 2015
INFO|Draco: ignoring empty build type, forcing release mode.
-- Selecting Windows SDK version  to target Windows 10.0.16299.
CMake Error at CMakeLists.txt:11 (project):
  Failed to run MSBuild command:

    C:/Program Files (x86)/MSBuild/14.0/bin/MSBuild.exe

  to get the value of VCTargetsPath:

    Microsoft (R) Build Engine version 14.0.25420.1
    Copyright (C) Microsoft Corporation. All rights reserved.

    Build started 30/10/2020 16:42:20.
    Project "C:\Users\neoke\usd_build\build\draco-1.3.5\CMakeFiles\3.18.2\VCTargetsPath.vcxproj" on node 1 (default targets).
    C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Platforms\x64\PlatformToolsets\v140\Toolset.targets(36,5): error MSB8036: The Windows SDK version 8.1 was not found. Install the required version of Windows SDK or change the SDK version in the project property pages or by right-clicking the solution and selecting "Retarget solution". [C:\Users\neoke\usd_build\build\draco-1.3.5\CMakeFiles\3.18.2\VCTargetsPath.vcxproj]
    Done Building Project "C:\Users\neoke\usd_build\build\draco-1.3.5\CMakeFiles\3.18.2\VCTargetsPath.vcxproj" (default targets) -- FAILED.

    Build FAILED.


[Q]  TDT/TOT PID packet with no complete section inside, cannot update

Hello all, this is Jesus
I'm using TSDuck v3.26-2349 running on a Raspberry PI 3
I wish to generate a TS from the scratch; When I add the TOT and TDT table files, the timeref pluggin show this message "* Warning: timeref: got TDT/TOT PID packet with no complete section inside, cannot update"   I wrote those table files in XML format from a RF captured signal. I revised the TOT and TDT files but I don't know what's wrong; maybe in the TOT file xml structure some parameter is missing, but I tried adding local_time_offset_polarity but the timeref pluggin say that is not valid parameter.
This is the TDT.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<tsduck>
	<TDT UTC_time="2021-06-01 11:20:18">
		<metadata PID="20"/>
	</TDT>
</tsduck>

and this is the TOT.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<tsduck>
	<TOT UTC_time="2021-06-01 11:20:18">
		<metadata PID="20"/>
		<local_time_offset_descriptor>
			<region country_code="ESP" country_region_id="1" local_time_offset="120" time_of_change="2021-10-31 01:00:00" next_time_offset="60"/>
		</local_time_offset_descriptor>
	</TOT>
</tsduck>

I think that both xml structures are fine, but the timeref is waiting something that I don't know.
The command is:
tsp --verbose --bitrate 600000 --max-flushed-packets 70 \
-I null \
-P regulate --packet-burst 14 \
-P inject pat.xml=500 --pid 0 --bitrate 17000 \
-P inject nit.xml=10000 --pid 16 --bitrate 15000 \
-P inject --poll-files eit_pf.xml --pid 18 --bitrate 32000 \
-P inject pmt100.xml --pid 100 --bitrate 15000 \
-P inject pmt200.xml --pid 200 --bitrate 15000 \
-P inject sdt.xml --pid 17 --bitrate 1500 \
-P inject ait.xml --pid 139 --bitrate 15000 \
-P inject tdt.xml tot.xml --pid 20 --bitrate 1300 \
-P timeref --start system --local-time-offset 120 --next-change "2021/10/24:01:00:00" --next-time-offset 60 \
-O ip --rtp <IP_ADDRESS>:20000 --packet-burst 7 --enforce-burst

and the log is:

timeref: current system clock is 2021/06/02 10:37:24.249
inject: loaded 1 sections from tdt.xml, repetition rate: unspecified
inject: loaded 1 sections from tot.xml, repetition rate: unspecified
inject: loaded 1 sections from ait.xml, repetition rate: unspecified
inject: loaded 1 sections from sdt.xml, repetition rate: unspecified
inject: loaded 1 sections from pmt200.xml, repetition rate: unspecified
inject: loaded 1 sections from pmt100.xml, repetition rate: unspecified
inject: loaded 2 sections from eit_pf.xml, repetition rate: unspecified
inject: loaded 1 sections from nit.xml, repetition rate: 10,000 ms
inject: loaded 1 sections from pat.xml, repetition rate: 500 ms
regulate: minimum packet burst duration is 10,000,000 nano-seconds
null: initial input bitrate is 600,000 b/s
regulate: regulated at bitrate 600,000 b/s
inject: transport bitrate: 600,000 b/s, packet interval: 35
inject: transport bitrate: 600,000 b/s, packet interval: 40
inject: transport bitrate: 600,000 b/s, packet interval: 18
inject: transport bitrate: 600,000 b/s, packet interval: 40
inject: transport bitrate: 600,000 b/s, packet interval: 40
inject: transport bitrate: 600,000 b/s, packet interval: 400
inject: transport bitrate: 600,000 b/s, packet interval: 40
inject: transport bitrate: 600,000 b/s, packet interval: 461
Warning: timeref: got TDT/TOT PID packet with no complete section inside, cannot update
inject: file eit_pf.xml created
inject: loaded 2 sections from eit_pf.xml, repetition rate: unspecified
inject: transport bitrate: 600,000 b/s, packet interval: 18
Warning: timeref: got TDT/TOT PID packet with no complete section inside, cannot update
Warning: timeref: got TDT/TOT PID packet with no complete section inside, cannot update
Warning: timeref: got TDT/TOT PID packet with no complete section inside, cannot update
Warning: timeref: got TDT/TOT PID packet with no complete section inside, cannot update
Warning: timeref: got TDT/TOT PID packet with no complete section inside, cannot update

Thank you for your help.
P.S.  The timeref appears that is updating the TDT and TOT tables:

I think I found the problem; I added to the tdt.xml and tot.xml the repetition period:
-P inject tdt.xml**=30000** tot.xml**=10000**
and now works fine.
Thank you.


562 2 tests

This is a pull request!
Subject: Test Implementation for FormPage Component
Test Goals for FormPage:
The goal of these tests was to verify the functionality and behavior of the FormPage component, ensuring a smooth and reliable user experience. The following objectives were targeted:
Rendering Validation:
Confirm that all essential elements of the form (e.g., input fields for Name, Gender, Email, Size, and the Submit button) render correctly and are visible to the user.
User Interaction Testing:
Simulate user input into all form fields to ensure that data is captured correctly.
Validate that the form submission works as expected by checking if the submitted data is logged correctly in the console (mocked for the test).
These tests help ensure that the FormPage works as intended and provides a solid foundation for further development or enhancements.
Let me know if you need further details or improvements!
test:
npm test form-page.test.tsx
Subject: Test Goals for EditUserRolePage Component
Objective: The purpose of the tests for the EditUserRolePage component is to ensure its core functionality works as intended and that different user roles are handled correctly. This includes testing the behavior of the page for admin users and non-admin users, as well as verifying the data flow and error handling.
Test Goals:
Admin User Access:
Verify that admin users can access the EditUserRolePage.
Ensure the UserCard components are correctly displayed with user data fetched from the API.
Non-Admin User Restriction:
Ensure that non-admin users see an UnauthorizedPageMessage and cannot access the admin functionality.
Data Fetching:
Mock the API response to confirm that the user data is fetched and displayed correctly.
Test how the component handles errors or missing tokens during the data-fetching process.
Error Handling:
Ensure the page gracefully handles missing or invalid tokens by rendering the appropriate error message.
Component Integration:
Verify the integration of child components like UserCard and UnauthorizedPageMessage using mocks.
npm test edit-user-role-page.test.tsx

ps: We implemented unit tests for two untested pages: FormPage and EditUserRolePage. The tests focused on ensuring key functionalities worked as expected. For FormPage, we tested the rendering of form fields, handling of user inputs, and form submission behavior. For EditUserRolePage, we tested role-based access control, verifying that admin users could view and interact with user data while non-admin users were shown an unauthorized message. Challenges included resolving path alias issues for mocking components, which required updates to tsconfig.json and jest.config.js. Dynamic routing in EditUserRolePage also posed difficulties, which we addressed by mocking API calls and router behavior. Despite these challenges, the tests were successfully implemented and verified through a detailed pull request, ensuring the functionality of these pages is now well-covered.

Good job on utilizing mocks in these tests. I will approve once Ozzie's changes have been addressed. Great catch! @Asfand00


[TASK] Implement New Design for Restake

A new version of the restake feature includes:

[ ] The restake card with inputs and tags. Screenshot
[ ] The asset info card to show the current selected asset information. Screenshot
[ ] The operator info card to display the current selected operator information. Screenshot
[ ] The onboarding card that explains and shows the steps of the restake feature. Screenshot
[ ] Logic for batching the bridge + deposit + delegate into 1 transaciton.

Full page screenshot

Update
For finalized approved design, the below components are no longer needs to be implemented:

The asset info card to show the current selected asset information.
The operator info card to display the current selected operator information.
The onboarding card that explains and shows the steps of the restake feature.

For the restake dapp, we will have single column for deposit and stake, and switch to two columns for unstake and withdrawal based on @drewstone 's request.
Screen Recording
https://github.com/user-attachments/assets/01a5f375-044c-4c1b-8e8a-e98c1a72e157


Error occured on webrequest: Exception calling "DownloadString" with "1" argument(s): "Unable to connect to the remote server"

Windows 10 running build file, error occurs with logger vm
WARNING: Something went wrong while attempting to build the logger box.
Attempting to reload and reprovision the host...
[main] Running vagrant_reload_host for: logger
[vagrant_reload_host] Running for logger
[vagrant_reload_host] Finished for logger. Got exit code: 1
C:\Users\Eric\DetectionLab\build.ps1 : Failed to bring up logger after a reload. Exiting
At line:1 char:1

.\build.ps1 virtualbox -VagrantOnly

  + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
  + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,build.ps1

[main] Running post_build_checks
[post_build_checks] Running Splunk Check.
[download] Running for https://<IP_ADDRESS>:8000/en-US/account/login?return_to=%2Fen-US%2F, looking for This browser is not supported by Splunk
Error occured on webrequest: Exception calling "DownloadString" with "1" argument(s): "Unable to connect to the remote server"
[post_build_checks] Splunk Result: False
[post_build_checks] Running Fleet Check.
[download] Running for https://<IP_ADDRESS>:8412, looking for Kolide Fleet
Error occured on webrequest: Exception calling "DownloadString" with "1" argument(s): "Unable to connect to the remote server"
[post_build_checks] Fleet Result: False
[post_build_checks] Running MS ATA Check.
[download] Running for https://<IP_ADDRESS>, looking for
Error occured on webrequest: Exception calling "DownloadString" with "1" argument(s): "Unable to connect to the remote server"
[post_build_checks] ATA Result: False
WARNING: Splunk failed post-build tests and may not be functioning correctly.
WARNING: Fleet failed post-build tests and may not be functioning correctly.
WARNING: MS ATA failed post-build tests and may not be functioning correctly.

Hi @Eric4Smith ,
Can you post the vagrant_up_logger.log file in your Vagrant directory into a gist and link it here?

Hi @Eric4Smith ,
Can you post the vagrant_up_logger.log file in your Vagrant directory into a gist and link it here?

https://gist.github.com/Eric4Smith/8f4710c02669ad92c518009bddf01442

Hi @Eric4Smith ,
Can you try putting the following lines right after #! /bin/bash inside of bootstrap.sh?
echo -e "    eth1:\n      dhcp4: true\n      nameservers:\n        addresses: [<IP_ADDRESS>,<IP_ADDRESS>]" >> /etc/netplan/01-netcfg.yaml
netplan apply
sed -i 's/nameserver <IP_ADDRESS>/nameserver <IP_ADDRESS>/g' /etc/resolv.conf && chattr +i /etc/resolv.conf

This is a weird DNS problem that I've also encountered and I'm not 100% sure what the exact root cause is, but this seems to fix it.

Hi @Eric4Smith ,
Can you try putting the following lines right after #! /bin/bash inside of bootstrap.sh?
echo -e "    eth1:\n      dhcp4: true\n      nameservers:\n        addresses: [<IP_ADDRESS>,<IP_ADDRESS>]" >> /etc/netplan/01-netcfg.yaml
netplan apply
sed -i 's/nameserver <IP_ADDRESS>/nameserver <IP_ADDRESS>/g' /etc/resolv.conf && chattr +i /etc/resolv.conf

This is a weird DNS problem that I've also encountered and I'm not 100% sure what the exact root cause is, but this seems to fix it.

Im running this on windows, is there a PS workaround?

Now it seems that the wef box gets hung when it attempts to shutdown. It is stuck in the stopped state in virtualbox

Bummer. If the logger issue is fixed, would you mind closing this issue and opening a new one to address the WEF one? I want to make sure there’s a 1 issue to 1 fix mapping in these issues.


Update DeveloperGuide (focus on delete feature)

Update DeveloperGuide delete feature content for v1.1

Codecov Report

Merging #14 (0755317) into master (a61abaa) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #14   +/-   ##
=========================================
  Coverage     72.15%   72.15%
  Complexity      399      399
=========================================
  Files            70       70
  Lines          1232     1232
  Branches        125      125
=========================================
  Hits            889      889
  Misses          311      311
  Partials         32       32

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a61abaa...0755317. Read the comment docs.

LGTM


Behaviour module generation

For the server side, I imagine that the user would specify a handler
module and pass it into the service. Then it would be called with the
appropriate arguments.
I thought it might be nice to have a behaviour for these modules that
users could be implement.
Behavoiurs here are fully typed with the correct success types.

Ebert has finished reviewing this Pull Request and has found:

1 possible new issue (including those that may have been commented here).

You can see more details about this review at https://ebertapp.io/github/pinterest/elixir-thrift/pulls/99.


SDL_FreeSurface causes a crash when used on surface drawn to by TTF_RenderTextSolid/Blended

I've encountered a weird issue where whenever I try to use SDL_FreeSurface on an SDL_Surface modified by TTF_RenderTextSolid or TTF_RenderTextBlended, I'm getting a crash where the message from my IDE is EXC_BAD_ACCESS. When I inspect the SDL_Surface variable following a crash using the debugger, it's pointing to a valid instance. What's unusual is I'm using SDL_FreeSurface calls elsewhere in my codebase for freeing other surfaces and not having the same issue.
Here is the code where I'm performing the calls:
`SDL_Surface *textSurface = TTF_RenderText_Solid("Arial.ttf", text.c_str(), color);
if (textSurface == NULL)
{
    std::cout << "TTF_RenderText failed: " << TTF_GetError() << std::endl;
    return;
}
else
{
    SDL_Texture *textTexture = GenerateSdlTexture(textSurface);

    if (textTexture == NULL)
    {
        std::cout << "SDL_CreateTextureFromSurface failed: " << SDL_GetError() << std::endl;
        return;
    }
    else
    {
        int w = textSurface->w;
        int h = textSurface->h;

        SDL_Rect destRect;
        destRect.x = destLoc->x;
        destRect.y = destLoc->y;
        destRect.w = w;
        destRect.h = h;

        // Calls another function with the SDL_RenderCopy call
        RenderTexture(textTexture, NULL, &destRect, true);
    }

    // This causes the crash
    SDL_FreeSurface(textSurface);

}`
I don't know if it makes any difference, but I'm developing on an M1X (Apple Silicon) Mac with macOS Monterey using Xcode 13.3.

If you're comfortable around CMake, can you try the cmake script at https://github.com/libsdl-org/SDL_ttf/pull/213?
git fetch origin pull/213/head:pr213
git checkout pr213

I'm not familiar with using CMake.
I don't know if this will lend any insight whatsoever as to what might be wrong with the SDL_Surface, but I attached a screenshot of the crash message and the details of the instance.


playback example: fix a bug where audio is choppy

Fix a bug where audio is choppy because the samples buffer passed to the onSendSamples callback is not being filled completely.  For this purpose, the io.ReadFull wrapper function was used as it blocks until there is enough data to fill the whole buffer (which mp3.(*Decoder).Read doesn't seem to be doing).

Great, thanks!


Client.channels does not include DMs after restart

Please describe the problem you are having in as much detail as possible:
When querying channels from Client.channels, I noticed that the DM channels weren't listed until a message gets sent in one of them
Code sample:
var Discord = require("discord.js");
var client = new Discord.Client();
client.login(token); // Log in with a token or whatever method you prefer

client.on("message", context => {
  console.log();
  console.log(Array.from(client.channels.filter(channel => channel.type == "dm").keys()).join("\n")); // Filter the DM channels, then list their ID
}

When the bot receives a DM, it'll start logging this DM channel's ID. However, if the bot restarts, it will no longer log these DM channels (until it receives a DM)
Further details:

discord.js version: 11.2.1
node.js version: 9.2.0
Operating system: Antergos Linux release 17.10 (ISO-Rolling)
Priority this issue should have – please be realistic and elaborate if possible: low

[ ] I found this issue while running code on an user account
[ ] I have also tested the issue on latest master, commit hash:

see discordapp/discord-api-docs#184 not something d.js can fix.


Release report editor

There is a report editor contained within the SAM development environment for internal use that we would like to release publicly as a largely unsupported bonus feature, which will involve moving it in front of the SAM interface password in the public version of SAM. We'll target this for our next release, and hope to accompany it with a short webinar to teach people how to use it.

Update 9/24:
Removing from the 2021 Release milestone, because this requires a lot of documentation and error checking (hard crashes from @mjprilliman) if released as is.

If we release a report editor, we need to upgrade it to something more usable. We'll see if we get any requests/funding for that type of thing.


[Request] Add workaround for Skylake and Kabylake R built-in Speaker and Mic Support

There was a bug on Skylake and Kabylake R laptop where built-in soundcard (ALC256) is not detected^1^3
There is a fix for that bug that require compile something with alsatplg and copy the file to /lib/firmware/ and add a line in /etc/modprobe.d/*.conf. Complete steps is here: https://gist.github.com/crojewsk/4e6382bfb0dbfaaf60513174211f29cb#topology
Could someone do this or atleast help me implement this? I read that modprobe can be added in config file but I'm not sure where /lib/firmware/ is, I can't find the folder in NixOS. When I use other distro I can find the folder and this workaround is working.

I tried with this nix file but it's not working (it was working when I apply it in other distro), anything I missed based on the guide on gist?
{ config, pkgs, ... }:

let alsa-topology = pkgs.alsa-topology-conf;
in
{
  # Add necessary packages
  environment.systemPackages = with pkgs; [
    alsa-ucm-conf
    alsa-topology-conf
    alsa-lib
    alsa-utils
  ];

  # Compile ALSA Topology
  hardware.firmware = [
    (pkgs.runCommand "alsa-hda-dsp-topology" { nativeBuildInputs = [ pkgs.alsa-utils ]; } ''
      mkdir -p $out/lib/firmware
      alsatplg -c ${alsa-topology}/share/alsa/topology/hda-dsp/skl_hda_dsp_generic-tplg.conf -o $out/lib/firmware/skl_hda_dsp_generic-tplg.bin
    '')
    pkgs.alsa-firmware
    pkgs.sof-firmware
  ];

  # Replace default DSP Firmware
  system.replaceRuntimeDependencies = [
    {
      original = pkgs.firmwareLinuxNonfree;
      replacement = pkgs.firmwareLinuxNonfree.overrideAttrs (super: {
        postInstall = ''
          cd $out/lib/firmware/intel/
          rm dsp_fw_release.bin
          ln -s dsp_fw_kbl_v3420.bin dsp_fw_release.bin
        '';
      });
    }
  ];

  # Add modprobe
  boot.extraModprobeConfig = "options snd-intel-dspcfg dsp_driver=1\n";
}

Have you checked if /run/current-system/firmware contains the changes you would expect it to have? An alternative to replaceRuntimeDependencies would be to use nixpkgs.config.packageOverrides to override firmwareLinuxNonfree


update README to include treatment info

Added a table of treatments for interpret ability of  analysis. Please let me know if this makes any sense?

@jenna-tomkinson I agree with all of these comments. Thank you for your input! I will make some changes and request a review tomorrow.


Add errors and fields to the Vue typings

🔎 Overview
Feat: It allows to use this.errors and this.fields directly in TypeScript code, as they are injected via the mixin and documented (see https://baianat.github.io/vee-validate/api/mixin.html). The $validator was already exposed.
🤓 Code snippets/examples (if applicable)
Allows a method like:
public isDirtyAndInError(name: string) {
  const field = this.$validator.fields.find({ name });
  if (field) {
    return field.flags.dirty && this.$validator.errors.has(name);
  }
  return false;
}

to be written as:
public isDirtyAndInError(name: string) {
   return this.fields[name] && this.fields[name].dirty && this.errors.has(name);
}

✔ Issues affected
Fixes #1704

Thanks for the PR, much appreciated!


Packagejsonupdates

This solves incompatibility with master and updates the package.json to reflect the release tag.

Coverage remained the same at 54.979% when pulling 627d18b541f487969d1b4685f494143ce9a6e9c2 on packagejsonupdates into 787139ddb382cb304b0208d1461ee235e1b9135b on release2.1.


Class not found exception JavaUtilLogCreator

Which version and edition of Flyway are you using?
5.1.4
Which client are you using? (Command-line, Java API, Maven plugin, Gradle plugin)
Java API
Which database are you using (type & version)?
Postgresql 9.6
Which operating system are you using?
CentOS 7
What did you do?
Using ant and flyway-ant: https://mvnrepository.com/artifact/cz.net21.flyway.ant/flyway-ant/1.1.4
Ran <flyway:info /> with ant
What did you expect to see?
Info table
What did you see instead?
 Could not create type antlib:org.flywaydb.ant:repair due to java.lang.ExceptionInInitializerError
 ...
 Caused by: org.flywaydb.core.api.FlywayException: Unable to instantiate class org.flywaydb.core.internal.util.logging.javautil.JavaUtilLogCreator : org.flywaydb.core.internal.util.logging.javautil.JavaUtilLogCreator
 ....
 Caused by: java.lang.ClassNotFoundException: org.flywaydb.core.internal.util.logging.javautil.JavaUtilLogCreator

The issue is already addressed in https://github.com/flyway/flyway-ant/issues/4

Was already fixed by https://github.com/flyway/flyway-ant/commit/ef14d5f79596ee55003b53d0edff64669c68bba8
I suggest to close this issue.


Improve Session misuse resistance

Me again! Hopefully I'm actually correct this time...
Since SessionConn always uses a counter starting at 1 for its GCM nonce, it must never be used with the same encryption key twice. Consider at least documenting this constraint, or if possible don't export SessionConn at all.
Similarly, a single Session must never be reused to connect to the same peer Session twice (since it would then generate the same shared key, violating the requirement above). Here I believe there is a simple fix: move the key generation from NewSession to Establish; as a bonus, you then don't even need to store ourPriv on Session.
Take a look at https://noiseprotocol.org/noise.html (especially section 14).

Good points! I'll be sure to document the constraint and look into moving the key generation over to Establish tomorrow; thank you :smile:.

Moved key generation from NewSession to DoServer and DoClient, and documented the constraint that the same cipher.AEAD suite should not be used for multiple SessionConn's.
Let me know what you think.


Include Customer Name in notification

Hi there,
Would it be possible to include the customer name in "A new customer account was created" notification?
Similarly, it would also be useful to include some information in the new order notification. For example, order number and amount.

@neilbradley This should be addressed in the the latest release.


regression: Slick failing, problem with Ornate

seen at e.g. https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-community-build/1395/consoleFull

cc @szeiger. I think it might work to just exclude the "root" project, since it isn't a real project anyway

2.11.x-jdk8 was affected as well. on that branch, I just froze it, we don't really need stuff unfrozen there: 75fcae36a68590cc5d0478bf0a16d6524822792a, green run https://scala-ci.typesafe.com/view/scala-2.11.x/job/scala-2.11.x-jdk8-integrate-community-build/358/

Oh, only a missing dependency because you don't build Ornate. I thought it would be something more sinister like strange exceptions being thrown from the internals of Rhino because a Java 8 installation on the build slave is a few patch levels too old :)

@szeiger would you like to stab at adding Ornate?

I don't think it would be a very useful addition. It's an application (not a library), written in pretty simple Scala (nothing interesting to test) and heavily using Nashorn (which provides many opportunities for things to go wrong that are unrelated to what we want to test in the community build)


Generation of the Firebase Android Resource File Failing Again

Unity editor version: 2021.3.20f1
Firebase Unity SDK version: 11.1.0
Source you installed the SDK: .unitypackage file from Firebase Console
Problematic Firebase Component: Analytics
Other Firebase Components in use: N/A
Additional SDKs you are using: N/A
Platform you are using the Unity editor on: Mac, Apple Silicon
Platform you are targeting: Android/iOS
Scripting Runtime: IL2CPP

I started integrating Firebase Analytics into our Unity Project yesterday, and today we are now getting the following error:
Generation of the Firebase Android resource file google-services.xml from [Unity Project]/Assets/../Assets/Configs/Firebase/google-services.json failed.
If you have not included a valid Firebase Android resources in your app it will fail to initialize.

"python3" "[Unity Project]/Assets/Firebase/Editor/generate_xml_from_google_services_json.py" -i "[Unity Project]/Assets/../Assets/Configs/Firebase/google-services.json" -o "[Unity Project]/Assets/../Assets/Plugins/Android/FirebaseApp.androidlib/res/values/google-services.xml" -p "com.exceptionullgames.golftower"

Traceback (most recent call last):
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py", line 762, in _get_writer
    write = file_or_filename.write
AttributeError: 'str' object has no attribute 'write'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "[Unity Project]/Assets/Firebase/Editor/generate_xml_from_google_services_json.py", line 498, in <module>
    sys.exit(main())
  File "[Unity Project]/Assets/Firebase/Editor/generate_xml_from_google_services_json.py", line 490, in main
    tree.write(output_filename, 'utf-8', True)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py", line 732, in write
    with _get_writer(file_or_filename, enc_lower) as write:
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py", line 768, in _get_writer
    file = open(file_or_filename, "w", encoding=encoding,
PermissionError: [Errno 13] Permission denied: '[Unity Project]/Assets/../Assets/Plugins/Android/FirebaseApp.androidlib/res/values/google-services.xml'

You can start to diagnose this issue by executing ""python3" "[Unity Project]/Assets/Firebase/Editor/generate_xml_from_google_services_json.py" -i "[Unity Project]/Assets/../Assets/Configs/Firebase/google-services.json" -o "[Unity Project]/Assets/../Assets/Plugins/Android/FirebaseApp.androidlib/res/values/google-services.xml" -p "com.exceptionullgames.golftower"" from the command line.
UnityEngine.Debug:LogError (object)
Firebase.Editor.GenerateXmlFromGoogleServicesJson:RunResourceGenerator (System.Collections.Generic.IEnumerable`1<string>,string,bool) (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/editor/app/src/GenerateXmlFromGoogleServicesJson.cs:545)
Firebase.Editor.GenerateXmlFromGoogleServicesJson:GenerateXmlResources (string,bool) (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/editor/app/src/GenerateXmlFromGoogleServicesJson.cs:588)
Firebase.Editor.GenerateXmlFromGoogleServicesJson:UpdateJson (bool,Firebase.Editor.GenerateXmlFromGoogleServicesJson/LogMessage,Firebase.Editor.GenerateXmlFromGoogleServicesJson/LogMessage) (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/editor/app/src/GenerateXmlFromGoogleServicesJson.cs:165)
Firebase.Editor.GenerateXmlFromGoogleServicesJson:UpdateJsonWithBundleIdChooserDialog (string,bool) (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/editor/app/src/GenerateXmlFromGoogleServicesJson.cs:481)
Firebase.Editor.GenerateXmlFromGoogleServicesJson:CheckConfiguration () (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/editor/app/src/GenerateXmlFromGoogleServicesJson.cs:254)
Firebase.Editor.GenerateXmlFromGoogleServicesJson/<>c:<.cctor>b__15_1 () (at /Users/runner/work/firebase-unity-sdk/firebase-unity-sdk/editor/app/src/GenerateXmlFromGoogleServicesJson.cs:103)
Google.EditorInitializer/<InitializeOnMainThread>c__AnonStorey1:<>m__2 () (at /Users/chkuang/Workspace/git/unity-jar-resolver/source/VersionHandlerImpl/src/EditorInitializer.cs:60)
Google.RunOnMainThread:ExecutePollingJobs () (at /Users/chkuang/Workspace/git/unity-jar-resolver/source/VersionHandlerImpl/src/RunOnMainThread.cs:409)
Google.RunOnMainThread:<ExecuteAllUnnested>m__12 () (at /Users/chkuang/Workspace/git/unity-jar-resolver/source/VersionHandlerImpl/src/RunOnMainThread.cs:541)
Google.RunOnMainThread:RunAction (System.Action) (at /Users/chkuang/Workspace/git/unity-jar-resolver/source/VersionHandlerImpl/src/RunOnMainThread.cs:343)
Google.RunOnMainThread:ExecuteAllUnnested (bool) (at /Users/chkuang/Workspace/git/unity-jar-resolver/source/VersionHandlerImpl/src/RunOnMainThread.cs:530)
Google.RunOnMainThread:ExecuteAll () (at /Users/chkuang/Workspace/git/unity-jar-resolver/source/VersionHandlerImpl/src/RunOnMainThread.cs:512)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions () (at /Users/bokken/build/output/unity/unity/Editor/Mono/EditorApplication.cs:356)

As far as I can see this seems to be the same, or a very similar issue to this closed issue.
However following the steps for the work-around patm1987 provided still leaves me with the same exact error message as before and doesn't seem to fix anything for us...
This has been a big headache, especially from the confusion caused by this issue too.
Reproduction Steps:

Make a new Unity Project
Create a Unity project on the Firebase Console
Follow the steps in the Firebase Console guide to download and import the .json and .plist files for Android and iOS
Import the .unitypackage given by the Firebase Console
Start logging analytics while running the game in Play Mode
End Play Mode in the editor and then observe the error message in the Unity Console.

This issue appears to be related to version controlling.
Specifically Perforce sets all files to Read Only upon submission, and having the generated: google-services.xml file marked as "Read Only" was breaking the functionality for the python script here.

Thanks for sharing the additional information about the cause of the issue.
In this case, I think it is correct for Firebase to be unable to modify a read-only file, so I am going to close this issue. If anyone else encounters the same problem, they should be able to look into perforce settings to find a way to not mark the file as read only.


Encountering LinAlgError despite validating data with isfinite() and isnan()

I am working on a school project where I have created a wrapper for the hierarchicalforecast library. The wrapper enables team members to use the library without preprocessing the data and enabling or disabling the aggregation as we do forecasts for all levels. However, I am encountering an error in the .reconcile() function. The error message is as follows:
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/statsmodels/stats/moment_helpers.py:252: RuntimeWarning: invalid value encountered in true_divide
  corr = cov / np.outer(std_, std_)
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/hierarchicalforecast/methods.py:413: RuntimeWarning: invalid value encountered in true_divide
  shape residuals_insample (n_hiers, obs)
397 self.Y_rec = self.hrec.reconcile(
    398     Y_hat_df=self.Y_hat_reshaped,
    399     Y_df=self.Y_fitted_df,
    400     S=self.summation_matrix,
    401     tags=self.tags)
    402 if self.Y_rec[self.forecasting_algorithm_used].equals(self.Y_rec.columns[-1]):
    403     print('Reconciliation did not change the prediction.')
...
    206 for a in arrays:
    207     if not isfinite(a).all():
--> 208         raise LinAlgError("Array must not contain infs or NaNs")

LinAlgError: Array must not contain infs or NaNs

I have already confirmed that the data is valid by using the following code:
# Check if data is finite or NaN
assert self.Y_fitted_df['y'].apply(
    np.isfinite).all(), 'Values in Y_insample are not finite'
assert self.Y_fitted_df[self.forecasting_algorithm_used].apply(
    np.isfinite).all(), 'Values in Y_hat_insample are not finite'
assert self.Y_hat_reshaped[self.forecasting_algorithm_used].apply(
    np.isfinite).all(), 'Values in Y_hat are not finite'
assert not self.Y_fitted_df['y'].apply(
    np.isnan).all(), 'Some values in Y_insample are NaN'
assert not self.Y_fitted_df[self.forecasting_algorithm_used].apply(
    np.isnan).all(), 'Some values in Y_hat_insample are NaN'
assert not self.Y_hat_reshaped[self.forecasting_algorithm_used].apply(
    np.isnan).all(), 'Some values in Y_hat are NaN'

As this is my first issue report, I kindly request your understanding and patience with my writing. Thank you in advance for any assistance you may provide in resolving this issue.

Hi @DennisToma,
From looking at the logs it seems that this line is having troubles:
corr = cov / np.outer(std_, std_)

Which means that some of your series might have std=0 => the series are a single constant.
Some of the methods may have troubles to deal with series without variance.
Some possible solutions:

Add some random Gaussian noise to them that would act almost like a ridge regression regularizer.
On our side add protections to check in advance for incompatibilities between constant series and reconciliation methods.

Would you be able to confirm the diagnosis?

Hi @kdgutier,
thank you for your fast reply. Unfortunately, I am unable to confirm the diagnosis. The standard deviation of all three series is std>0.
The data does not contain any 0 or negative values.

What was the solution @DennisToma?


Bug: D3D12: Buffer contents view for matrix is incorrect

Description
When inspecting buffer contents matrices are not displayed correctly. For example: float4x3 is displayed as a float4x4. float3x4 is displayed as a float3x3. Contents for members proceeding a matrix are correct (i.e. the byte offset for that data seems correct). But overall stride seems to be  wrong (e.g. following elements will be shifted when displayed).
Example:
A buffer with format:
struct Data
{
  float4 pos;
  float4x3 transform;
  float4 color;
}

And displayed in the buffer contents view (I've made elements 1 and 2 be the same to demonstrate what's happening):

Element
Position
Transform
Color

0
5.0 6.0 1.0 1.0
1.00  0.00  0.00  2.8226
2.8226 3.0677  2.5881 0.5

0.00  1.00  0.00  3.0677

0.00  0.00  1.00  2.5881

-5.1  10.0  8.09  0.5

1
1.00 0.00 0.00 -5.1
0.00  0.00  2.8226  x
x y z w

1.00  0.00  3.0677 y

0.00  1.00  2.5881  z

10.0  8.09  0.5  w

Looking at Element 0, Transform bleeds into color, but color is still displayed correctly. Looking at Element 1, if you notice, the contents are shifted by one float4. Here, x/y/z/w is bleeding into the position for the element at index 2 (not shown).
If I convert the format to the following:
struct Data
{
  float4 pos;
  float3 transform0;
  float3 transform1;
  float3 transform2;
  float3 transform3;
  float4 color;
}

Then the buffer contents are displayed correctly.

Steps to reproduce

Take a capture of a d3d12 application
Inspect the buffer contents of any buffer containing float4x3/float3x4 data.

Environment

RenderDoc version: 1.7 @ ( a56af589 & f8c5c287 )
Operating System: Windows 10 1903
Graphics API: D3D12
GPU: Nvidia 2080
Driver: 441.66

This was caused by some buggy handling of column-major rectangular matrices, some bits were using the row size and others the major-dimension-size which led to the problems you described. That commit should fix the issues.

Awesome, thanks!


How long between CNAME status checks?  Any way to invoke one manually?

Just wondering....

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f40df04f-087d-8a08-2074-cb259c564ad1
Version Independent ID: 4d489f4e-54d5-c882-6cbd-497f80d2d92f
Content: Get-DkimSigningConfig (ExchangePowerShell)
Content Source: exchange/exchange-ps/exchange/Get-DkimSigningConfig.md
Service: exchange-powershell
GitHub Login: @chrisda
Microsoft Alias: chrisda

@JeremyTBradshaw, unknown. You should open a support case and ask.

Thanks.  Will close this now.


GCC cannot find lua headers on Ubuntu

Xmake Version
v2.8.7+20240401
Operating System Version and Architecture
Ubuntu 24.04 LTS x86_64
Describe Bug
I am trying to compile a simple program inside an Ubuntu Docker container, but the compiler cannot locate Lua header files.
All necessary packages for building are installed inside the container, including lua5.1 and liblua5.1-dev. xmake is installed as instructed on the website. Here is the .Dockerfile contents. If you're not familiar with Docker, it's basically taking latest headless Ubuntu distribution as a base system and runs the commands between ALL_RUN ... ALL_RUN.
# syntax=docker/dockerfile

FROM ubuntu:latest

RUN <<ALL_RUN
add-apt-repository ppa:xmake-io/xmake
apt-get -y --no-install-recommends --no-upgrade update
apt-get -y --no-install-recommends install git ca-certificates build-essential unzip xmake
apt-get -y --no-install-recommends install lua5.1 liblua5.1-dev
apt-get -y clean
groupadd -r developer && useradd -m -g developer developer
ALL_RUN

USER developer

WORKDIR /home/developer

What I end up is a command line version of Ubuntu with xmake, gcc and lua5.1 installed.
Then I try to build this simple program:
#include <lua.hpp>
#include <iostream>

int main(int argc, char* argv[]) {
	std::cout << "Hello world" << std::endl;
	return 0;
}

I've attached the error message as a screenshot. Searching for lua.hpp gives one result in /usr/include/lua5.1/lua.hpp.
So I suspect this has something to do with the header being inside a version folder and that folder is not passed to the compiler as include directory.
Is it an xmake bug where it fails to provide proper include directories? Or is it a known Ubuntu+Lua feature, that requires special treatment? Btw, same thing happens on Debian. Am I doing something wrong here?
I'd appreciate any help.
Expected Behavior
The program is built correctly with xmake -v luatest
Project Configuration
set_xmakever("2.8.6")

add_requires("lua")

target("luatest")
    set_kind("binary")

    add_files("main.cpp")

    add_packages("lua")

Additional Information and Error Logs

you can disable system package. add_requires("lua", {system = false})
or open a pr to improve to find lua from system in xmake-repo.
add on_fetch or add_extsources to improve it.
https://github.com/xmake-io/xmake-repo/blob/dev/packages/l/lua/xmake.lua

@waruqi hello. I've finally returned to research this issue.
There is a number of things to consider. Some systems put all Lua headers inside version subfolder, like /usr/include/lua5.1/ on Ubuntu.
Arch and Manjaro do the same, but they additionally put lua.h, lua.hpp and some other headers of the current version to /usr/include.
Therefore Ubuntu expects that a programmer would use:
#include <lua5.1/lua.h>
And Arch/Manjaro allow using the default system one with:
#include <lua.h>
I checked how CMake handles this, and they have an extensive search through different version folders and some heuristic for choosing the most relevant one. See here: https://gitlab.kitware.com/cmake/cmake/-/blob/master/Modules/FindLua.cmake#L91
This allows using #include <lua.h> on Ubuntu with CMake. CMake would take care of adding correct include directories to project.
Here's the question for Xmake. Does it sound like Xmake's responsibility to automatically provide include directories to most relevant Lua version? Say, if a developer uses add_requires("lua", {system = true})?
What if developer sets specific version for Xmake to use: add_requires("lua 5.4", {system = true})?

we can use pkgconfig to find it, and use pkg-config --modversion to detect version.
if on_fetch return info with version, add_requires("lua 5.4", {system = true}) will match version.
add add_extsources("pkgconfig::lua") or call package:find_package("pkgconfig::lua") in on_fetch, it will detect version if exists

xmake does correctly find Lua in the system. But it doesn't add correct include directory.
xmake adds /usr/include, but what should be added is /usr/include/lua5.1.
I tried to edit the package file at .xmake/repositories/xmake-repo/packages/l/lua/xmake.lua:

placed add_includedirs("/usr/include/lua5.1")
placed package:add("includedirs", "/usr/include/lua5.1") inside on_fetch and inside on_load
When I then build my project, these directories are not passed to gcc.

please do not use add_includedirs, it's only for xmake package, not system libs.
see:

add add_extsources("pkgconfig::lua") or call package:find_package("pkgconfig::lua") in on_fetch, it will detect version if exists

and we can add more extsource for more linux os. e.g. add_extsources("pacman::lua") add_extsources("apt::lua"), it will improve to find lua too.

https://github.com/xmake-io/xmake-repo/blob/6bdd1eb2268ae298fc7e30ee6bd0aec54241037d/packages/g/glib/xmake.lua#L37-L61


[BUG] got a lot of message when requesting both of stories&medias

stories = cl.user_stories(user_id)
print(stories)
medias = cl.user_medias(user_id, target_post_count)
print(medias)
->
Status 200: JSONDecodeError in public_request (url=https://www.instagram.com/) >>> Instagram

Seems ip got rate limit. You can't make many requests without session


Surfaces have no colors

Hello,
I have just installed vis-graph3d.min.js with npm. The surfaces have no colors, they are grey. This is the case for this example, whatever the colors configuration I choose.

Weird, that works in Vivaldi, but not in Chrome.

That also works in Firefox and Chromium, but not in Chrome. Maybe my Chrome version is outdated?

Yeah, I've updated Chrome and this works now. Sorry.

Hi @stla,
can I just ask what version of Chrome was it? Because if it is one from the following list https://browserl.ist/?q=>+0.1%25+or+not+dead then it is a bug that should be addressed.
Thanks.

@Thomaash
It was Chrome 59 (was high time to update it).
I also have this issue in the RStudio browser. I wrapped vis-graph3d in a R package and the surfaces are grey in the RStudio viewer. But maybe my version of RStudio is too old.


Unnamed images after successful test run

Good morning,
I have a test case where i create two docker images using the ImageFromDockerfileBuilder. Everything is working as expected. But when i look into docker desktop after the test run I found two images called <none>. Every time I run the test, two new images (same size) named <none> are added.

The images are created with the following code:
new ImageFromDockerfileBuilder().WithDockerfileDirectory("../../../../ProxyServer")
            .WithName("ProxyServer")
            .WithDeleteIfExists(true)
            .WithCleanUp(true)
            .Build();

I'm not sure if it's a bug or if i miss something.

I am sorry, I cannot reproduce it with your configuration. It just keeps the images with WithDeleteIfExists(false). Which makes sense (it tags the new image with an existing name). See my example below. Can you share some more code or information of your configuration?
public sealed class GitHub : IAsyncLifetime
{
  private IImageFromDockerfileBuilder imageFromDockerfileBuilder = new ImageFromDockerfileBuilder()
    .WithDockerfileDirectory(CommonDirectoryPath.GetGitDirectory(), Path.Combine("tests", "Testcontainers.Tests", "Assets"))
    .WithDeleteIfExists(false)
    .WithName("github:issue602");

  [Fact]
  public async Task Issue602_1()
  {
    await this.imageFromDockerfileBuilder.Build()
      .ConfigureAwait(false);
  }

  [Fact]
  public async Task Issue602_2()
  {
    await this.imageFromDockerfileBuilder.Build()
      .ConfigureAwait(false);
  }

  public Task InitializeAsync()
  {
    return Task.CompletedTask;
  }

  public Task DisposeAsync()
  {
    return Task.CompletedTask;
  }
}

Sure, i've created a new solution testcontainers-test.zip and put all relevant files on it.
If you execute the test you will see one image named <none> (now it's only one image, because in the test it's only one image which is being created).
Maybe my Dockerfile is not correct?

If you run docker inspect ${image_id}, you will see that those "images" are related to your Dockerfile. As far as I know this behavior is intended and a result of the multi-stage Dockerfile, see: https://github.com/moby/moby/issues/34151#issuecomment-316049721.
@kiview How does tc-java handle it? Maybe we can label the intermediate images with the resource reaper session?

@HofmeisterAn I just checked and we don't have a test for a multi-stage Dockerfile build in testcontainers-java. We are adding the labels to the BuildImageCmd, but I can't tell if this means that they will be automatically added to the intermediate images as well, this is something we would need to verify (/cc @eddumelendez).

the behavior is the same in testcontainers-java. After test finished there is a unamed images which serves as a cache for upcoming builds as explained in https://github.com/moby/moby/issues/34151#issuecomment-316049721

I don't know what to now (i'm very new to docker)? Should I remove those images manually? Are you looking for an automated solution?

I do not think you need to bother with those intermediate images (layers). As mentioned in the comment, this is the default behavior of Docker. Docker caches those images to improve the performance.
It is important to understand, that those intermediate images do not take (a lot of) additional space. It will not store the same image twice or something like that.
If you still would like to remove the images you can use one of the suggestions from the discussion on the Moby Project.

With Testcontainers for .NET there is an additional way to get rid of the intermediate images while running the tests:

Add the testcontainers.resource-reaper-session label to each stage (line 3-4, 15-16).
Pass the Resource Reaper session ID to the Docker image build.

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-env
ARG CSPROJ_FILE_PATH="src/WeatherForecast/WeatherForecast.csproj"
ARG RESOURCE_REAPER_SESSION_ID
LABEL "testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID
WORKDIR /app

COPY . .

RUN dotnet restore $CSPROJ_FILE_PATH

RUN dotnet publish $CSPROJ_FILE_PATH --configuration Release --output out

FROM mcr.microsoft.com/dotnet/aspnet:6.0
ARG CSPROJ_FILE_PATH="src/WeatherForecast/WeatherForecast.csproj"
ARG RESOURCE_REAPER_SESSION_ID
LABEL "testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID
WORKDIR /app

EXPOSE 443

COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "WeatherForecast.dll"]

_ = new ImageFromDockerfileBuilder()
  .WithBuildArgument("RESOURCE_REAPER_SESSION_ID", ResourceReaper.DefaultSessionId.ToString("D"))
  .Build();

I do not think there is much more we can do, but we will discuss about it the following week.

@HofmeisterAn your solution works for now. Thanks a lot.

We have decided not to continue working on this topic. Testcontainers for .NET (incl. other implementations) behave like Docker in this case. Detecting the intermediate images by the Resource Reaper would be great, unfortunately the API is not capable of that.
We think the workaround is an acceptable solution for now. If you have anything else in mind just reopen the issue and let us know.