BadHttpRequestException: Reading the request body timed out due to data arriving too slowly

We have an asp.net core 2.1 app running on .net 4.7 in azure web app service.
Recently we started getting A LOT of the following error:

Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.

Our app is under pretty constant heavy load (from incoming webhooks mostly). During big spikes, we tend to see more of these.
We used to run on asp.net core 2.0 / .net 4.6.1 for many months and we never saw that error before.
It seems to have started happening following our recent upgrade to asp.net core 2.1 / .net 4.7.
We'd like to get to the bottom of this issue, but we are not even sure where to start.
Any thoughts?

@sebastienros have you worked up an FAQ wiki yet?

I wonder if it's because if thread pool starvation... Is your application fully async? Do you have any other logs from Kestrel? Can you capture a dump or a profile? Is the application CPU bound or IO bound?

It just happened a couple of times just earlier, usually it's happening for several successive requests in a short time.

So even when there is no apparent bottleneck, and the app is very responsive, it can still happen occasionally.
Yes it is fully async.

Do you have any other logs from Kestrel?

No other warnings/errors. Probably info level but I'm not persisting those, although there is probably a way to get these.

Can you capture a dump or a profile?

I should be able to capture a dump from the azure portal if you think this will help?
Not sure what a profile is?

Any thoughts on how to get to the bottom of this issue?
We're not sure if the errors are caused by legit requests.
Is it possible find out the url that was requested when this message gets generated?
We have been getting a couple of hundreds of these per day, even though all metrics look healthy.
Any particular log source that I can turn on in azure/asp.net core maybe?
The stack trace doesn't give us much.
Exception: Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.
   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.TryRead(ReadResult& result)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.OnConsumeAsync()

.ReadToEnd(); is a red flag, that's blocking sync IO. This data rate error has often been reported in applications that are having trouble with too many blocked threadpool threads. In this case there's an easy ReadToEndAsync() replacement. Have you checked for thread starvation issues elsewhere?

But the stream here is a memory stream. That's why it's using the sync version. No I/O right?
The number of threads is low and metrics are healthy.
This issue seems quite sporadic but we get these errors in batches of 5 or 10...
Could it be caused by the GC causing a pause?

Ah, you got me. I wasn't reading closely enough.

They are pretty small json payloads. Order, Customer, Product, this kind of thing...

What size is pretty small?

Typically under 10k characters, although there a few outliers over 100k characters.
We get a lot of requests though. Roughly 500k per day.

Are there any further logs that can be turned on to get to the bottom of this issue?
This is really polluting our log files and we're still not sure why it occurs in the first place.

We appear to have gotten rid of the issue.
We are not entirely sure but most likely it was caused by some code accessing the request object from a background thread.

We are not entirely sure but most likely it was caused by some code accessing the request object from a background thread.

Using the IHttpContextAccessor? What did that code look like?

Sorry I had to re-open.
I thought it was fixed after our deployment on Sunday evening, but the absence of errors was simply due to the very low activity at this time of the week.
Once Monday kicked in, this error started flooding again.
David, that deployment included a fix related to the IHttpContextAccessor that you help me resolved here. I incorrectly thought this might have fixed this issue as well.
The call stack varies slightly when the error is captued in the log. Usually only asp.net frames, but it also sometimes happen in this new piece of code, which is very similar to the one I posted above handling webhooks:
Simplified a bit for clarity:
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Http.Extensions;
using Microsoft.AspNetCore.Http.Internal;
using Microsoft.Extensions.Logging;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace Middlewares
{
    public class RequestLogScopeMiddleware
    {
        private readonly RequestDelegate _next;
        private readonly ILogger<RequestLogScopeMiddleware> _logger;

        public RequestLogScopeMiddleware(RequestDelegate next, ILogger<RequestLogScopeMiddleware> logger)
        {
            _next = next;
            _logger = logger;
        }

        public async Task Invoke(HttpContext context)
        {
            using (_logger.BeginScope(await GetHttpInfo(context)))
            {
                await _next(context);
            }
        }

        private async Task<HttpInfo> GetHttpInfo(HttpContext ctx)
        {
            try
            {
                var req = ctx.Request;
                req.EnableRewind();
                string body = await new StreamReader(req.Body).ReadToEndAsync();
                req.Body.Position = 0;
                try
                {
                    body = JsonConvert.SerializeObject(JsonConvert.DeserializeObject(body), Formatting.Indented, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore });
                }
                catch
                {
                    //body may not be json
                }
                return new HttpInfo(req.Method, req.GetDisplayUrl(), body, string.Join(Environment.NewLine, req.Headers.Select(kv => $"{kv.Key}: {kv.Value}")));
            }
            catch (Exception ex)
            {
                _logger.LogWarning(ex, "Failed to extract http info from request");
                return null;
            }
        }
    }
}

Logged error:
Failed to extract http info from request

    HttpInfo:

    Exception: Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.
   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.TryRead(ReadResult& result)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.OnConsumeAsync()

I'm pretty sure that it is related to await new StreamReader(req.Body).ReadToEndAsync();

Here is another stack pointing clearly showing the reading of the stream is the issue:
Error from Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware:
    An unhandled exception has occurred while executing the request.

    HttpInfo:

    Exception: Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.
   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
   at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.d__35.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.StreamReader.d__97.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.StreamReader.d__62.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Middlewares.RequestLogScopeMiddleware.d__5.MoveNext()

Yes we are using this in prod.
We haven't suffered any performance issues but I'm listening if you have ideas on how to make this more performant.
Having a log of the body is very useful for investigating why a given request had errors...

Do you have structured logging in place? You don't need to attach this much information to the scope, you just need enough information attached to every request so that you can find all correlated logs. You would log the body in chunks and then look at the entire payload by looking at all logs for a specific request id.

I guess you could say it is structured, yes.
Logging the body in chunks? I'm not too sure what you mean?
Does that mean that I have to reconstruct the body from multiple log entries?
It certainly would be much simpler to have a single entry with the full body.
Also, the vast majority of requests we get are webhooks, that can take various shape (but always in json format) and we must validate that they are authentic by hashing the body.
(see the Webhook() method I posted above)
So we can't use [FromBody] because the payload can differ a lot from request to request.

It's a DoS waiting to happen. I can just send a large payload to your application and boom, out of memory.

I see how this could be a problem.
Should I use something like BigContentManualGood() from your linked sample?
As far as I can tell it still loads the entire json in memory, into a JObject, so I fail to see how this is different?
I guess there must be some sensible max-request-size limit default as well in asp.net core?
Going back to the original issue, I just wanted to point out that the pattern we're seeing is that these errors occur in batch.
There won't be any issues from a while (5 minutes or even up to a couple of hours) and then all of sudden we get many times that error in a rapid succession (say between 3 and 10+).
That may be a useful piece of information.
This got me thinking that it might be caused by a longer that usual GC pause... Just an idea...

What is there a difference between the following two?
await new StreamReader(Request.Body).ReadToEndAsync();
await new HttpRequestStreamReader(Request.Body, Encoding.UTF8).ReadToEndAsync();

We deployed yet again today to try to fix this.
We do not collect the body anymore when logging, based on @davidfowl 's good comments about perf and security.
We removed all usages of HttpAccessor and we are left with the only other piece of code that reads the entire body and causes the error, in our Webhook() action, which now looks like this:
        [HttpPost]
        public async Task<OkResult> Webhook()
        {
            var body = await new HttpRequestStreamReader(Request.Body, Encoding.UTF8).ReadToEndAsync();
            bool isValidRequest = AuthorizationService.IsAuthenticWebhook(Request.Headers, body, _options.SecretKey);
            if (!isValidRequest)
            {
                throw new UnauthorizedAccessException("This request is not an authentic webhook request.");
            }
            //handle webhook
            return Ok();
        }

Using HttpRequestStreamReader hasn't fixed the problem.
We also wanted to rule temporary thread pool exhaustion (even though the thread count counter in the azure portal looks ok) because we thought that the threadpool might take time to ramp up threads, and maybe a spike of requests might cause temporary starvation.
But, it looks like there are plenty of threads and the error is still happening...
public override void OnException(ExceptionContext context)
{
    context.ExceptionHandled = true;
    ThreadPool.GetAvailableThreads(out var workerThreads, out var completionPortThreads);
    ThreadPool.GetMaxThreads(out var maxWorkerThreads, out var maxCompletionPortThreads);
    _logger.LogError(context.Exception, $"Error in global mvc exception filter. Available Threads = ({workerThreads:N0}/{maxWorkerThreads:N0}, {completionPortThreads:N0}/{maxCompletionPortThreads:N0}).");
}

Error in global mvc exception filter. Available Threads = (32,757/32,767, 1,000/1,000).

    Exception: Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.
   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.TryRead(ReadResult& result)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.OnConsumeAsync()

We are running out of ideas.

@clement911 What is your App Service scale?
We are having what looks like the same issue. ASP.NET Core 2.1 app in an Azure App Service (Windows) deployment with similar behavior - fine for a few minutes, then a group of connections all throw this exception, then fine again.
All request are POSTs with a JSON body that is being read and deserialized in a middleware.

Hi @tstuts,
It's a deployment with 3 to 5 S2 instances  and we get a ton of requests, mostly webhooks.
Several 100k per day.
I'm not sure if it is related to that issue, but we also noticed we have been getting a lot more of the following errors. Nowhere near as much, but we used to maybe get it once a day and now we get it a lot more, and I feel like they might be happening around the same time as the orginal issue.
Warning from Microsoft.AspNetCore.Server.Kestrel:
    Heartbeat took longer than "00:00:01" at "11/01/2018 19:48:02 +00:00".

That usually means there's thread pool starvation... Hmm

@davidfowl I don't understand how this is possible given that ThreadPool.GetAvailableThreads is reporting there are plenty of threads available?
We don't create manual threads.
We have a few Task.Run() and also several periodic background jobs scheduled via Observable.Timer, which uses the ThreadPool under the hood.
Here is a screenshot from Azure Portal showing our thread count over the last 12 hours.
I don't believe 200+ threads is excessive?

200 threads is a lot more than you have cores. If those threads are all active then some of them may not be getting enough CPU time (similar to thread pool starvation). We're making some changes in the data rate timers to mitigate these scenarios. https://github.com/aspnet/KestrelHttpServer/pull/3070.

If that is the case then I guess increasing the number of instances might fix the issue.
Pardon my ignorance here but I don't understand.
CPU and memory seems healthy to me (see screenshots below).
We are running S2 instances but Azure portal is recommending S1.
I tried to find out how many cores S2 has but all I can tell is that they have 200 ACU (Azure compute units), whatever that means.
I went through the most recent MinRequestBodyDataRate errors and the difference between the available treads and the maximum threads is always less than 100, so there are less than 100 worker threads out of a supposedly sensible (?) default maximum of 32,757! 100/32757 is less than 1/3 of 1%.
I suppose I could increase the number of instances and maybe this will fix the issue, but I guess I would like to understand what is the bottleneck as it is not obvious to me at all. (rather than throwing money at the problem).

See https://blogs.msdn.microsoft.com/vancem/2018/10/16/diagnosing-net-core-threadpool-starvation-with-perfview-why-my-service-is-not-saturating-all-cores-or-seems-to-stall/. For details on how to diagnose starvation.

Are you using Task.Wait or Task.Result anywhere?

Thank you @davidfowl  that was a good refresher.

Are you using Task.Wait or Task.Result anywhere?

No.
I will keep looking for any blocking calls I might have missed.
By the way we don't use .net core.
We use .net 4.7.1, but I assume that the thread pool works in a similar way.

https://github.com/benaadams/Ben.BlockingDetector

@clement911 yes its fundamentally the same (core is a bit more optimized)

The .NET Threadpool DOES try to inject more threads, but does it at a modest rate (e.g 1 or 2 a second), so it makes little difference in the short term (you need many minutes to get up to 1000).

I wonder if this is a problem. That doesn't sound like a great strategy to handle temporary spikes.
Thanks, I will check Ben.BlockingDetector.
All our webhook method does is validate that it is authentic (CPU bound) and save it in the db with ef core (async IO).
I noticed that when the MinRequestBodyDataRate error occurs, it is likely (but not always) to happen in conjunction with a warning for multiple other webhooks saying the save to the database took an absurd amount of time (say, over 40 seconds and even up to 180 seconds!) to save the webhook data.
I thought it might be the EF core retry strategy that does multiple retries, but as far as I can tell the database is very responsive and I haven't found an event in EF core to log when a retry occurs...
It seems like a weird coincidence so I'll assume it might be due to the same thread exhaustion.

Yes those symptoms sound exactly like there's some starvation going on.  Taking a memory dump when the problem occurs can help verify. If you have that many threads, you can also look at what those threads are doing and it may hint at the problematic area.

What HTTP Status code will be return to the client when a MinRequestBodyDataRate / BadHttpRequestException occurs?
I would think a 4xx? (even though it is likely a internal issue in my case)
We get a lots of these errors in our logs but the Azure portal metrics claims there are hardly any HTTP 4xx or 5xx responses. I don't get it.

The metric below from the dump caught my attention though, but the finalizer thread stack has no user code.

By the way, I was surprised to see that 3 of the threads were taken by the ConsoleLogger MessageProcessor Queue. That got me wondering why it is enabled in prod by default...

So I've been battling with this and wanted to report my findings.
We had to act quickly so and we decided to deploy an exact copy of our app into a new App Service Plan.
Then, we moved all your webhook subscriptions to point to the new instance.
So, the original app service only handles user facing requests and it made it very stable. No errors.
The new app service handles webhooks and DOES NOTHING ELSE.
The new app service is not even exposed to users and the only requests hitting are a simple Webhook() action that does a hash of the body and stores the body in the db by making an async call via SqlCommand with an await cmd.ExecuteNonQueryAsync();
Well, the MinRequestBodyDataRate error still occurs! (on the new instance of course).
Given that there is very little code being exercised in the new instance, I'm confident that we don't accidentally call any sync API or block any threads.
In the logs, we can clearly see that the errors occur when there is a spike of requests.
I'm thinking that the issue may be either in the sql connection pool or the thread pool.
With regular user traffic, one doesn't expect to have huge sudden peak of requests. It usually ramps up slowly and the thread pool seems pretty good at that.
On the other hand, with webhooks, we can sometimes get extremely sudden peaks because some systems somewhere generated a ton of events somewhere.
But the thread pools spawns thread very slowly.
I'm thinking of experimenting with ThreadPool.SetMinThreads and ADO.net 'MinPoolSize' setting.
It might be a good strategy to keep a lot of threads and/or connections running to handle peaks.
If anyone has experience in that area I'm listening.
Those settings are easy to set, but there is very little documentation on how it works behind the scene and it feels very black boxy. Probably a good reason except when you run into a scenario like this...

Sounds like you're on the right track. One alternative we've considered for scenarios like these is request throttling. This would reduce the demand on the threadpool.

Not sure exactly how you're implementing your throttling and how this would solve the problem?
One issue is that most webhook providers (the systems/APIs sending the requests) typically require listener (like us) to respond with a HTTP 200 pretty quickly or they considered it a failure.
In our case it's 5 seconds.

It's only theoretical at this point, but ASP.NET 4 avoided this class of issues using a similar approach. Your requests are not extremely expensive to process, there's just a lot of them and they compete for resources like threads, CPU time, memory, etc.. If you put those requests in a queue and only process a few at a time then there's no longer contention and they can complete quickly. A naive version of this would be a middleware that used a mechanism like SemaphoreSlim to limit the number of concurrently processed requests. As long as everything is flowing smoothly there's no reason you shouldn't be able to handle large bursts in a timely fashion. You just have to keep the burst from crushing you.

@clement911 can you reproduce the issue on the site with webhooks if you hammer it with similar traffic?

@Tratcher that's an interesting suggestion. But if a request waits a SemaphoreSlim, will this count towards the ThreadPool's IO threads.
I think the hard part will be finding the sweet spot of number of threads in the pool to achieve maximum throughput.
@davidfowl both app services host exactly the same code. I tried to reproduce the issue locally with a naive local program issuing request but that didn't work. It's probably possible to reproduce with a proper load testing tool if that is what you are asking.

SemaphoreSlim has an async waiter so it doesn't block threads.

@davidfowl I think I've got a very similar issue with 2.1.5 handling Azure DevOps webhooks in a test web service. Number of requests is low (bursts of 6 only when queuing builds). In my case it repros on maybe 5-10% of bursts.  Again I'm reading/hashing the body to validate against a signature.

@clement911 @mmitche Are you guys seeing a lot of this exception too?

Microsoft.AspNetCore.Connections.ConnectionResetException: An existing connection was forcibly closed by the remote host

@tstuts yes.

I'm guessing that a request that is not getting processed fast enough will lead to the client-side timing out and closing the connection, leading to this exception. Just another symptom of the same root issue. Does that sound right @davidfowl?

Given the very spiky nature of this error, I've also been wondering if it might be Azure running some periodical tasks on the VM, which might be causing this... I remember reading somewhere about some diagnostics jobs that can spike the machine every so often...

I don't recall that exception, but I'm going to test my service a little more now so I'll let you know what falls out.

We're also experiencing this with an API that takes GeoJSON and writes it to an Azure SQL database. The requests come in bursts and there are hundreds of requests at once, because we split the data in batches to avoid hitting request length limits.
It also looks like thread starvation, we're running on a single S3 instance:

I'll try to resolve the problem based on the information in this discussion, and if successful I'll report my findings here.

Not a lot of progress to report so far. I do need to correct my previous comment: we host a number of ASP.NET and ASP.NET Core applications in a single App Service, as Virtual Applications.
I assumed the problem to be in one specific API since that one receives most of the load, but I haven't found any obvious blocking calls. I tried out Ben.BlockingDetector on my local machine, but nearly all reported instances were apparently caused by logging ('ETL' appeared in the stack trace).
I also tried PerfView but didn't succeed in getting the desired output from it.
Regarding the thread count, we reach an average of 180 due to w3wp + the various applications running on it. So perhaps there's not a huge issue in this area. I'll continue load testing and doing diagnostics to locate the source of the problem.


[Question] would this be possible with Luasnip?

Let say I have an if statement where the cursor is currently inside the first condition like;
if x == 1 {
<cursor>
}

of the cursor at the end of the if statement, like;
if x == 1 {

} <cursor>

Would it be possible to create a snippet with the cursor at either locations and have it add another branch to the if statement and have the first node at the new branch condition, like;
if x == 1 {

} else if <cursor>(first node)  {

}

Is there any way to achieve this with Luaship + Treesitter?

Oh, that's possible, for sure:
I whipped up one option for the trigger at the end, since that provides lots of the treesitter-fun (not :P) via treesitter_postfix
treesitter_postfix({
	trig = ".ei",
	matchTSNode = {
		query = "(if_statement) @prefix",
		query_lang = "c",
		select = "longest"
	},
	reparseBuffer = "live" }, {
	d(1, function(_, parent)
		if parent.env.LS_TSMATCH == nil then
			return s(nil, t"")
		end
		-- tricky: remove indent on lines containing LS_TSMATCH. The
		-- indentation is provided by the captured `if`, and should not
		-- be prepended again by us.
		return sn(nil, {
			isn(1, fmt([[
				{} else if ({}) {{]], {t(parent.env.LS_TSMATCH), i(1)}), ""),
			t{"",""},
			sn(2, fmt([[
					{}
				}}
			]], {i(1)})) })
	end)})

I've defined this one for c, it should be possible to adapt to other languages as well, as long as its treesitter-grammar can handle if in its entirety :D

Awesome! Thanks I'll give it a try :)


[Existing API Name] : Marvel API

Brief Description 🤓
I want to add a work to display the working of marvel api
Provide reference!
https://developer.marvel.com/
Select the open source program you are contributing to
Other
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINES of this project.

/gssoc24


My radio isn't working

When im ingame my radio doesn't work i can hear other players in the radio but i can't speak.

Try CAPS


Convert PNG to ICNS automatically

I'd be willing to submit a PR that would allow PNG files to be used as icon input. Are you open to this?

Hi! I think it conversion of pngs to icon input has already been implemented a few versions back

Oh sweet! I misread the README. Thanks for creating and maintaining this great tool, @jiahaog.


Fetch feeds failed - object does not exist...

Hello,
It seems as though the crawler does not work any more. It presents with an exception "Fetch feeds failed", and I've noticed that in the log there is a message saying "Object does not exist, cannot be loaded due to missing permission or reviewable feature...".
I've tried multiple URLs, as well as passing my own AppId - which also produces a "Fetch feeds failed" exception but does not produce the "Object does not exist..." message.
Is this a fault of mine, or did the changes in the Graph API disable the crawler?
EDIT:
I am aware of the changes brought on by Facebook regarding the Graph API and I have gotten a valid access token, or rather an app id and app secret.

Hi @matesicsimun! I'm new to Graph API. I followed the exact instructions and when I run the command facebook_page_crawler $app_id $app_secret $targets $since $until to run the code, I encountered the error "Fetch feeds failed".
I was wondering:

Do I need to modify anything inside the source code to adjust to my case?
What does $target refer to, is it a keyword we put into search, or ID, or ?

Thank you in advanced if you can help with this!

Hello, @daph-td!
I believe the changes made to Graph API are such that this code cannot run anymore.
If you are interested in something such as parsing comments from news sites, perhaps you can contact me by email so I can explain an alternate solution.


Update MS docs: Messaging related page

As title.
Please refer to parent issue(#95) to get more information.

close for pr merged.

Please link all the related PRs here

Please link all the related PRs here

updated.


"Getting Started" links all 404

"Getting Started" section links all 404.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 30921c17-f076-3747-00e4-0615e376a241
Version Independent ID: d637355d-cb58-500f-f05c-d0b48fca4d7a
Content: Introduction to Azure Queue Storage - Azure Storage
Content Source: articles/storage/queues/storage-queues-introduction.md
Service: storage
Sub-service: queues
GitHub Login: @normesta
Microsoft Alias: normesta

@theficus
Thanks for your feedback! We will investigate and update as appropriate.

@theficus
The links seem to be working fine now. Can you confirm once if it is still giving 404 Error.

@theficus
We are going to close this thread as resolved but if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.


Ping pong service

Service which would return a pong to a ping call via regular REST endpoint cal

Push into develop branch.


Time stack memory improvements

This implementation of time stack now only loads two gimages ever at one time. This should mean it can be used with time stacks of hundreds of images.

lgtm!


No frames created on extract frames

Finished extracting frames from video
Exception in thread Thread-2 (swap_face):
Traceback (most recent call last):
  File "/Users/abhishekvakharia/.pyenv/versions/3.12.4/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/Users/abhishekvakharia/.pyenv/versions/3.12.4/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/abhishekvakharia/workplace/Stable diffusion/extensions/face2video/main.py", line 95, in swap_face
    files = os.listdir("extracted_frames/")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'extracted_frames/'

Facing this issue after

added video in input_video
added face image in input_face
and then did run main.py, and clicked on choose face, choose video
After that clicked on extract frames, it shows message extracted successfully
And on swap face it throws the above error.

Hello, this looks like a simple pathing error. This script expects the files to be run in the same folder that the created folders are placed in. I created a version that uses the full system path instead. You can find this updated version on the "issue_10_fix" branch (https://github.com/Arminius-Software/face2video/tree/issue_10_fix). Please let me know if this version fixes your issue.

Thanks for the quick response @Arminius-Software
Made the following changes (https://github.com/Arminius-Software/face2video/compare/main...abhishekvakharia:face2video:fixPaths). it's a bit different because of the file structure on Mac.
It resolved couple of errors but not again stuck with another path issue
Exception in thread Thread-2 (swap_face):
Traceback (most recent call last):
  File "/Users/abhishekvakharia/.pyenv/versions/3.12.4/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/Users/abhishekvakharia/.pyenv/versions/3.12.4/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/abhishekvakharia/workplace/Stable diffusion/extensions/face2video/main.py", line 108, in swap_face
    automatic1111_api.api_change_face(file, self.input_face, input_model, file_path, self.processing_unit, source_choice)
  File "/Users/abhishekvakharia/workplace/Stable diffusion/extensions/face2video/automatic1111_api.py", line 20, in api_change_face
    with open(input_image, "rb") as image_file:
         ^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/extracted_frames/frame144.png'

Am able to see the files created as well it picks up the path correctly but still throws this error when I click on Swap facce

I had to pull your code since this error is not present in my version. You simply need to change: 'file_path = os.path.join(self.main_path, "/extracted_frames/", file)' to 'file_path = os.path.join(self.main_path, "extracted_frames", file)' in main.py swap_face (this fixes the error in Windows at least). Please change your pull request from the main branch to the issue_10_fix branch. Once we have fixed all the errors and you can confirm that it runs without error on macOS, we can change that branch into an official mac version.

if i directly use the branch I see this error
also the extracted_frames folder is empty whereas in the branch I pushed, I was able to create frames in extracted_frames folder
Running] python3 -u "/Users/abhishekvakharia/workplace/Stable diffusion/extensions/face2video/main.py"
Finished extracting frames from video
Exception in thread Thread-2 (swap_face):
Traceback (most recent call last):
  File "/Users/abhishekvakharia/.pyenv/versions/3.12.4/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/Users/abhishekvakharia/.pyenv/versions/3.12.4/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/abhishekvakharia/workplace/Stable diffusion/extensions/face2video/main.py", line 96, in swap_face
    files = os.listdir("extracted_frames/")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'extracted_frames/'

It appears that the issue is that having the "/" or "" symbol causes problems on Mac even if you use the os.path.join function. I can try to replace all those instances with clean names only and then only ever combine things with the os library. The problem here is that I can't recreate the errors that you run into since I don't use macOS.

Yes @Arminius-Software you are right removing / helped. got to the next step,I am able to see the response is successful. but now I don't see anything being saved in finished_frames
Anything which could be possibly updated?
Once we get through this we can mark it as Mac branch

Can you check the response in the automatic 1111 console? If the reactor fails to swap/save the image, then it should be displayed there. Also check if the response.status_code returns 200.
Once everything works i will merge all your commits into a Mac Version branch and of course give you credit in the readme as well.

yes responses are fine not sure they are not being saved, will test that out. and one it works I will push it to my remote branch and you can take it up from there.
Thanks for the support!

Hi @Arminius-Software,
I was able to generate video successfully made few changes in the path, I have it on fixPath as well as on main. you can decide how you want to manage branch for Macs.
I would push some changes later.
TODO:

put check if the path is already in finished_frames and finished_videos then do not change the directory which is done manually using chdir()

Awesome, I have opened a Mac-Version branch and merged your changes. It would be great if you could confirm that everything was merged correctly and the code on the Mac-Version branch runs on your system. If there are still changes to be made, you can just open another pull request.

@Arminius-Software it works verified, and we are good to close this.
Thanks a lot

os.path.join(current_directory, "extensions", "face2video", "finished_frames")
This line was causing issues for me, as I do not have F2V as a subdirectory of A1111
A1111 console:
23:17:33 - ReActor - STATUS - Restoring the face with CodeFormer (weight: 0.5)
23:17:34 - ReActor - ERROR - Error while saving result: [Errno 2] No such file or directory: '/Users/user1/extensions/face2video/finished_frames/frame52.png'

This code worked for me:
current_directory = os.path.dirname(os.path.abspath(__file__))
path = os.path.join(current_directory, "finished_frames")

@buffoff thank you for your feedback. I am assuming you are also using macOS? The pathing behaves differently than on Windows. I never had any issues with the face2video folder not being a subfolder of automatic1111 on Windows. Maybe @abhishekvakharia can check if the changes that you suggested also work on his machine. It would be nice if you could open a pull request into the Mac branch for that.

@Arminius-Software will check and update, will wait for @buffoff to clarify if he is on Mac, also @buffoff can you share your directory structure

@buffoff thank you for your feedback. I am assuming you are also using macOS? The pathing behaves differently than on Windows. I never had any issues with the face2video folder not being a subfolder of automatic1111 on Windows. Maybe @abhishekvakharia can check if the changes that you suggested also work on his machine. It would be nice if you could open a pull request into the Mac branch for that.

Apple Silicon, macOS Sonoma 14.6.1, Python 3.12.5

@Arminius-Software will check and update, will wait for @buffoff to clarify if he is on Mac, also @buffoff can you share your directory structure

.
├── face2video
│   ├── __pycache__
│   ├── extracted_frames
│   ├── finished_frames
│   ├── finished_videos
│   ├── input_faces
│   ├── input_faces_models
│   └── input_videos
└── stable-diffusion-webui
    ├── __pycache__
    ├── cache
    ├── config_states
    ├── configs
    ├── embeddings
    ├── extensions


JSDoc for interactiveTarget

Added JSDoc typing to the interactiveTarget.js file
Moved hitArea out as a property from DisplayObject, since it's an interactive only property
Allowed JSDoc to now show within the DisplayObject page that there can be the interactiveTarget mixin. Previously there was no way of knowing these properties existed as interactiveTarget wasn't generated anywhere in the documentation at all.
DisplayObject now shows:

And clicking on that link takes you to:

👍


configure terraform link fixed

There were a pair of extra parentheses ( in the configure terraform link in this page, which were causing a 404 for that page.
Removing the parentheses seems to have fixed the issue

fixes #1639

Thank you @mrityunjaygr8 !


Validate filter values before calling registry.

This moves the call to the registry/hub for search command after the validation of the filters 🐯.
This way, if the filters aren't valid, we don't call search anymore :angel:
Closes #23016.
/cc @runcom @thaJeztah @coolljt0725
🐸
Signed-off-by: Vincent Demeester<EMAIL_ADDRESS>

LGTM :+1:

LGTM


export setupMSVCDevCmd to allow library usage

@ilammy This exports the function that performs the installation. It allows me to use this function directly.

@aminya, you don't need this to be published on npm or anywhere, as you'd just include this code via submodules, right?


增加weid各组件版本兼容说明链接

Summary

Concisely describe the issue and your solution:
Link to existing issue (type "#" to check):
Specify the tests done so far:

Solutions (Optional)
Please describe the details of your solution if it might be too complicated for reviewers.

Codecov Report
Base: 43.37% // Head: 43.37% // No change to project coverage :thumbsup:

Coverage data is based on head (c49f538) compared to base (a8e6a38).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff             @@
##             develop     #347   +/-   ##
==========================================
  Coverage      43.37%   43.37%
  Complexity      1608     1608
==========================================
  Files            237      237
  Lines          12783    12783
  Branches        1663     1663
==========================================
  Hits            5544     5544
  Misses          6540     6540
  Partials         699      699

Impacted Files
Coverage Δ

...main/java/com/webank/weid/service/BaseService.java
43.02% <0.00%> (-2.33%)
:arrow_down:

.../main/java/com/webank/weid/config/FiscoConfig.java
70.37% <0.00%> (+1.23%)
:arrow_up:

.../com/webank/weid/service/impl/AbstractService.java
44.44% <0.00%> (+5.55%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


new version, more redaction types, tests

The following types of strings are currently redacted:

Certificates and private keys
Electron application resource directories (Windows, OS X, Linux)
Generic user directories (Windows, OS X, Linux)
Email addresses
IP addresses
URLs
MongoDB connection strings

This looks great! I had a few specific questions about the regex methodology, but other than the nits this is awesome!

published in v0.1.0


Add rel=me to the link in footer.html

Add rel=me value used to indicate profile equivalence.
Relevant documentation about rel=me here and here.

Thanks! I'll merge your other PR after trying it out when I'm back on my computer shortly.


Investigate re-write into a Babel plugin

This won't be a problem for today or for MVP - but eventually we're going to get to scaling problems and it might be worthwhile re-writing to be a Babel plugin.
But! It also might be fine. So let's find out over time.

to help migrate / mitigate works, maybe use babel plugin macros?

Yeah we could definitely write a babel macro today as well. Would you be interested throwing up a spike? (Using the current babel plugin as a base?)

sweeeet! so I dont know how to take next steps, plus I'm pretty busy right now (everyone wants web things 🤦♀) so if anyone wants to work on this too let me know, otherwise ill see if i can find some time to make a stab at a PR.
Also, we use jira so I understand the idea of "spike" but dont know how you guys use that in github 😂... so let me know if there are next steps i need for that.
If anyone else is interested in helping please @ me so I can make sure we coordinate / dont duplicate work.
thx fo the quick response @madou !

all good no pressure :smile:
the spike would be just putting a PR up that has a proof of concept of babel macros working with the current code 👍

Don't we already have the plugin which internally uses ts-transform?

Yeah we do so there's no major need to go down this path. This is more here for a "what if" scenario 😄

ok we're going down this path 😄


produce empty tiles when querying outside of area of use of projection (wip)

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

50 of 56 (89.29%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.01%) to 84.118%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

operators/src/processing/reprojection.rs
48
54
88.89%

Totals

Change from base Build<PHONE_NUMBER>:
0.01%

Covered Lines:
23691

Relevant Lines:
28164

💛  - Coveralls

 Fixing Projections outside of area of use and general overhaul of GeoTransform and TilingStrategy


Single Choice Dialog with wrong coloring when selecting items

Steps:

Create a Single Choice Dialog with three items.
Select the first one.
After select another one.
Radio stroke color is wrong. Stroke color must be gray (or default color for disabled options).

https://github.com/afollestad/material-dialogs/issues/564


Posts all appearing in the home directory

Hi All, kinda new to using Sculpin. great work by the way, nice and easy to use. I just have a question or two in regards to posts.
When I generate a post, I follow the guidance on the sculpin.o site, where it states to cd into the source directory then the _posts directory. Then touch with the name of the file (year-month-day-name-of-file.md). When I run sculpin generate and watch for any changes whilst the local server is running, the home page has all my posts on it. I'm wondering if there is a way for me to only show one post at a time?
Any advice would be greatly appreciated.

Hi there,
When using sculpin-blog-skeleton, the homepage will show the 3 most recent posts and provide pagination to access older posts.
If you want it to only show the latest post, edit the source/index.html file so that the top matter looks like this:
---
layout: default
title: Home
generator: pagination
pagination:
    max_per_page: 1
use:
    - posts
---

By changing max_per_page to 1, the homepage will only show the newest post, and provide links to view older posts.
Hopefully this answers your question.

Hi Beryllium, thank you for your reply and it has helped me out a treat, Is this currently on the sculpin guiance? Greatly appreciated the help here.
Kind regards,
bladedev


[MC] AWS - Drain nodes before rolling

[MC] AWS - Drain nodes before rolling

We could use https://docs.aws.amazon.com/autoscaling/ec2/userguide/configuring-lifecycle-hook-notifications.html#sqs-notifications with a modified version of the azure-scheduled-events app

Also check https://github.com/aws/aws-node-termination-handler

First version of the app is ready, first version of terraform is ready.
Need to test upgrading from older releases to this one and creating a new installation from scratch.

[ ]  akita
[ ]  alpaca
[ ]  anaconda
[ ]  anchovy
[ ]  anteater
[ ]  antelope
[ ]  argali
[ ]  bandicoot
[ ]  eagle
[ ]  flamingo
[ ]  fox
[ ]  gaia
[ ]  gauss
[ ]  ginger
[ ]  giraffe
[ ]  gorilla
[ ]  kudu
[ ]  otter
[ ]  tiger
[ ]  valkyrie
[ ]  viking
[ ]  viper
[ ]  visitor

no permission on otter

@paurosello we need to ask the customers owning the unticket MCs in the previous comment for the sqs:* permission in the Policy bound to the GiantSwarmCD role.
Can you please drive that?
Here is a possible request:
In order to make future management cluster upgrades more reliable and less disruptive, we would like to deploy an upstream component named `aws-node-termination-handler` (see https://github.com/aws/aws-node-termination-handler).
Such component uses the SQS service to automatically drain nodes being part of an ASG before the ec2 instance is terminated.
We kindly need you to grant us the `sqs:*` permission in the IAM Policy attached to the `GiantSwarmCD` role.

@paurosello I think we have already the needed permissions for a* clusters as both GiantSwarmAdmin and GiantSwarmCD have AdministratorAccess. At least for non-china installations.

Still waiting for permission from VF and BS to finally deploy in their MC as well

deployed everywhere


Support GraphQL Subscription in Ballerina GraphQL Client

Description:
Currently, the Ballerina GraphQL client does not support subscription operations. We should be supporting it through a WebSocket client.

Closing this as the proposal is added via #3560


cmake: silence "WARNING: ASSERTs enabled" when CONFIG_TEST is true

TEST configurations don't need to be warned that they're using test
techniques with some side-effects.
On a typical sanitycheck invocation, this warning is one of the only two
that appears in most test runs. In other words this commit gets rid of
half of the entire grep -ri '[[:blank:]]warn' noise that obscures any
work-in-progress warnings or platform specific warnings in the
logs (typically: device tree warnings).
Signed-off-by: Marc Herbert<EMAIL_ADDRESS>

There are 13 samples/ with CONFIG_ASSERT=y that still produce this warning. Any objection to adding CONFIG_TEST=y to their prj.conf ?
git grep CONFIG_ASSERT= -- samples/

samples/basic/threads/prj.conf:CONFIG_ASSERT=y
samples/net/mqtt_publisher/boards/cc3220sf_launchxl.conf:CONFIG_ASSERT=y
samples/net/sockets/echo/boards/cc3220sf_launchxl.conf:#CONFIG_ASSERT=y
samples/net/sockets/echo_async/boards/cc3220sf_launchxl.conf:CONFIG_ASSERT=y
samples/net/sockets/http_get/boards/cc3220sf_launchxl.conf:CONFIG_ASSERT=y
samples/net/sockets/sntp_client/boards/cc3220sf_launchxl.conf:CONFIG_ASSERT=y
samples/philosophers/prj.conf:CONFIG_ASSERT=y
samples/philosophers/prj_tickless.conf:CONFIG_ASSERT=y
samples/portability/cmsis_rtos_v1/philosophers/prj.conf:CONFIG_ASSERT=y
samples/portability/cmsis_rtos_v1/timer_synchronization/prj.conf:CONFIG_ASSERT=y
samples/portability/cmsis_rtos_v2/philosophers/prj.conf:CONFIG_ASSERT=y
samples/portability/cmsis_rtos_v2/timer_synchronization/prj.conf:CONFIG_ASSERT=y
samples/sensor/ti_hdc/prj.conf:CONFIG_ASSERT=y # validation

@jukkar ^^ (you may have an opinion about the net samples)
@marc-hb CONFIG_TEST sets native_posix to default to non-real time (meaning blazing fast). Samples like the philosophers app running in that mode by default has already confused people in the past. So if you go that route, you may want to at least add a native_posix board specific CONFIG_NATIVE_POSIX_SLOWDOWN_TO_REAL_TIME=y

samples/net/mqtt_publisher/boards/cc3220sf_launchxl.conf:CONFIG_ASSERT=y
samples/net/sockets/echo/boards/cc3220sf_launchxl.conf:#CONFIG_ASSERT=y
samples/net/sockets/echo_async/boards/cc3220sf_launchxl.conf:CONFIG_ASSERT=y
samples/net/sockets/http_get/boards/cc3220sf_launchxl.conf:CONFIG_ASSERT=y
samples/net/sockets/sntp_client/boards/cc3220sf_launchxl.conf:CONFIG_ASSERT=y

No objections from me for adding CONFIG_TEST to above files.

CONFIG_TEST sets native_posix to default to non-real time (meaning blazing fast). Samples like the philosophers app running in that mode by default has already confused people in the past

Thanks @aescolar for catching this. I tried the philosophers examples in warp speed and it was... entertaining :-) I found something much more confused than myself: sanitycheck. Warp speed philosophers make sanitycheck hang forever, it never times out. The infamous python pipes deadlock maybe? Wild guess. So I would have discovered this anyway - but the harder way.

native_posix board-specific CONFIG_NATIVE_POSIX_SLOWDOWN_TO_REAL_TIME=y

I don't how to add a board-specific CONFIG_ in a test, is that possible? POSIX_SLOWDOWN does make other boards complain and fail as you anticipated.

No objections from me for adding CONFIG_TEST to above [cc3220sf_launchxl.conf] files.

Thanks! However when scanning test logs I realized these configs are not used by default. The only ones actually making warning noise are:
samples/basic/threads/
samples/sensor/ti_hdc/prj.conf
samples/portability/cmsis_rtos_v1/timer_synchronization/

samples/philosophers/
samples/portability/cmsis_rtos_v1/philosophers/
samples/portability/cmsis_rtos_v2/philosophers/

native_posix board-specific CONFIG_NATIVE_POSIX_SLOWDOWN_TO_REAL_TIME=y

I don't how to add a board-specific CONFIG_ in a test, is that possible? POSIX_SLOWDOWN does make other boards complain and fail as you anticipated.

Place native_posix specific config options into boards/native_posix.conf file in the test app.


Enable bevy/png for the async-collider feature in the examples

Objective
When running the async colliders example, the following warnings are printed, and Ferris does not appear.
2023-10-27T23:27:51.794954Z  WARN bevy_gltf::loader: Error loading glTF texture: You may need to add the feature for the file format: failed to load an image: The image format Png is not supported
2023-10-27T23:27:51.794964Z  WARN bevy_gltf::loader: Error loading glTF texture: You may need to add the feature for the file format: failed to load an image: The image format Png is not supported
2023-10-27T23:27:51.794967Z  WARN bevy_gltf::loader: Error loading glTF texture: You may need to add the feature for the file format: failed to load an image: The image format Png is not supported

Solution

Enable bevy/png for the async-collider feature in the examples

Async colliders don't need PNGs, that model just happens to use textures that do. We'd have the same issue if the model had e.g. a JPEG or WebP image texture. I'd say it makes more sense to enable that feature for Bevy in examples_common_3d/Cargo.toml and to not have it enabled for bevy_xpbd itself.


Constructued DiscordRPC class gets destroyed when doing anything related to Sessions in Unreal Engine.

When hosting or joining a session. The DiscordRPC class gets destroyed/becomes invalid and it won't be able to send presences or check for callbacks anymore. OnDisconnected & OnErrored does not fire either when this happens.
However, if you construct the class again after it's become invalid everything seems to work like normal again. OnConnected does not re-fire when you reconstruct the class.

It might be garbage collected. In the UDiscordRpc::Initialize function, try adding AddToRoot();, that prevents Unreal from garbage collecting anything :) Report back if it works!


Add 'description' as Input property on aws-apigateway:index:RestAPI

Hello!

Vote on this issue by adding a 👍 reaction
If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Hi, please add 'description' as an Input property on aws-apigateway:index:RestAPI. This is very helpful when trying to identify items in the AWS console. the awsx provider doesn't expose it.
Error: Property description does not exist on 'aws-apigateway:index:RestAPI'
      on Pulumi.yaml line 144:
     144:       description: ${pulumi.stack}
    Cannot assign '{stageName: string, description: string, routes: List<{localPath: string, method: string, path: string}>}' to 'aws-apigateway:index:RestAPI':
      Existing properties are: routes, stageName, swaggerString, apiKeySource, requestValidator and 2 others

Many thanks in advance
James
Affected area/feature

Thanks. For future readers, relates closely to #117 . If we could do both properties at once that would be great.


Not possible to mock relative paths after upgrading to 0.4.x

I might not have properly analyzed this, but version 0.4.1 is not working in a couple of projects and I think it comes down to handling of relative URLs.
My projects run in a Web / Web extension context so there's a defined base URL to resolve relative URLs against. As a result fetch('folder/file.json') is valid and I've been mocking that in my tests.
However, it seems like with the switch from cross-fetch to native Node fetch, we force the fetch parameters into a native Request object here: https://github.com/IanVS/vitest-fetch-mock/blob/66aeefd26577e48693df2c79dd232cb8217fbebb/src/index.ts#L312
Apparently, Node's implementation of the Request constructor doesn't handle relative URLs since there's no base URL to resolve against and this will throw with a TypeError ("Failed to parse URL from folder/file.json"). As a result the fetch fails.
I'm not really sure what the correct solution is. I can probably work around it in my projects by detecting when we are running tests and supplying a dummy base URL but it would be nice if that wasn't necessary and we could maintain compatibility with 0.3.0. Perhaps we could wrap native Request somehow?

This really is a bummer. It's a bit silly that constructing a Request with a relative URL is prohibited in Node.js. I would expect this error only to be thrown when the Fetch API is actually invoked.
I have several solutions in mind:

detect any non-absolute URL and convert them into absolute ULRs vitest-fetch-mock; e.g. foo becoming http://<IP_ADDRESS>/foo. However, this trick would not entirely be "invisible" to users, since the Request object is passed to the ResponseProvider function argument of vitest-fetch-mock. This seems to be solution with the least effort, but we need to investigate the impact.

use a custom Request class, e.g. from a polyfill like cross-fetch. This is rather annoying though, since this would introduce another dependency that we just got rid of. Note that other polyfills like node-fetch also seem to have the same limitation. Alternatively, we could extend Request ourselves:

/**
 * A custom Request implementation that permits relative URLs
 * to enable non-Node.js use cases.
 */
class MockedRequest extends Request {
    private inputUrl: string;

    constructor(input: RequestInput, init?: RequestInit) {
        const inputUrl = input instanceof Request ? input.url : input instanceof URL ? input.toString() : input;
        const patchedInput = input instanceof Request ? input : input instanceof URL ? toAbsolute(input.toString()) : toAbsolute(inputUrl);
        super(patchedInput, init);
        this.inputUrl = inputUrl;
    }

    // @ts-expect-error We know what we're doing
    override get url(): string {
        return this.inputUrl;
    }
}

/**
 * Converts non-absolute URLs into absolute URLs
 * @param url
 */
function toAbsolute(url: string) {
    if (!new RegExp('^(?:[a-z+]+:)?//', 'i').test(url)) {
        return `http://<IP_ADDRESS>/${url}`;
    }
    return url;
}

we change vitest-fetch-mock to stay away from constructing actual Request objects by changing the normalization step. This would however lead to breaking changes, like changing the ResponseProvider interface to accept a different "response like" object instead of the actual Response. At this point, I'm would not be in favor of this since it's a lot of effort with a lot of impact.

@IanVS what is your opinion?

Bummer indeed.  I think we could probably take the same approach as MSW here: https://github.com/mswjs/interceptors/commit/f9193cd74602bbd78c30086651b6a456d8662fd0, using location.origin when possible.  This will mean that users will need to have something like jsdom in place, but if they're running browser tests in node, they should already.  (Actually they should use Vitest browser mode instead, but that's another issue).

Well, what about the Vitest browser mode? Isn't that exactly meant to solve environment-specific issues like this?

Yes, though it is still marked as "experimental" and perhaps not all projects can use it.  I think this tool should still work with jsdom + node, since that's a much more common approach still.

I'll work on this to try to unblock folks.  It looks like the :+1: are piling up.

The fix has been released in 0.4.2.


Streaming updates

Refactoring to make changes to the localStorage & sessionStorage streaming.

@staltz @TylorS this is ready for a review if you fancy it?

I mainly reviewed the driver source and the example. Looks ok to me!

Cool, thanks! I'll just go ahead and merge away.

:+1:


PowerToys Run Smaller Font Size

Description of the new feature / enhancement
The powertoys run is a great feature for sure but it would be even better if you could minimise it a bit. The font size is too big so changing just the font size would be a great improvement in my opinion.
Scenario when this would be used?
for everyday life a smaller font is better. the current font makes it look comical or a rip-off of something.
Supporting information
No response

Tracked in https://github.com/microsoft/PowerToys/issues/4794


feat(logger): use RUST_LOG-like EnvFilter for logging

Description
Replace level filter with EnvFilter which support setting specific log level for different iroha modules.
Examples
The same can be done through configuration in config.toml.
# set global logging level to debug
LOG_LEVEL="debug" ./scripts/test_env.py setup
# set global logging level to error and info for iroha_core
LOG_LEVEL="error,iroha_core=info" ./scripts/test_env.py setup
# receive only logs from sumerag with trace level or heigher
LOG_LEVEL="iroha_core::sumeragi" ./scripts/test_env.py setup
# receive all logs at trace level, except axum for which use error
LOG_LEVEL="trace,axum=error" ./scripts/test_env.py setup

Linked issue

Closes #4829

Benefits
More granular control over logging.
Downsides
It's harder to check correctness for example if someone would write LOG_LEVEL=infa iroha won't return any error because this would be parsed as accept logs from module infa at trace level.
I believe we can mitigate this, by custom parsing, but it would make our type more strict than orginal EnvFilter.
It's smt i would prefer to leave for separate PR.

It's harder to check correctness for example if someone would write LOG_LEVEL=infa iroha won't return any error because this would be parsed as accept logs from module infa at trace level.

why is default TRACE?

idk, lowest possible level is assumed by tracing crate


Nodes that are disjoined stick around in previous state

When you disjoin a node, the node_states metric doesn't remove the ID of the node.
We should have a disjoined state in the controller API or have the exporter notice it's gone and remove it/add it to a disjoined status. The exporter has a local state it keeps that we could try and access.

I need to track down how to access the memory and remove entries that don't exist.

Solved with metric.Reset()


[Feature Request]  Wrist Overlay shows Queue Position?

Explain in detail what your suggested feature would be used for.
I see that VRCX will shows where are you in the queue in the application itself on the top why not put it onto the Wrist Overlay?
Describe how it would look if it requires a UI.
I'd suggest at the top of the Overlay having a permanent section just for Queue status so it's easier to see.
Explain why people would want to use it.
So it's easier to see ur queue status without seeing ur desktop or go into the VRC Menu ofc

https://github.com/vrcx-team/VRCX/issues/581
The reason I ignored this request in the past was because the API doesn't send enough information about your current queue status to reliably use it, for example when you cancel your queue in-game VRCX can't tell that it was canceled.

Ah That make sense why the tab is still there even tho I already close VRChat xD
Maybe can have a timer if the queue doesn't change for like a few minutes and consider it's canceled?


Add concatenate to BatchedArray

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (bcfe6af) 100.00% compared to head (664f6e5) 100.00%.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #583   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           28        28
  Lines         2302      2325   +23
  Branches       291       296    +5
=========================================
+ Hits          2302      2325   +23

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Favorites/Monitor triggered on program launch even when interval hasn't passed.

I am on 7.6.2. I have a large number of tags favorited, and my favorites.json file has every tag set with intervals of 86400, 259200, and 158400. So 24 hours minimum.
All of a sudden on these newer versions, I have noticed that when Grabber launches, no matter how long it has been since the last time it autodownloaded all the tags, it will start autodownloading all of the tags. It acts like I haven't run the program in multiple days. If I let it run, and it updates the favorites.json file with the "lastCheck" for all the tags, and then i immediately close the program when it finishes and relaunch it, it does all of them all over again. This did not happen on older versions of grabber, so something likely changed when it comes to the monitor running that is triggering this to occur.

Hmm, maybe this is related to  #2401?
I just noticed the setting "Use system locale for dates and numbers." Maybe this is the problem, like its reading the date wrong in one spot and always assuming its been forever since the last run.
Also, just noticed that if i de-select that setting and hit OK and then re-open settings, it shows selected again...

Doesn't seem to be related to that setting, at least not on the face of it. If i change useSystemLocale in settings.ini to false and re-launch, it still immediately starts downloading everything.

It was caused by the "check now" button that was "on" on startup for some reason. 👍


Add \n support for image.print #865

What's Changing and Why
While printing, it now starts on a new line in case it encounters a newline character in the text.
What else might be affected
Every text that contains a newline, will be effected. Before it just printed a space instead of a newline.
Tasks

[x] Add tests

N/A

[ ] Update Documentation
[ ] Update jimp.d.ts
[ ] Add SemVer Label

I like to ensure that I have provided all necessary elements for its completion. Could you please guide me on any additional steps or documentation required to finalize the pull request successfully?


Generating MDX from Google style docstrings

Motivation
We would like to produce beautifully formatted MDX files for the Docusaurus v2 website built from the Python docstrings. The current system extracts all symbols, builds a stub for each module, class, function, but does not parse and format Google style docstrings.
Changes proposed
This PR will do extra MDX formatting based on interpreting Python docstrings according to the Google docstring format. Also, it fixes some miscellaneous issues with MDX formatting.
Future PRs will improve the layout/alignment of the MDX formatting and add docstrings for all public symbols.
Test Plan
Run
python scripts/generate_api_docs.py
cd website/
yarn start

and evaluate output in web browser.

Codecov Report

Merging #93 (b31dca7) into main (45e91ca) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main      #93   +/-   ##
=======================================
  Coverage   98.23%   98.23%
=======================================
  Files           6        6
  Lines         227      227
=======================================
  Hits          223      223
  Misses          4        4

Flag
Coverage Δ

unittests
98.23% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

tests/test_imports.py
86.36% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 45e91ca...b31dca7. Read the comment docs.


Convert map.txt to generate GridSpace combos instead of individual Tile/Blocks

Right now the map.txt generates Blocks and only Blocks according to Block ID's. This is nice and all, but it doesn't work for when we get Tiles in the game so we'll need a different flatfile format to hold those. Instead of tracking them per-block in this file I figure I'll just hardcode (for now) some block and tile combinations in the new MapGenerator. This way -  in the future - MapLoader can read the new maps.txt format, figure out what kind of map it wants, tell MapGenerator to say, generate a corridor or something, then MapGenerator will go through and figure out the proper type of Block/Tile combo to create the GridSpaces to do so.
This means we can store the new maps as a matrix that directly corresponds to the GridSpace grid, as ID's that reference individual GridSpaces. And just let the MapGenerator figure out the rest. Which greatly simplifies the map storage/generation from flatfile.
It still might be nice to store the different types of GridSpaces as we currently have them as GridSpace types, so perhaps just rename the current map format into gridtypes.txt or something and refactor the MapLoader classes accordingly.

Just gonna hardcode this for now... going to also disable the map loading in MapLoader (which might require some work to preserve the Block auto-generation we rely on right now, since we don't load Block prefabs yet) and just wing it until this is absolutely necessary

From another ticket:
I would like the map.txt file to be changed to a blockgridspaces.txt file, which will reflect it's existence as a list of types of GridSpaces made from Blocks. The file would then be loaded in and indexed, and instead of serially loading the GridSpaces into the world as we do now, the MapGenerator could then use said indices to build the map in parts (rooms, corridors, etc). We call it blockgridspaces instead of just GridSpaces since we will also have a tilegridspaces.txt which will have a list of indices for the arrangement of tile prefabs. Creation of GridSpaces would be easier with some sort of external tool, perhaps we should open a ticket for that. For now you can just write them with the text file and keys provided.

Once the MapLoader pulls in the prefabs of Tiles/Blocks, attaches them to a Tile/Block object, and then configures GridSpaces out of said GridObjects according to blockgridspaces.txt and tilegridspaces.txt the MapLoader should then store these GridSpaces by index in an object pool : see #24

Commit e8c18d897752605d524bd3a61cd2f819c9773ea9 has completely moved the MapGenerator over to the new XML defined GridSpace configuration IDs. It's only a trivial matter to make this work for Block based ConfigIDs as well, but I'm not prioritizing this because it's not extremely necessary right now.
So this is basically complete. Just needs to migrate map.txt to a blockgridspace.xml file.


Best way to globally override Job defaults ?

Hi,
I'd like to change the default parameters for all my Jobs (e.g. generate a key via the job name and its arguments, setting the ttl to -1, increasing the timeout, etc.), but I have a bunch of jobs (10+).
I thought about defining sort of a default_job_kwargs and passing it to enqueue(), but it looks easy to forget and thus error-prone. Or I also thought about defining my jobs as instances of Job instead of functions, and using this default_job_kwargs here instead of when enqueueing them, but then again, easy to forget when defining a new job.
Is there a preferred way to globally override the defaults instead of passing those arguments every time I enqueue/implement a job ?
If that's a feature you think would be good to have, do you have an opinion/idea about how it should be implemented ?
Thanks for the help!

The way I do this is as you described, I have a default job kwargs that I override when enqueueing jobs. Another way you can do it is to make a subclass of job with the defaults you want. i don't think it's necessary for another global way to do this. But happy to chat more.

Having SAQ configuration for certain defaults might make sense to me (e.g. default_ttl or default_timeout). Maybe as attributes on the Queue.
Others might be messier (e.g. "generate a key via the job name and its arguments"). That sounds like it depends on some application specific logic. There is currently a "before_enqueue" hook for Queues where you could inject such logic... but that feels like overkill.
What about regular ol' composition?:
class MyQueue:
    def __init__(self, saq_queue):
        self.saq_queue

    async def enqueue(self, job_or_func, **kwargs):
        default_kwargs = {
            "ttl": -1,
            "timeout": ...,
            "key": get_job_key(job_or_func, **kwargs)
        }
        default_kwargs.update(kwargs)
        return await self.saq_queue(job_or_func, **default_kwargs)

Or regular ol' inheritance?:
class MyQueue(Queue):
    async def enqueue(self, job_or_func, **kwargs):
        default_kwargs = {
            "ttl": -1,
            "timeout": ...,
            "key": get_job_key(job_or_func, **kwargs)
        }
        default_kwargs.update(kwargs)
        return await super().enqueue(job_or_func, **default_kwargs)

In my experience, complex layers of global configuration is more "error prone" than just setting the config explicitly in code. I also think it's often a good practice to wrap broad third party interfaces with narrow application-specific interfaces.

Thanks for the quick answers, I ended up going the inheritance way: with this approach there's only one place to define those default kwargs in my app.


feat: OpenAPI integration

related #855
cargo check -F openapi
cargo nextest run --test-threads 1 -F openapi

TODO:

[x] config.yaml

maybe no enable: true since that's handled by the featureopenapi:
  redoc:
    !Redoc
      url: /redoc
      spec_json_url: /redoc/openapi.json
      spec_yaml_url: /redoc/openapi.yaml
  scalar:
    !Scalar
      url: /scalar
      spec_json_url: /scalar/openapi.json
      spec_yaml_url: /scalar/openapi.yaml
  swagger:
    !Swagger
      url: /swagger-ui
      spec_json_url: /api-docs/openapi.json
      spec_yaml_url: /api-docs/openapi.yaml

[x] finish .merge(Redoc::with_url("/redoc", api.clone()))
[x] openapi.josn and openapi.yaml endpoints for all types

[ ] split feature openapi into feature swagger-ui redoc scalar
[x] SecurityAddon

put impl Modify for SecurityAddon somewhere, maybe with config
set the jwt token location

[x] tests

[x] update src/tests_cfg/db.rs:86:1
[x] src/tests_cfg/config.rs
[x] config from file test_from_folder_openapi()
[x] snapshots

[ ] docs
[ ] codegen/document adding #[derive[utoipa::ToSchema)] if used in utoipa::path
[ ] maybe auto fill / wrap utoipa::path if possible
[ ] check that get in get(get_action_openapi) is still grabbed with routes!(get_action_openapi)
[x] fix AppContext - check that api_router.routes(method.with_state::<AppContext>(())) doesn't break the ctx with .layer
[ ] add .set for JWT_LOCATION.get_or_init so cargo test works, nextest works correctly with JWT_LOCATION.get_or_init

cc @DenuxPlays

Looks like a good start.
I think the yaml also should contain a few other things:

openapi and ui endpoints (I am not sure if this is want you mean in your description)
whether  it serves json, yaml or both (Perhaps this can be controlled by using features to exclude unnecessary dependencies)

Hey @NexVeridian
Can I help you finish this pr?

@DenuxPlays yeah of course

How can I help?
Also just one Note to your pr description.
I wouldn't wrap the utoipa path macro.
Its working very well just how it is and wrapping it would make upgrading and maintaining very difficult

Also just one Note to your pr description. I wouldn't wrap the utoipa path macro. Its working very well just how it is and wrapping it would make upgrading and maintaining very difficult

thanks!
this would be nice if possible:
impl Routes {
pub fn add_openapi(mut self, method: UtoipaMethodRouter<AppContext>) -> Self {}
}

any of the unchecked ones would be great:
maybe cargo loco generate controller --openapi and docs first


Update runtime-options.js

General Changes:

Color of blocks changed to a grey as blue in Scratch commonly associated with "motion".
"Framerate" on blocks re-specified as "framerate limit" to avoid confusion.

New blocks:

get clone # : reports the amount of clones on the stage

set username to () : changes the runtime username

run green flag : simulates a green flag click
Newer Thumbnail Idea
I didn't commit this but here's also a thumbnail to match the other main TW extensions

I don't know how, but ok.

Oh also, I don't know why, but whenever you change the stage size, the quality drops. But this also happens when you turn on fullscreen.

set username is fine, color change is fine, framerate rename is fine, green flag is marginal but fine, but not convinced we need another clone # block

Oh also, I don't know why, but whenever you change the stage size, the quality drops. But this also happens when you turn on fullscreen.

probably not extension's fault, report separately

@LukeManiaStudioss Maybe add isTerminal: true for the green flag?

Alright, that's fine, we can remove that


Fix very small potential issue

Fix the issue mentioned in #483.

@housseindjirdeh just getting started 😜


Fix qt5-creator and qtwebkit builds with  QT 5.15+

Update qtcreator to 4.12 which is required to get working with 5.15 even though 4.12 is still in RC1 phase
Use a more (5.212) maintained branch for qtwebkit, which is required to working it with QT 5.15

@schnitzeltony please review the qtwebkit one

I cannot test it now (am on warrior & dunfell). If there is a later branch building with 5.15 and builds without python2 I am very happy about it. Thanks!

@schnitzeltony meta-qt5/master (and master-next) is compatible with dunfell
OK, I'll merge it and we'll see who complains.

I cannot test it now (am on warrior & dunfell). If there is a later branch building with 5.15 and builds without python2 I am very happy about it. Thanks!

yes it works with py3

Hope this gets merged soon.

In master now.


What exactly does the trending.py file return? Is it the trending hashtags in my area?

It depends on what region of the Internet you are using!

Ah okay, thank you! Also, are the returned videos the ones that initially appear on a new account created in that region?

Those are the videos that are trending in that area, there may be old videos or new videos


blade forelse's empty case true if a variable isn't passed

Laravel Version: 5.3.22
PHP Version:
Database Driver & Version:

Description:
Please let de empty case of a forelse be true when a certain variable isn't passed so you don't have to put an if(!empty( function around your forelse.
Steps To Reproduce:
When i do this:
                @forelse ($category->categoryGallery as $image)

                @empty
                @endforelse

i get the error:
Invalid argument supplied for foreach() (View: /......../category.blade.php)
but when i do:
            @if(! empty($category->categoryGallery))
                @forelse ($category->categoryGallery as $image)

                @empty
                @endforelse
            @endif

it works, while the whole idea about forelse should be that you don't have to use an if empty check anymore.
Bart

see https://laracasts.com/discuss/channels/general-discussion/forelse-doesnt-seem-to-work

Why not @brunogaspar ?

@BartHuis If i understood your request correctly, it means that if no variable is defined it'll silently fail, which is bad and will lead to developers wonder why the foreach doesn't work as sometimes a simple typo can lead to this type of situations and if it silently fails you have no clue where to look first.
If i understood it incorrectly, my apologies.

only if you use a forelse, not with foreach. when defining it as forelse you're choosing to go through when something isn't passed. now the behavior is strange in my opinion, because when the blade isn't getting one of the variables i want it to not do anything with it, as i explained in the laracasts topic.

So your $category->categoryGallery is null at that time,  but you still want the forelse to evaluate properly?

forelse uses a regular php foreach loop under the hood, you should put this in mind while passing data to this directive.
Closing since this isn't actually a bug report but rather a feature request, please continue on the internals repo.
For suggestions and feature requests please use the https://github.com/laravel/internals repo.

A simple fix for your issue, since your $category->categoryGallery returns null
@forelse ($category->categoryGallery ?: [] as $image)

@empty
Nothing to show here
@endforelse

But either way this type of logic belongs on the app level not on the framework side.

So your $category->categoryGallery is null at that time, but you still want the forelse to evaluate properly?

Yes, because when i fill category with 20 different items if they excist, you'll want to show them if they excist...

@forelse ($category->categoryGallery ?: [] as $image)

yeah, thats a way too...


[GLUTEN-4012][CH] Add metrics for GenerateExecTransformer

What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
(Fixes: #4012)
How was this patch tested?
manual tests

please add a test case for this change

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query
log/native_4014_time.csv
log/native_master_12_25_2023_e7040cd12_time.csv
difference
percentage

q1
32.18
31.71
-0.475
98.52%

q2
25.75
25.25
-0.492
98.09%

q3
37.85
37.88
0.028
100.07%

q4
40.11
39.70
-0.415
98.97%

q5
72.91
72.46
-0.445
99.39%

q6
5.41
7.15
1.738
132.14%

q7
87.36
87.64
0.276
100.32%

q8
87.15
87.41
0.265
100.30%

q9
122.81
126.55
3.739
103.04%

q10
43.68
42.96
-0.725
98.34%

q11
20.00
19.94
-0.053
99.73%

q12
25.73
23.98
-1.749
93.20%

q13
46.47
47.08
0.607
101.31%

q14
17.87
14.65
-3.220
81.98%

q15
27.64
28.24
0.596
102.16%

q16
15.21
15.59
0.387
102.54%

q17
103.58
104.91
1.327
101.28%

q18
153.69
151.05
-2.635
98.29%

q19
13.93
12.72
-1.215
91.28%

q20
27.00
28.28
1.284
104.76%

q21
230.10
225.43
-4.667
97.97%

q22
13.83
14.57
0.732
105.29%

total
1250.25
1245.13
-5.115
99.59%


geni-convert-logs does not check whether it succeeded at inserting entries

I ran:
/usr/bin/geni-convert-logs -f cvs_history.aslund

Because the idp_account_actions table hadn't yet been initialized, this had no effect.  However, the script gave me no indication that it had failed, and simply returned 0.
Obviously, i'll notice during the live migration if all'' submissions fail, but the script still needs to do error-checking so i can get some indication if ''some submissions fail.
Imported from trac ticket #25,  created by chaos on 09-08-2013 at 16:31, last modified: 10-03-2013 at 17:15

I'm declaring this OBE (and effectively fixed by another ticket).
Trac comment by chaos on 10-03-2013 at 17:15


Analysis Metadata in config.yaml?

Thinking towards the examples repository, could we put relevant metadata in the config.yaml to indicate provenance of the yaml file?
Preliminary suggestion:
provenance:
  author: John Smith III
  email<EMAIL_ADDRESS>  date: 2020-04-01
  reference: Genome-Wide Analysis in Vivo of Translation with Nucleotide Resolution Using Ribosome Profiling, Ingolia et al 2009
  DOI: 10.1126/science.1168978
  notes: >
    Re-analysis of data from Ingolia 2009 to an updated yeast transcriptome.

Request comments from @mikej888 and @FlicAnderson

I think its a good idea. Users can add any additional content they wish to the YAML, RiboViz will ignore anything it does not use. Providing a schema for the provenance, with examples in YAML we write ourselves would be helpful for others.

An update, to accommodate multiple authors, their website, and version TAG or COMMIT hash of the version of RiboViz they ran their script under.
provenance:
  authors:
  - author: John Smith III
    email<EMAIL_ADDRESS>  - author: ...
    email: ...
  website: https://www.ed.ac.uk/some-bio-project
  date: 2020-04-01
  riboviz-version: TAG | COMMIT
  reference: Genome-Wide Analysis in Vivo of Translation with Nucleotide Resolution Using Ribosome Profiling, Ingolia et al 2009
  DOI: 10.1126/science.1168978
  notes: >
    Re-analysis of data from Ingolia 2009 to an updated yeast transcriptome.

Will this need to be complemented with documentation, for their example, to capture information about all the data files used within their example and any changes they may have made? (i.e. along the lines of https://github.com/riboviz/riboviz/blob/master/docs/reference/data.md)

Yes, complemented with documentation, and perhaps checked (optionally) in #172 ?

Put it into To Do, although some progress has been made, because we are not scheduling it yet.
It needs some documentation, @FlicAnderson opined.

Still requires the documentation for this, ties into #201 documentation of full size example datasets perhaps?

We have been putting metadata in config.yaml files for months, with instructions in example-datasets README.md. It seems to be working.
Shouldn't we close the issue?

I've created a PR (https://github.com/riboviz/example-datasets/pull/60) that removes a problematic symbol from the config yaml example in README.md on example-datasets.
This should close the issue I think, since the metadata info is great otherwise and being used without problems


After connecting to Ambire Wallet, I try to sign a message and I get an "Invalid address" error

Describe the bug
After connecting to Ambire Wallet, I try to sign a SIWE message with SignerCtrl from @web3modal/core and I get an "Invalid address" error.
SDK Version
"@web3modal/core": "^2.0.0-beta.3",
"@web3modal/ethereum": "^2.0.0-beta.3",
"@web3modal/ui": "^2.0.0-beta.3",
To Reproduce
Steps to reproduce the behavior:

Copy WalletConnect link from Web3Modal;
Choose Ambire Wallet on Web3Modal;
Go to wallet.ambire.com, paste the WalletConnect link on "Connect dApp" and successfully connect wallet to dapp;
Try to sign a message with SignerCtrl.signMessage({ message: "..." }) from "<EMAIL_ADDRESS>Get "Invalid address" error.

Expected behavior
The message should be successfully signed and complete sign in on my dapp
Screenshots

Desktop (please complete the following information):

OS: [e.g. iOS] MacOS Ventura 13.0.1
Browser [e.g. chrome, safari] Brave
Version [e.g. 22] Version 1.45.118 Chromium: 107.0.5304.91 (Official Build) (arm64)

Additional context
The very same message can be signed without issues with MetaMask injected wallet and with Exodus iOS wallet via WalletConnect.

There were some substantial changes in beta.4, please see new docs that moved to https://docs.walletconnect.com/2.0/introduction/web3modal/about
We are no longer exposing our own hooks and instead moved to fully support Wagmi for much better and reliable dev experience.
For vanilla users we are now relying on methods from @wagmi/core package instead of our controllers. Please see updated example for html as well, as initial setup steps changed. We are working with wagmi team on documentation for their package, meanwhite you can inspect functions that are exported by the package. (similar to our controllers, they have things like getAccount / watchAccount / getSigner etc.)

Hello @IljaDaderko . Thank you so much for your response.
I've upgraded my SDKs according to the vanilla example, but I still get the same error. I've opened a ticket on Ambire support and they suggested that it might be caused by wagmi doing its own signature validation, which doesn't support Smart Contract wallets. They sent me this link https://blog.ambire.com/ambires-signature-validator-library-unlocks-smart-wallet-support-for-dapps/
Looking at wagmi issues I could confirm that smart contract wallets signature validation is not supported by the package yet https://github.com/wagmi-dev/wagmi/discussions/90. Should Ambire and other smart contract wallets be removed from the list of wallets supported by web3modal until wagmi becomes EIP-1271 compliant?

@blocknomad I'm not too familiar with the topic, but if you just use raw signer (from their useSigner) would that work? As it exposes standard ethers instance.

@IljaDaderko I just tried what you suggested and it indeed doesn't work. Then I checked out ethers documentation and they say that Contract-based Wallet or Meta-Transaction-based Wallet message signing isn't supported https://docs.ethers.io/v5/api/signer/#Signer--signing-methods. Recently, they have also added warnings for it saying that developers need to implement EIP-1271 compliance separately if they want to support wallets such as Gnosis and Ambire https://github.com/ethers-io/ethers.js/pull/3481

I see, well if wagmi does not implement that soonish, you can always use modal in "standalone" mode (no wagmi / lightweight mode) that works with any dev-tooling.
here is an example of this with our v2 sign sdk https://github.com/WalletConnect/web3modal/blob/V2/examples/react-standalone/src/pages/index.tsx

@IljaDaderko this also means that web3modal woud be broken with Argent, Gnosis Safe, Sequence and others.
Supporting smart contract wallet signatures is vital IMO


Replace unnecessary use of .clone() with Arc<T> in background task loops

https://github.com/vexx32/thread-tracker/blob/582e6924dd172f15bb5c1413e47e931412b1aebb/src/background_tasks.rs#L39-L75
All library functions which need the Context and Database inputs work with shared references. These functions were written to clone them instead and pass them around by-value simply to avoid lifetime hassles with the use of Futures that may never return.
We can improve this situation by wrapping these inputs in Arc<T> and cloning that reference pointer instead, reducing the unnecessary copies.

Resolved in bda081629bae34fa0394b30d699636470052c708


Fix IndicesRequestCacheIT flaky tests

Description
This makes an attempt to fix two flaky test in IndicesRequestCacheIT mentioned here
I was not able to reproduce this even when running these tests for more than 500 iterations. So pretty rare. But the fix is based on a pretty good assumption that it will work based on past experience as well.
Details below:

testCacheWithFilteredAlias: This fails at a point when it expects cache hits but gets none(because items got invalidated). It gets invalidated because of background refreshes/merges which was also observed in rest of the tests in same file, and those don't report any issues now after the fix. So the fix is to disable refresh/merge and should work.
testTimedOutQuery: This tests tries to deliberately time out the query to test cache functionality and validate. Here we create a deliberate time out of 500ms(by Thread.sleep()) at a certain step(query.createWeight()) and set a timeout of 0ms. But 0ms is pretty aggressive and sometimes query times out before even reaching above step. For the fix, we increase the timeout to 5ms, this logic is the same which is being used in SearchTimeoutIT which works fine.

Related Issues
Resolves #[Issue number to be closed when this PR is merged]

Check List

[x] Functionality includes testing.
[x] API changes companion pull request created, if applicable.
~[ ] Public documentation issue/PR created, if applicable.~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@sgup432 - Seems you're missing the lucene version upgrade changes from main:
[Test Result](https://build.ci.opensearch.org/job/gradle-check/49251/testReport/) (1 failure / ±0)

    [org.opensearch.qa.verify_version_constants.VerifyVersionConstantsIT.testLuceneVersionConstant](https://build.ci.opensearch.org/job/gradle-check/49251/testReport/junit/org.opensearch.qa.verify_version_constants/VerifyVersionConstantsIT/testLuceneVersionConstant/)

@dbwiddis

LGTM as an attempt to fix. How are we tracking the follow-up (does this work? If not should we revert?)

We will be tracking the subsequent failed tests as part of https://github.com/opensearch-project/OpenSearch/issues/14288. If we don't see any cases where above tests fail, I think we should be good. Otherwise need to revisit and try again.


KeyError in method get_requested_aosp_permissions

Describe what you wanted to do

is this error specific to a single file? no
if possible give a minimal working example where the error happens

a, d, dx = AnalyzeAPK("some.apk")
a.get_requested_aosp_permissions()

line 873, in get_requested_aosp_permissions
    if perm in list(self.permission_module["AOSP_PERMISSIONS"].keys()):
KeyError: 'AOSP_PERMISSIONS'

Describe what actually happened
The dict self.permission_module is indexed by android permission name (e.g. android.permission.SEND_SMS); looks like the mapping is build in a wrong way
System Information

Androguard Version: 3.2.1 (latest from pip3)
Python Version: 3.6
Operating System: Ubuntu 18.04

Mhh I think that all permissions in the permission module are the AOSP permissions... thus, removing ["AOSP_PERMISSIONS"] would give the correct result IMO

Hello, I meet the same problem, I try my best, but I can't solve this problem. Could you give me some suggestions? Thanks.

Hello. I faced the same problem. @reox is right: all the AOSP permissions are in permission_module, so replacing in androguard/core/bytecodes/apk.py at line 874:
if perm in list(self.permission_module["AOSP_PERMISSIONS"].keys()):

by
if perm in list(self.permission_module.keys()):

fixes this bug and indeed returns AOSP permissions:
$ androlyze.py foo.apk
Loading apk foo.apk...
Please be patient, this might take a while.
Androguard version 3.2.1 started
In [1]: a.get_requested_aosp_permissions()
Out[1]:
['android.permission.ACCESS_NETWORK_STATE',
 'android.permission.READ_PHONE_STATE',
 'android.permission.INTERNET']

@lukhio if you like, please send in a PR!


high system cpu usage when streaming container stats

We are using Nomad to orchestrate our docker containers. We observed abnormally high cpu usage on our machines even when containers are idle (services receive no requests at all and do nothing).
We observed it with different docker versions on different operating systems.
It turned out that high cpu usage is caused by Nomad streaming stats from docker containers.
For every container Nomad launches a process which receives stats for this container using streaming docker api (?stream=1).
The more containers are running, the more noticeable extra cpu usage is.
The problem is not Nomad specific, it can be easily reproduced with pure docker.
Just launch ~50 containers and then execute docker stats command for every running container.
Launch 50 containers with redis:
#!/bin/bash
for i in `seq 1 50`; do
   docker run -d redis
done

So far there will be no cpu usage, because redis is idle and doing nothing.
Then launch docker stats for every container:
#!/bin/bash
for id in `docker ps --format "{{.ID}}"`; do
    docker stats $id >/dev/null &
done

Now there will significant cpu usage by docker-containerd:

Also there will be periodic spikes of la which, i assume, originate from lock contention. There is high system cpu usage so, since there is no I/O, i assume it is futexes.

Is Nomad doing it wrong and stats api is not supposed to be used like this, or is there a bug in docker that affects performance?
(the corresponding issue in Nomad repo: https://github.com/hashicorp/nomad/issues/3662)

Going to close this one for now.  This is from the 0.2.x branch that is no longer supported.  With 1.0 we have a prometheus api that can dump all the container stats at once.


[feature] add inheritance for specified keys

I utilize BigQuery's column-level access control by setting policy_tags in numerous model.yml and source.yml files within dbt.
It would be beneficial if I could inherit policy_tags.
To this end, I have introduced a feature that allows not only the inheritance of policy_tags but also specified keys.

Very nice PR, LGTM. Will trigger new release

Thank you for merging 😀


Day name dosen't change when local change and reload

When i change local of calendar and reload it. It dosen't change the  name of days but change the name of months. but when i am creating new calendar and change the local and then set delegate it change the name of day.

Thank you, it's fix.