Add array syntax for route definitions.

This allows for control over the matching order of routes in order to control the precedence of some routes over others.  I've based this off #16 where @corytheboyd provided a good starting point.  However, there didn't seem to be any activity on this front and my team is also in need of this feature.  Hopefully this can begin the process of adding ordered routes to this excellent library.

Looks great @paulcoyle. Thanks for putting this together!

I'll work on getting a release out very soon.

My pleasure @jfairbank; thank you for a great library!


Remove named tensors support

The named tensor support in pytorch is lagging behind with other developments in pytorch, so it would be better to remove the named support.

Related to #85, and likely also for pytorch 1.9.

Fixed.


FileLines::all().to_json_spans() returns [] which is parsed as empty map

This surfaced at https://github.com/rust-lang/rls/issues/1463 (fixed in https://github.com/rust-lang/rls/pull/1497).
The underlying data structure discerns a missing/"all" state in addition to actual set (possibly empty) of lines:
https://github.com/rust-lang/rustfmt/blob/26d370e9e52f5c2727b7289e339499aef0b7b3ec/src/config/file_lines.rs#L151-L157
whereas to_json_span only encodes a specified set of those changes (can't represent FileLines(None)):
https://github.com/rust-lang/rustfmt/blob/26d370e9e52f5c2727b7289e339499aef0b7b3ec/src/config/file_lines.rs#L218-L221
I think we should either:

panic/return err on None match in to_json_spans (since vec![] returned for FileLines(None) won't be parsed back into the FileLines(None) but rather FileLines(Some(HashMap::new))
Somehow discern or accept null lines in --file-lines JSON and make to_json_spans() return Option<Vec<JsonSpan>>

Thoughts?
cc @jsgf @ljw1004

since vec![] returned for FileLines(None) won't be parsed back into the FileLines(None) but rather FileLines(Some(HashMap::new)

I think that it would be simpler to fix this part instead, vec![] should be parsed to FileLines(None)?

This would mean that --file-lines [] would correspond to FileLines::all() - wouldn't that also be surprising?

Yeah, you are right. Actually rustfmt --help=file-lines explicitly states that Specifying an empty array will result in no files being formatted.
In that case, I would prefer to go with your second option.

If we go with @Xanewok's second option, then it will introduce an update to the public API (FileLines:: to_json_spans), which in theory requires a major version update of rustfmt. However I am not planning to release 2.0 anytime soon, so the fix will be postponed for a few months. This does not sound great.
Rather, we could implement Serialize and Deserialize on FileLines to fix the issue, and deprecate FileLines::to_json_spans. This does not introduce any breaking changes.
A different approach would be to relax the stability guarantee of rustfmt so that 'small' breaking changes can be introduced with a minor version update. This sounds fine, as the main (or the only?) consumer of rustfmt-as-a-library is rls.

I'd probably lean towards the last option since it's more of a bugfix (comment mentions serializing into format accepted by --file-lines JSON but in this case the None match is wrongly deserialized to []).
However, having a (De)Serialize impl sounds like a useful thing on its own.
What's puzzling is the difference between representation - file-lines accepts Vec<(File, Range)> while internally (which makes more sense) it's stored as HashMap<File, Vec<Range>>. Is it for backwards-compatibility and the internals were switched at some point?
I'd probably find it puzzling to see a slightly different Serialized data layout to the one used internally (when the original could be trivially serialized as-is), so I'm -0.5 on the fix using these impls.

@topecongiro I can take a look at this one

Opened #3954 that implements the second option detailed by @Xanewok
I didn't include the Serialize and Deserialize implementations for FileLines as it wasn't clear to me from this thread if we wanted to do so, and it also seems like we're already using those to bubble up some errors:
https://github.com/rust-lang/rustfmt/blob/7713d05c5bda4fb3bc3b01fd18b335b96d114d8a/src/config/file_lines.rs#L333-L354


Python error handling

In case of errors we should

print all output, so we can add print() calls for debugging
handle it right for the "Create graph in Python" box too.

This seems work on local instances, but not on try and pizzabox.


fix: release package

#281  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @MartinBernstorff and the rest of your teammates on  Graphite

Merge activity

Jul 29, 1:47 PM GMT+2: @MartinBernstorff merged this pull request with Graphite.


[BUG] disabling Harvester node driver should not remove its CRD and rke-machine-config

Rancher Server Setup

Rancher version: v2.7-head
Installation option (Docker install/Helm Chart): Helm

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE2

Proxy/Cert Details:

Information about the Cluster

Kubernetes version: v1.24.4+k3s1
Cluster Type (Local/Downstream): local

If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider):

User Information

What is the role of the user logged in? Admin

Describe the bug
After disabling the Harvester node driver, it will delete the harvesterconfigs.rke-machine-config.cattle.io resources and delete the harvestermachines.rke-machine.cattle.io CRD, which will cause the cluster to be unreachable and users can also not create a new RKE2 cluster.
To Reproduce

Import the downstream Harvester cluster via the virtualization page.
Spin up a new RKE2 k8s cluster using the Harvester node driver and wait for the cluster to be ready.
Disable the Harvester node driver on the Drivers -> Node Drivers page.
The existing cluster becomes unreachable with HarvesterConfig.rke-machine-config.cattle.io "fleet-default/nc-gc-test-pool1-n477p" not found
Also the user can't create a new RKE2 cluster again since the harvestermachines.rke-machine.cattle.io CR is pending on delete.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    objectset.rio.cattle.io/id: dynamic-driver-crd
    objectset.rio.cattle.io/owner-gvk: management.cattle.io/v3, Kind=DynamicSchema
    objectset.rio.cattle.io/owner-name: harvesterconfig
    objectset.rio.cattle.io/owner-namespace: ""
  creationTimestamp: "2023-03-15T08:41:15Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2023-03-31T15:12:26Z"
  finalizers:
  - customresourcecleanup.apiextensions.k8s.io
  generation: 1
  labels:
    auth.cattle.io/cluster-indexed: "true"
    cluster.x-k8s.io/v1beta1: v1
    objectset.rio.cattle.io/hash: 46b06b88ff348746623bcd87cd8d121a864a831a
  name: harvestermachines.rke-machine.cattle.io
  resourceVersion: "23536171"
  uid: 4dd97ebe-fab6-4fd2-a3fe-539d8ee4f2b3

Expected Result
disabling the Harvester node driver should not remove its CRD and related harvesterconfigs.rke-machine-config.cattle.io resources.
Screenshots

Additional context

SURE-6103

same issue for digitalocean node driver

Root cause
No validation when disabling/deleting node drivers -> leading to CRDs getting destroyed and then cleanup (deleting VMs)
What was fixed, or what changes have occurred
Updated the webhook to validate that there are no machines that are managed by the node driver when attempting a disable/delete -> if there are the request is denied.
Areas or cases that should be tested
Create both RKE1 and RKE2 clusters one at a time and try disabling the node driver while the machines are there. It should fail if there are any nodes using the driver.
What areas could experience regressions?
n/a
Are the repro steps accurate/minimal?
Yes.

Test Environment:
Rancher version: v2.7-head 58aede6
Rancher cluster type: HA
Docker version: 20.10
Downstream cluster type: rke2 Harvester node driver cluster

Testing:
Tested this issue with the following steps:

Import the downstream Harvester cluster via the virtualization page.
Spin up a new RKE2 k8s cluster using the Harvester node driver and wait for the cluster to be ready.
Disable the Harvester node driver on the Drivers -> Node Drivers page.

Results
The existing cluster is still reachable and able to interact with.


feat: call afterQuery only when query is fully executed

Make sure that:

[x] You have read the contribution guidelines.
[x] You have created a feature request first to discuss your contribution intent. Please reference the feature request ticket number in the pull request. #94
[x] You use the code formatters provided here and have them applied to your changes. Don't submit any formatting related changes.
[x] You submit test cases (unit or integration tests) that back your changes.

Issue description
The afterQuery method is called even if the query is not executed. This can happen when binding is used.
New Public APIs
No new public API.
Additional context
This PR is a fix for issue #94.
With this PR the afterQuery function is called when all result are consumed, this mean that the query is fully executed.

@ttddyy I created a new object QueriesExecutionCounter (I can't find a good name)
I tried this code with this project https://github.com/deblockt/r2dbc-issue-reproducer all seem good.

Hi @deblockt,
Thanks a lot for this PR.
I have merged this PR to my local branch and adding some more changes.
I'll try to push them this week.

Hi @deblockt
I have pushed all the changes to  gh-94_0.8.x_after-query branch in my repo including a merge from your PR.
Also, I have tested with your repro project and it is working as expected.
Please double check the changes in the branch. If all looking good, I'll merge to 0.8.x branch.
Thanks for your PR and repro, they were very helpful.

Hi @ttddyy,
All seem fine on my side, I tested it on my real project and all is ok.
Thanks for your work :)


The connection was closed because the response was not read by the client at the specified minimum data rate

Hello,
I have a .net 4.6.2 project (for my web apis) that is referencing, among other libraries, Microsoft.AspNetCore 2.1.3. I recently upgraded it from version 1.1. Since the upgrade I experienced the following issues on my production win 2016 server:

my api logs often show the following messages:

Connection id ""xxxxxx"", Request id ""yyyyyy"": the connection was closed because the response was not read by the client at the specified minimum data rate

the request timed out because it was not sent by the client at a minimum of 240 bytes/second".

[Warning] Connection processing ended abnormally.
Microsoft.AspNetCore.Connections.ConnectionAbortedException: The connection was aborted
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.
d__1881.MoveNext()--- End of stack trace from previous location where      exception was thrown ---   at     System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at     System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task     task)    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.    <ProcessRequestsAsync>d__1871.MoveNext()

The apis become unresponsive (they are hosted on iis 10 on a windows 2016 server)

IIS logs show, at the same time, 502 errors while trying to call the api endoints

I also noticed heavy peaks of the CPU.

Remarks: the web api load has not increased and there haven't been any other major changes, apart from the upgrade of the  Microsoft.AspNetCore that I mentioned above, so I am still trying to figure out what the problem could be. Any advice that could point me to the right direction?
Thanks
Lorenzo

Upgrade to 2.1.4 and revisit https://github.com/aspnet/Announcements/issues/316

Hi Ben,
thanks for you reply: I tried but I realized that in my api project, which has been created using .NET 4.6 + .NET core (see first screenshot), the Nuget Packages manager doesn't show any release after the 2.1.3 (see second screenshot).
I also noticed that the upgrade to 2.1.4 is instead available for "pure" .NET core projects. Do I need to install anything on my developer machine?

Ah, good point. Add a reference to (or upgrade) System.IO.Pipelines to 4.5.1 instead

Ok, thanks; I just added the System.IO.Pipelines refrerence (seems it wasn't installed before).

seems it wasn't installed before

It would have been brought in as a dependent reference from one of the other libraries.
Not sure if it will fix the data rate issue you are seeing; but it could fix the CPU spikes; and at a stretch they could be related. So see if it still happens.

Thanks again Ben; do you think that referencing the System.IO.Pipelines directly as I did (but still having the Microsoft.AspNetCore 2.1.3 referenced, which seems not using it) will make Kestrel to load  it anyway?

Yes, Kestrel uses System.IO.Pipelines and adding a reference to 4.5.1 will cause it to use the patched version instead.

Unfortunately the problem occurred again even after the upgrade; I enabled the failed requests trace (see below) and I noticed that the response body's data buffers are read fine up to a certain point, and then the 502.3 error is raised by the  AspNetCoreModule (so it seems that is not the database query that is running slow, but the way the retrieved data is processed by the underlying layers).
Moreover, the api logs also reveal the following error: Connection id ""xxxxyyyyy"" bad request data: "Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate".

Hi ben,
just to let you know that I found this: https://github.com/aspnet/KestrelHttpServer/issues/2104 which seems to describe exactly my situation. I indeed have some error handling methods that call something like this:
string  _result = ReadRequest(context).GetAwaiter().GetResult();
Could this cause the mentioned prtoblems? Adding something like this to main():
        var host = new WebHostBuilder()
            .UseKestrel()
             .UseLibuv(options =>
             {
                 options.ThreadCount = 1000;
             })
             .UseContentRoot(Directory.GetCurrentDirectory())
            .UseIISIntegration()
            .UseStartup<Startup>()
            .UseApplicationInsights()
            .Build();

could also help mitigate the problem?
Thanks again

Wondering if the synchronous Read could deadlock the read for regular sockets; which will then get killed off due to its data rate dropping to 0 byte/s /cc @halter73

    options.ThreadCount = 1000;

This will create 1000 managed threads outside the ThreadPool which is way too much. Generally, we find the best performance comes from having about logical core count / 2 libuv threads. I don't recommend changing the libuv ThreadCount unless you've done extensive benchmarking proving it's better for all your scenarios.
As far as synchronous reads go, I'd advise against it since it can contribute to ThreadPool starvation. ThreadPool starvation (or even having 1000 libuv threads) is a possible cause request body rate timeouts if the starvation is so bad that read callbacks don't get scheduled in a timely manner.
I don't think a synchronous read could cause a deadlock in 2.1. There was a synchronous read deadlock in our 2.2 preview with HTTP/2 but that's in code that doesn't exist in 2.1. For HTTP/1.x, we always dispatch for reads unless you use a pubternal API to set the ApplicationSchedulingMode to Inline which you should never do if you're mixing sync and async I/O.

I started seeing a similar, sporadic issues yesterday in one Azure App Service and now am seeing it in a second:
 Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.PumpAsync()   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)   at System.IO.Pipelines.Pipe.GetReadAsyncResult()   at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.ReadAsync(Memory1 buffer, CancellationToken cancellationToken)   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory1 buffer, CancellationToken cancellationToken)   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.Read(Byte[] buffer, Int32 offset, Int32 count)   at System.IO.StreamReader.ReadBuffer()   at System.IO.StreamReader.ReadToEnd()   at ServiceStack.Text.JsonSerializer.DeserializeFromStream(Type type, Stream stream)   at ServiceStack.Serialization.JsonDataContractSerializer.DeserializeFromStream(Type type, Stream stream)   at ServiceStack.Host.Handlers.ServiceStackHandlerBase.CreateContentTypeRequest(IRequest httpReq, Type requestType, String contentType)
I've only seen it happen on http POST operations.  I upgraded one of the App Service's code base to use 2.1.4 (and verified no transitive dependencies were pointing to the affected versions) and am still seeing the issue.  So far, it's not been reproducible locally, but it is sporadic, so I'll keep trying.
As a workaround, I could configure kestrel to increase the MinRequestBodyDataRate, but that's obviously not a long term solution.
This issue is reminiscent of #2151 .

@noelstieglitz How long were you running your app without seeing issues before seeing issues yesterday? Did anything else notable change regarding your app around the time you starting seeing the request body rate timeouts?
Also, what makes this reminiscent #2151 to you? One thing that helped us resolve #2151 was getting info from the reporter about how to reproduce the issue on our own hardware allowing us to investigate the issue more rapidly without losing time going back and forth getting more and more diagnostics from a third party.
One thing you can try that might help if the issue is that read callbacks are getting dispatched slowly due to ThreadPool starvation or similar is to change the gracePeriod component of the MinRequestBodyDataRate MinDataRate value. The default is 5 seconds, but it can be increased to whatever you want without changing the actual rate that's expected for large reads.

@halter73 I think I may have some more details about the root of my 502.3 api errors in my environment: I run a stress test on a specific asp.net core web api using the code suggested by @benaadams on a previous issue: https://github.com/aspnet/KestrelHttpServer/issues/2104#issuecomment-337539008
When the number of requests per minute hit 100 or more I noticed that the problems started to appear: api unresponsive, requests ending up in a 502.3 error, logs showing "Connection id ""0HLGU6OA7QP1G"", Request id ""0HLGU6OA7QP1G:00000001"": the connection was closed because the response was not read by the client at the specified minimum data rate", kestrel to be restarted in order to wake the server up again.
The api returns a json array and it seems that the errors starts when the total size of the returned json object (i.e. considering the number of requests by the size of the single response payload) exceeds a certain limit. This assumptions is due to the fact that, when strassing the very same api executing the very same code, with the only modification of returning an empty array instead of the calculated json object the test runs smoothly and no errors whatsoever show up. I am not very expert with the IIS and kestrel internal, so do you think that what I am saying makes sense? What is the parameter I could modify, apart from scaling up the servers?

It sounds like that throughput error is a symptom of your application having higher layer performance issues. If it can't handle 100 r/m then there is a significant bottleneck in the design. What to check next:

How large is the response when you start having that problem? It's possible you are saturating your network bandwidth, though this should require responses in the GB range.
Is the response very computationally expensive to generate? A profiler would be able to show you this. If your application is overtaxing the CPU then other parts of the system may start to fail. Making the generation more efficient or scaling up the application are your options here.
Another version of this is that you may be generating and serializing the response inefficiently with regards to blocking threads. This can cause a lot of problems for other parts of the application trying to use the thread pool. If you attach a debugger and break then you would see a lot of threads blocked on wait handles.

@halter73 - thanks for the quick response.
I did some more digging and found that the exception happened once on 8/21, 8/27, and 8/30.  Then a few times on the 14th and we've seen ~75 errors this week.  This could be due to increased traffic.  I'll need to write a fancy query to find out.

How long were you running your app without seeing issues before seeing issues yesterday?
This app has been running for months.

Did anything else notable change regarding your app around the time you starting seeing the request body rate timeouts?

We deployed a change that I initially suspected was related (based on the timing), but after rolling back that change I was still able to reproduce the error.  The only other piece that (I know) changed was the rollout of .net core v2.1.4 to Azure App Services, which was applied to the specific app service instance on 9/14 (the same day I first see the error in app insights).  I didn't update the app's reference from v2.1.2 to v2.1.4 until yesterday (9/18), so I wouldn't expect that to be related?

Also, what makes this reminiscent #2151 to you?
It's the same exception type (BadHttpRequestException), HTTP response (400) and hosted in Azure App Services (proxied through IIS first).  It happens sporadically.  The stack trace is different though and it doesn't mention MinRequestBodyDataRate, so maybe not related at all.

it can be increased to whatever you want without changing the actual rate that's expected for large reads.

The response body is not large, in fact for one of the operations, the response body is empty.  The other response (including headers) is about 1,300 bytes.  Neither operations is CPU intensive.  They both make several asynchronous calls to other endpoints (some of which are parallel).  Typically the entire call takes something like a few hundred ms to maybe 2k ms.  When it fails, the request takes ~6k-7k ms.  The related code hasn't changed recently. Given these details, do you still think it makes sense to adjust the MinRequestBodyDataRate::GracePeriod?

getting info from the reporter about how to reproduce the issue on our own hardware
Of course.  Short of creating a demo project, what would be the best way to do that?  Fiddler session?

@Tratcher thank you for the very precise analysis; I definetely believe it is the third scenario since the response body can hardly make up to GB in size and the response is generated quite efficiently in terms of CPU and resources; besides when I run this:
    [HttpGet]
    public JsonResult TesterApiMethod()
    {
        IEnumerable<myDTO> response = GetData();
        return Json("OK");
    }

everything works like a charm (I tried up to 350 requests/min),
while if I run this:
    [HttpGet]
    public JsonResult TesterApiMethod()
    {
        IEnumerable<myDTO> response = GetData();
        return Json(response);
    }

Kestrel stops responding and processing the requests even around 150 requests/minute. myDTO is a quite standard DTO object.

@Tratcher the IEnumerable is of type List, so I haven't gained any performance benefit; as far as the response size is concerned, it is roughly 90k, hence nothing too critical. I am then still struggling to find why the threads get blocked this way during serialization.

@Tratcher I enabled response buffering and it dramatically impacted the performance: so far I stress tested with 1000 req/min without any problem (I am confident it can handle much more now), while when disabling response buffering the server gets stuck even with only 150 req/min. Does it make sense with my limited response size? I am still very surprised for this behavior.

That's not a good sign. Can you attach the debugger when it gets stuck, break all, and look at the current thread call stacks? Are there a lot of threads blocked on wait handles? What are the common call stacks?

@Tratcher I managed to dump the hanging process and run a report (available as a .mht file at https://drive.google.com/open?id=1oFEvZl-RmDJbGnen_z4lnCSMCg7WTaJc), which confirms the amount of threads blocked on waits.

Trimming some noise from the thread stacks:
System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
mscorlib_ni!System.Threading.Monitor.Wait(System.Object, Int32, Boolean)+17
mscorlib_ni!System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)+226
mscorlib_ni!System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)+9c
mscorlib_ni!System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)+155
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(Byte[], Int32, Int32)+4d
Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.FlushInternal(Boolean)+51
Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.Write(Char)+20
Newtonsoft.Json.JsonTextWriter.WriteEnd(Newtonsoft.Json.JsonToken)+29
Newtonsoft.Json.JsonWriter.AutoCompleteClose(Newtonsoft.Json.JsonContainerType)+69
Newtonsoft.Json.JsonWriter.WriteEndObject()+b
Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(Newtonsoft.Json.JsonWriter, System.Object, Newtonsoft.Json.Serialization.JsonObjectContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty)+2db
...
Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(Newtonsoft.Json.JsonWriter, System.Object, Newtonsoft.Json.Serialization.JsonObjectContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty)+f9
...
Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(Newtonsoft.Json.JsonWriter, System.Object, System.Type)+a9
Newtonsoft.Json.JsonSerializer.SerializeInternal(Newtonsoft.Json.JsonWriter, System.Object, System.Type)+311
Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter.WriteObject(System.IO.TextWriter, System.Object)+47

Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter+<WriteResponseBodyAsync>d__11.MoveNext()+b8
Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter.WriteResponseBodyAsync(Microsoft.AspNetCore.Mvc.Formatters.OutputFormatterWriteContext, System.Text.Encoding)+73
Microsoft.AspNetCore.Mvc.Formatters.TextOutputFormatter.WriteAsync(Microsoft.AspNetCore.Mvc.Formatters.OutputFormatterWriteContext)+11e
Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor.ExecuteAsync(Microsoft.AspNetCore.Mvc.ActionContext, Microsoft.AspNetCore.Mvc.ObjectResult)+1b6
Microsoft.AspNetCore.Mvc.ObjectResult.ExecuteResultAsync(Microsoft.AspNetCore.Mvc.ActionContext)+2b
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeResultAsync>d__20.MoveNext()+7a
...

Looks like
Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter.WriteResponseBodyAsync
via
Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject
ending up calling
Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.Write(Char)
which then hits
System.Threading.Tasks.Task.InternalWait

This issue was moved to aspnet/Mvc#8486


fix: crash on web

📜 Description
Fixed a crash on web.
💡 Motivation and Context
It was a regression since 1.10.0 version. PlatformColor is not available on web and it will throw exception, because KeyboardToolbar is exported from index file and will be evaluated as soon as something gets imported from the package (i. e. KeyboardProvider).
To fix the problem I created two files: colors.ts (all platforms excluding mobile) and colors.native.ts (for mobile).
📢 Changelog

JS

created two files: colors.ts (all platforms excluding mobile) and colors.native.ts (for mobile)
moved types to common types file;
restructed imports to look into types file instead colors;

🤔 How Has This Been Tested?
Tested on web project.
📸 Screenshots (if appropriate):

📝 Checklist

[x] CI successfully passed
[x] I added new mocks and corresponding unit-tests if library API was changed

When will this be released 🤔 ?

@hyochan will try to do tomorrow or over the weekend 👀


[bug] Hosted Gorilla:

Exception: Invalid response object from API: '{"object":"error","message":"Only gorilla-7b-th-v0&&gorilla-cli-v0&&gorilla-mpt-7b-hf-v0&&gorilla-7b-hf-v0&&gorilla-azure-cli-v0&&gorilla-7b-hf-v1&&gorilla-aws-cli-v0&&gorilla-gcp-cli-v0 allowed now, your model gorilla-llm/gorilla-7b-hf-delta-v0","code":40301}' (HTTP response code was 400)
Failed model: gorilla-llm/gorilla-7b-hf-delta-v0, for prompt: I want to build a robot that can detecting objects in an image ‘cat.jpeg’. Input: [‘cat.jpeg’]

@ShishirPatil, could you take a look at this issue?


Create like_users_from_file

adding new example, to like users from file

Codecov Report

Merging #696 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #696   +/-   ##
=======================================
  Coverage   36.89%   36.89%
=======================================
  Files          26       26
  Lines        2266     2266
=======================================
  Hits          836      836
  Misses       1430     1430

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e07f642...2e59215. Read the comment docs.

This is not a *.py file, add an extension to a file.


./bin/owl release

Transform wd commands to nightwatch compatible commands. This allows user to use nightwatchjs as a framework with wd sdk so that magellan can drive wd test with the hand of nightwatchjs

Codecov Report

Merging #92 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master     #92   +/-   ##
======================================
  Coverage    98.8%   98.8%
======================================
  Files          21      21
  Lines         924     924
  Branches      153     153
======================================
  Hits          913     913
  Misses         11      11

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0eb9f00...fe1a9b7. Read the comment docs.


unhandled rejection if .bidsignore is a directory

In 1.7.3 and 1.8.1 validator will fail with EISDIR error trying to read .bidsignore as if it were a file. We should fail better here.
Found an old install of 1.3.4 on the HPC and interestingly it did not fail in this case.

See also #648

Need a test case.


feat: add type coverage

Motivation and Resolution
This PR adds a new ts:coverage npm script that uses typescript-coverage-report to generage a TypeScript coverage report available in the coverage-ts directory.
On initial run, every files had >80% coverage except 4:
src/utils/assert.ts
src/utils/calldata/formatter.ts
src/utils/calldata/parser/parser-2.0.0.ts
src/utils.responseParser/index.ts
I added some additional typings on these files to reach >80% coverage (not sure if these modifications are 100% correct though).
Resolves: #1088
RPC version (if applicable)
0.7.1
Usage related changes

None
Development related changes
New npm script added: ts:coverage
Checklist:

[X] Performed a self-review of the code
[X] Rebased to the last commit of the target branch (or merged it into my branch)
[X] Linked the issues which this PR resolves
[ ] Documented the changes in code (API docs will be generated automatically)
[ ] Updated the tests
[X] All tests are passing

@ivpavici I removed the coverage-ts folder and switched to type-coverage as requested.
The ts:coverage command will now display a warning eg. The type coverage rate is lower than the target(95%).
The 95% target is arbitrary idk if we should target higher or lower.
Also I couldn't find a way with this package to display precisely the specific files having a low type coverage contrary to typescript-coverage-report.

thanks!
Maybe we couldn also add a note for future devs about running this step manually here:
https://github.com/starknet-io/starknet.js/blob/develop/CONTRIBUTING.md#other-notes

@PhilippeR26 Thanks for the review, I added typescript-coverage-report back and ignored the default output coverage-ts in .gitignore.
@ivpavici I've added a small note about the 2 new npm scripts introduced in CONTRIBUTING.md

In general looking lgtm

@saimeunt can you please fix the last comments from Phil? and we can merge 👍

Will do tonight!

@PhilippeR26 I was unable to make the warning disappear, it's caused by a non-critical issue in semantic-ui-react which is a dependency of typescript-coverage-report, this dependency will have to be updated to get rid of the issue.
@ivpavici I'm closing this PR in favor of #1120 which is properly targetting next-version.


Project name 'tifig' confusable with another project

There is already another project named tifig (for Swift development based on Eclipse). https://www.tifig.net
Considering another name might be a good idea.

Thanks for the hint. We might consider that in the future.


@wraps breaks introspection

https://github.com/dask/dask-ml/blob/cb3d0ea392a277cade35405f6b156f22a1616551/dask_ml/datasets.py#L11
The use of @wraps here leads to wrong behavior of some introspection tools, like ?? in Jupyter (will show you the code for the sklearn version).
I feel that it's best to wrap thing manually (assigning the __name__, __doc__ and __module__ attributes for inner manually), after line 17, before the return.
This issue is making some unpleasant workarounds necessary in the xarray_filters codebase (same datasets.py file) (where we are trying to give more helpful docstrings users beyond what's in sklearn). The issue for me revolves around knowing whether I'm going to wrap the make_blobs from dask_ml or from sklearn. I could do it by inspecting the signature, or the __module__ attribute, but those attributes are set equal to the ones in sklearn right now.
I'll put links to relevants parts of the xarray_filters codebase here as soon as I push the new code up.

Awesome, thanks, @TomAugspurger!


Add custom natspec and ability to generate full documentation

Motivation

Solution

Codecov Report
Base: 85.56% // Head: 62.90% // Decreases project coverage by -22.65% :warning:

Coverage data is based on head (6115e27) compared to base (5c40089).
Patch has no changes to coverable lines.

Additional details and impacted files
@@             Coverage Diff             @@
##              1.2     #869       +/-   ##
===========================================
- Coverage   85.56%   62.90%   -22.66%
===========================================
  Files          52       50        -2
  Lines        3103     2750      -353
  Branches      414      382       -32
===========================================
- Hits         2655     1730      -925
- Misses        446      899      +453
- Partials        2      121      +119

Flag
Coverage Δ

foundry
62.90% <ø> (+0.11%)
:arrow_up:

production
?

reference
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

contracts/lib/Consideration.sol
94.11% <ø> (-0.33%)
:arrow_down:

contracts/zones/PausableZone.sol
0.00% <ø> (ø)

contracts/Seaport.sol
0.00% <0.00%> (-100.00%)
:arrow_down:

contracts/lib/TypehashDirectory.sol
0.00% <0.00%> (-100.00%)
:arrow_down:

reference/lib/ReferenceConsiderationBase.sol
0.00% <0.00%> (-100.00%)
:arrow_down:

reference/conduit/ReferenceConduitController.sol
0.00% <0.00%> (-100.00%)
:arrow_down:

contracts/conduit/ConduitController.sol
2.59% <0.00%> (-97.41%)
:arrow_down:

reference/lib/ReferenceVerifiers.sol
25.00% <0.00%> (-73.08%)
:arrow_down:

contracts/lib/ConsiderationBase.sol
23.52% <0.00%> (-72.63%)
:arrow_down:

reference/lib/ReferenceAssertions.sol
33.33% <0.00%> (-66.67%)
:arrow_down:

... and 36 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Allow to render some branches on left

Currently all branches are rendered on right. Having huge mind map, makes the right side of the picture very dense and the left side of the picture very sparse. If it was possible to specify (via some custom markdown note) some second level paragraphs to render on left, it would be more balanced picture and would add additional meaning to mind map (for example, if my mind map is about requirements for a product, I would render on left all functional requirements and on right all non-functional requirements). Does it make sense?

This makes total sense, it was also discussed before: https://github.com/dundalek/markmap/issues/3
I just haven't got time to get to it yet.

@dundalek +1 for this feature.
Just a suggestion ... rather than implementing using rules, could it use an existing layout library like GoJS or https://github.com/leungwensen/mindmap-layouts?
The Chrome extension mindmap-tab also does a pretty good job of rendering a mindmap and seems to have some lightweight logic.
I'll a novice at programming, but will take a look if I can add the logic somehow. thanks.

@abhayghatpande mindmap-layouts is a great find, I think that could work well. Another candidate I found recently is d3-flextree. Better layout algorithm is long due.
I would be really glad for your contributions, thank you. Let me know if I can help, clarify some things, or point you into the right direction.

Hello, I saw this issue is opened since 2017, 6 years after, were they anything made to adress this layout feature ? Thank you

Hi, is there any update on this request?


M3-1449 controlled input warning

To Test

Go to /users and click on a user to visit the details page
Check the console; controlled to uncontrolled input warning should be gone

Coverage increased (+0.002%) to 27.034% when pulling d5e4428924cfb1b020323e2fb97532581eb39fae on Jskobos:M3-1449-controlled-input-warning into 437c1076394bbda7729501ce4df44c259367b34e on linode:develop.


Allow easy strip of all chords

Allow easy strip of all chords, leaving lyrics alone,

Blocks #21 etc.

Done at c3110f925c8ba6fb702793527b5060c3afff067f


BUG: options page not working in mobile browsers. Other than that TNT works on mobile/touch browsers!

TNT options pop-up modal doesn't allow to change selection menu items with touch screen radio buttons can be selected and unselected also titles expand if so. But selection menus doesn't work - one cannot change the defaults in selection menus :(
Other than that Tweak New Twitter works flawlessly on mobile android browsers (Kiwi) touch screen interface.
_At the moment extensions can be used in few mobile browsers - such as Firefox, Kiwi and Yandex browsers. But extensions are coming - I reckon for Vivaldi and Opera next. Chrome android/ios - maybe not in the coming years. MS Edge probably too.
_

Thanks for the report, I've only tested the options page in Firefox Nightly for Android, will have a look at fixing it in Kiwi when I get a chance.

Other extension settings pages work but those use full page/new tab. I have Firefox too but Kiwi (and Yandex) browsers are the only beta-stable browsers atm for whole CWS extensions.
options_page method instead of options_ui
For example this ext works and it's selection menus:
https://chrome.google.com/webstore/detail/search-by-image/cnojnbdhbhnkbcieeekonklommdnndci
https://addons.mozilla.org/en-US/firefox/addon/search_by_image/
Also uBlock origin's


Revalidation should also work with Last-Modified header

Right now, because of this line, revalidation using HTTP Cache Headers only works with ETags.
Shouldn't Last-Modified/If-Last-Modified also be allowed in there ?

Duplicate of https://github.com/apollographql/datasource-rest/issues/34


Checkstyle fixes for async-rest sample.

Tested manually by deploying to http://async-rest-test-dot-java-docs-samples-swast-1.appspot.com/.
When doing so, I noticed that the mvn gcloud:deploy command would not run, so I updated the POM to the latest version of the gcloud maven plugin, too.
Just a cherry pick of https://github.com/GoogleCloudPlatform/java-docs-samples/pull/119, this time without the git submodule piece.

LGTM - though you could add a license header to the pom.xml and remove most of the xml header.  <project> is all you need.

Done. I've pushed a new commit. I'll wait until tests pass before merging.


[Xpt] Title

Description to be added

Hi team 👋,
While you made a great effort in this project, unfortunately, I cannot proceed to review your code.

Invalid Code Review Request 🛑
You have submitted a project, that doesn't have the following 👇

Link to your GitHub project
Information about the number of people in your group
List of GitHub usernames assigned to each team member

Your Code Review Request will be marked as invalid in your Dashboard, so please submit a new one once you are ready 🙏
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

Hi team @codepantha, @Felipe-Perez-Ferraro  🙋♀,
Great work so far! 💯
There are some issues that you still need to work on to improve your kanban board according to the requirements. You are almost there! 💪🏾
Required Changes ♻
The empty Kanban board is meant to help you start the process of distributing tasks among teammates. You have to manually add each task based on the project requirements. The project requirements  has already divided tasks into group tasks, team member #1 and team member #2. Use the points to create tasks, it is up to you how you divided the task into different cards as long as you do not put two different features into one tasks. You can add as many reasonable tasks as you want.

Ensure that the requirements listed as group tasks have a corresponding card in the Kanban board and have both team members assigned.
Decide who will be Team member #1 and Team member #2. Then make sure requirements from the list for a given team member in the project requirements divided by team member  file have a corresponding card in the Kanban board.
Kindly make sure that the number of points for each student ([Xpt] in the card title is allocated.
Each card should have the corresponding student assigned.
The cards should have meaningful titles and descriptions based on the points in the project requirements.

Cheers 🥂  and Happy coding!!! 👯
Feel free to leave any questions or comments in the issue thread if something is not 100% clear.
Please, remember to tag me @KDlamini  in your question so I can receive the notification.
Please, do not open a new issue for re-reviews. You should use the same issue submitted for the first review, either valid or invalid unless it is requested otherwise.

As described in the Code reviews limits policy you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.

Hello @KDlamini I didn't know it sorry. I made the changes requested

Hi @ZabihullahNooriWardak, @Felipe-Perez-Ferraro  👋,
Great work on making the changes requested by a previous reviewer 👏
You've done well implementing some of the requested changes, but there are still some which aren't addressed yet.
Highlights

Proper kanban board template is used ✔
All tasks are assigned team members

Required Changes ♻

Please make sure all the requirements are assigned for each team member as I noticed team member 2 doesn't have the below requirement assigned:

All data should be preserved by saving collections in .json files.

Kindly make sure to provide descriptions for all the cards

For the requirement "Create a schema.sql file with tables that will be analogical to the structure of the classes that you created" kindly make sure to provide a good description as to which table will be created because there are multiple tables and this could lead to confusion

Optional suggestions

N/A

Every comment with the [OPTIONAL] prefix won't stop the approval of this PR. However, I strongly recommend you take them into account as they can make your code better. Some of them were simply missed by the previous reviewer and addressing them will really improve your application.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.

Hi @Felipe-Perez-Ferraro, @ZabihullahNooriWardak
Approved 🟢 🟢
Your project is complete!  There is nothing else to say other than... it's time to merge it :shipit:
Congratulations! 🎉

Highlights

Kanban board setup correctly 💯
All cards are assigned to the correct student👍
Tasks distributed equally 🟢 👍

Optional suggestions
Every comment with the [OPTIONAL] prefix won't stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better. Some of them were simply missed by the previous reviewer and addressing them will really improve your application.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.

As described in the Code reviews limits policy you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


PP-5454 Add DAO class for transaction events

WHAT

Adds DAO class TransactionEventDao and method to query events by external_id & gateway_account_id
Uses both Transaction and Event tables as gateway_account_id & amount (should be available for each event) are available in Transaction and the rest of information in Event.
Filters events by parent_external_id to find refunds and related events (avoids self joining transaction table)
Relevant integration tests

Superseded by https://github.com/alphagov/pay-ledger/pull/177


[SUPPORT]HoodieException: Commit<PHONE_NUMBER>5215 failed and rolled-back ! at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:492)

Hi Team,
our hudi jobs are failing with below exception.
Can you please help me on this.
HoodieException: Commit<PHONE_NUMBER>5215 failed and rolled-back !
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:492)
Environment Description

Hudi version :0.8.0

Spark version :3.2.1

Storage (HDFS/S3/GCS..) : AZURE blod storage

we are running the apache hudi spark in azure databricks .
using below dependancies .
org.apache.spark:spark-avro_2.12:3.1.1
org.apache.hudi:hudi-utilities-bundle_2.12:0.8.0
org.apache.hudi:hudi-spark3-bundle_2.12:0.8.0

@pratyakshsharma
HI what is stacktrace ?, i am not aware of it ,can you please help with more details.

@nleena123 I meant to say please share the detailed error log and not just the exception message. :)

Hi @pratyakshsharma
Please find the attached complete error log details.
metrics_job_logs.txt

can you furnish the write configs you use to write to hudi.

#hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.CustomKeyGenerator
hoodie.datasource.write.hive_styling_partioning=true
hoodie.datasource.write.partitionpath.urlencode=false
#hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.ComplexKeyGenerator
hoodie.datasource.write.recordkey.field=interaction_id
hoodie.datasource.write.partitionpath.field=create_datetime:TIMESTAMP
hoodie.deltastreamer.keygen.timebased.timestamp.type=DATE_STRING
hoodie.deltastreamer.keygen.timebased.input.dateformat=yyyyMMdd HH:mm
hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM
hoodie.deltastreamer.keygen.timebased.timezone=UTC-6:00
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.CustomKeyGenerator
#20211119 13:39
#hoodie.index.type=GLOBAL_BLOOM
#hoodie.bloom.index.update.partition.path=true
schema provider configs
#hoodie.deltastreamer.schemaprovider.registry.url=https://azure-prod-schema-registry.extscloud.com/subjects/async.messaging.interaction.metrics-value/versions/3
hoodie.deltastreamer.schemaprovider.registry.url=https://azure-prod-schema-registry.extscloud.com/subjects/async.messaging.interaction.metrics-value/versions/latest
Kafka Source
hoodie.deltastreamer.source.kafka.topic=async.messaging.interaction.metrics
#Kafka props
#hoodie.auto.commit=true
enable.auto.commit=true
#hoodie.deltastreamer.kafka.source.maxEvents=1000000
group.id=async.messaging.metrics
bootstrap.servers:443=prod-hcc-kaas-central-13.centralus.cloudapp.azure.com
#metadata.broker.list:443=kaas-prod-elr-a.optum.com
auto.offset.reset=latest
auto.reset.offsets=latest
#schema.registry.url=https://azure-test-schema-registry.extscloud.com
schema.registry.url=https://azure-prod-schema-registry.extscloud.com:443
schema.registry.ssl.keystore.location=/dbfs/mnt/ulp/artifacts/topics/certs/keystore.jks
schema.registry.ssl.keystore.password=prmcert
schema.registry.ssl.truststore.location=/dbfs/mnt/ulp/artifacts/topics/certs/truststore.jks
schema.registry.ssl.truststore.password=prmcert
schema.registry.ssl.key.password=prmcert
security.protocol=SSL
sasl.mechanism=PLAIN
ssl.keystore.location=/dbfs/mnt/ulp/artifacts/topics/certs/keystore.jks
ssl.keystore.password=*******
ssl.truststore.location=/dbfs/mnt/ulp/artifacts/topics/certs/truststore.jks
ssl.truststore.password=**********
ssl.key.password=prmcert

Sorry for being late here. I see the class DFAvroKafkaSource in logs. Are you having your own custom class for reading the data?

yes  , below DFAvroKafkaSource.java file we are using to read the data  which extends AvroSource.
DF3HoodieDeltaStreamer.java which extends HoodieDeltaStreamer ,  and attached pom.xml file
code.zip
in the folder please have a look on it and let me know ,if any changes  required in my code .

and can you share your deltastreamer configs please?

are you trying out for the first time and running into issues. or was this running ok for sometime and all of a sudden you hit this when you upgraded or changed any configs?

Hi @nsivabalan
Below attached property file contain all configs that we used to This job.
And  used passed below arguments to databrick job (we are running hudi job through Azure databricks)
metrics.properties.txt
["--table-type","COPY_ON_WRITE","--source-ordering-field","CDC_TS","--source-class","com.optum.df.hudi.sources.DFAvroKafkaSource","--target-base-path","/mnt/ulp/dataassets-lake/metrics/","--target-table","metrics","--schemaprovider-class","org.apache.hudi.utilities.schema.SchemaRegistryProvider","--props","/mnt/ulp/artifacts/properties/metrics.properties"]

I see hoodie.datasource.write.hive_styling_partioning should be changed to hoodie.datasource.write.hive_style_partitioning.
Looking at your timeline,<PHONE_NUMBER>3554__commit__COMPLETED is the last time the job ran successfully, after that you have been running into issues. So we want to understand if you changed anything since this last successful commit?

Yes, changing to hoodie.datasource.write.hive_style_partitioning will not fix your issue. That was just a typo in your configs (unrelated to this error), so I just highlighted this.

Also I guess there is some confusion with respect to the code that you shared and the logs that you shared. In the code, I see multiple logs getting printed which are not there in the logs file you shared.
Also the schema getting registered in the logs is not having CDC_TS field. Can you share latest logs? That might help in debugging the issue better.

We are adding new field CDC_TS to Kafka data through  code which present in DF3HoodieDeltaStreamer.java program.
Code  :-
static GenericRecord getDataWithCDC_TS(GenericRecord record, String ts) {
	Schema base = record.getSchema();
	List<Schema.Field> baseFields = base.getFields().stream()
			.map(field -> {
				Schema.Field f = new Schema.Field(field.name(), field.schema(), field.doc(), field.defaultVal());
				field.aliases().forEach(f::addAlias);
				return f;
			})
			.collect(Collectors.toList());
	List<Schema.Field> newFields = new ArrayList<>(baseFields);
	ArrayList<Schema> optionalString = new ArrayList<>();
	optionalString.add(Schema.create(Schema.Type.NULL));
	optionalString.add(Schema.create(Schema.Type.STRING));
	//JsonNode nullDefault = JsonNodeFactory.instance.nullNode();

	//Schema.Field f = new Schema.Field(CDC_TS, Schema.createUnion(optionalString), CDC_TS, nullDefault);
	Schema.Field f = new Schema.Field(CDC_TS, Schema.createUnion(optionalString), CDC_TS, null);
	if (!baseFields.contains(f))
		newFields.add(f);

	Schema newSchema = Schema.createRecord(
			base.getName(),
			"New schema by adding a new field,CDC_TS",
			base.getNamespace(), base.isError(),
			newFields);

	LOG.info("newSchema is:" + newSchema);

	GenericRecord newRecord = new GenericData.Record(newSchema);

	base.getFields().stream().forEach(field -> newRecord.put(field.name(), record.get(field.name())));
	newRecord.put(CDC_TS, ts);

	return newRecord;

}

i have attached  Latest log file , please have look on it.
log4j-active_latest.txt

I have the same problem, and I haven't solved it yet

Hi @xushiyan , @pratyakshsharma
From very long back , i am seeing this issue ,
Can you please update on this , or please suggest me the resolution steps ?

Hi Team,
Can you please update me on the above issue ??

HI @qianchutao ,
Are you able to fix the issue ?
and can you please let me know which version of hudi are you using ?

I am also facing the similar exception, using EMR on AWS with kafka. Even exception is not self explanatory if any config is missing or what is exactly wrong
Exception:
22/08/09 07:30:39 INFO TaskSetManager: Finished task 0.0 in stage 66.0 (TID 2033) in 113 ms on ip-10-151-46-136.us-west-2.compute.internal (executor 1) (1/2)
22/08/09 07:30:39 INFO TaskSetManager: Finished task 1.0 in stage 66.0 (TID 2034) in 140 ms on ip-10-151-46-136.us-west-2.compute.internal (executor 1) (2/2)
22/08/09 07:30:39 INFO YarnScheduler: Removed TaskSet 66.0, whose tasks have all completed, from pool
22/08/09 07:30:39 INFO DAGScheduler: ResultStage 66 (collectAsMap at HoodieSparkEngineContext.java:151) finished in 0.151 s
22/08/09 07:30:39 INFO DAGScheduler: Job 32 is finished. Cancelling potential speculative or zombie tasks for this job
22/08/09 07:30:39 INFO YarnScheduler: Killing all running tasks in stage 66: Stage finished
22/08/09 07:30:39 INFO DAGScheduler: Job 32 finished: collectAsMap at HoodieSparkEngineContext.java:151, took 0.153794 s
22/08/09 07:30:39 ERROR HoodieDeltaStreamer: Shutting down delta-sync due to exception
org.apache.hudi.exception.HoodieException: Commit<PHONE_NUMBER>3005189 failed and rolled-back !
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:649)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:331)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:675)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
22/08/09 07:30:39 ERROR HoodieAsyncService: Service shutdown with error
java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: Commit<PHONE_NUMBER>3005189 failed and rolled-back !
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:103)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:189)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:186)
at org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.sync(HoodieMultiTableDeltaStreamer.java:416)
at org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.main(HoodieMultiTableDeltaStreamer.java:247)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1000)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1089)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1098)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hudi.exception.HoodieException: Commit<PHONE_NUMBER>3005189 failed and rolled-back !
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:709)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hudi.exception.HoodieException: Commit<PHONE_NUMBER>3005189 failed and rolled-back !
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:649)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:331)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:675)

HI @qianchutao , Are you able to fix the issue ? and can you please let me know which version of hudi are you using ?

Were you able to fix this issue?

@nleena123 @ROOBALJINDAL : looks like the stacktrace does not have sufficient info to triage. So, can you folks enable debug logs and share what you see. We should probably get some insights into whats happening. Also, if we can somehow get a reproducible code, would really help us triage it faster. Since timeline events are very tough to grok manually looking at the logs.

@nsivabalan I tried enabling debug logs by changing root log4j.rootLogger=INFO to log4j.rootLogger=DEBUG in Hudi jar for following files
log4j.properties
log4j-surefire.properties
log4j-surefire-quiet.properties
Also updated some other properties in these files:
log4j.rootLogger=DEBUG, CONSOLE
log4j.logger.org.apache=INFO
log4j.logger.org.apache.hudi=DEBUG

I am using this custom hudi jar by following spark command:
spark-submit  \
  --class org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer s3://slava-redshift-test/hudi/roobal-hudi-Debug-logger/hudi-utilities-bundle_2.12-0.11.0_edfx_debug.jar \

I have also tried this:
spark-submit  \
  --conf spark.executor.extraJavaOptions=-Dlog4j.debug=true \

Can you help what I am missing?
Note: I am using EMR cluster on AWS.

log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark-project.jetty=WARN
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

# Example
log4j.logger.org.apache.hudi=DEBUG

sample log4j file.
--conf spark.driver.extraJavaOptions="-Dlog4j.configuration=file:/home/hadoop/log4j.properties" --conf spark.executor.extraJavaOptions="-Dlog4j.configuration=file:/home/hadoop/log4j.properties"

@ROOBALJINDAL : can we have some updates.

@nsivabalan yeah, issue worked for me. I logged separate issue and I was able to get help and resolve. Thank you team 👍

oh ok ok, got it. thanks. may I know which issue you are referring to.

@nleena123 : are you still facing issues or are we good to close it out ?

@nsivabalan this was my issue which I have already closed.
https://github.com/apache/hudi/issues/6348

still i could see the same issue do i need to follow the below step to fix the issue  ?
--conf spark.driver.extraJavaOptions="-Dlog4j.configuration=file:/home/hadoop/log4j.properties" --conf spark.executor.extraJavaOptions="-Dlog4j.configuration=file:/home/hadoop/log4j.properties"
but i am getting this issue while running data bricks job  ?

above command is just to enable debug logs. that's not the fix as such. we might need proper driver logs, and contents of ".hoodie" to investigate the issue.

@nleena123 : are you still looking for assistance.

@nleena123 : gentle ping.

@nleena123 can we close this issue if you are fine?


Build auto-deriving impl for CliMessage

Why
Implements automatic derivation of the CliMessage trait for an enum
This lets us separate view logic easily by having any command (or helper) functions take the signature:
fn my_command(...) -> impl CliMessage {

and then create message enums for them.
Supports magic derive for message codes, so an enum variant RelayCreate::Successwill derive a message code of relay-create-success.
example:
#[derive(CliMessage)]
pub enum RelayCreate {
   #[message("Relay created successfully")]
   Success,
   #[message("A relay with the domain {} already exists")]
   AlreadyExists(String),
   #[message("Docker service isn't running", exit_code = 70)]
   DockerNotRunning,
   #[message("Domain name is too long", code = "too-long")]
   TooLong
}

How
write a macro.. fairly clear and well tested + shouldn't need to change very often

this is cool but too much maintenance!


Add create-react-app SSR template

It looks like the templates have moved to using create-react-app. This is great and makes things more standard.
We still need server side rendering.
I've been looking into this a little and there seems to be some libraries that claim to add SSR to a CRA project without ejecting. Here are a couple.
https://github.com/antonybudianto/cra-universal
https://github.com/smooth-code/loadable-components
It would be nice if there was an option when doing dotnet new to create a react/redux project that has SSR built in for ASPNET Core. dotnet new reactreduxssr or something like that.
Not having SSR really makes CRA apps not usable for public websites.

Please consider reopening this feature request. There may be no community involvement, but it is still a valid feature request.

I'm happy to talk about what features we would like. I know @SteveSandersonMS is a busy guy so I didn't want to bug him about this.

You mentioned there used to be a template with SSR which was removed in favor of a CRA based on. Maybe there is value in finding how compatible the previous SSR solution would be with CRA? For myself, I would like the ultimate solution to .NET Core SSR to be standalone so that it can be used without starting off from a CRA based template, but even when that would not be an option, there is still value in making it work with a template that encompasses both the FE and BE.

If you create the project using core 1.1, you'll get the old SSR template. I think moving to CRA is a good idea since that is the industry standard now. I think SSR just needs to be added into it.

Sorry, that is what I meant to express in my previous comment.

@JoshClose Are you looking for a project like ReactJS.NET? They have outlined a guide on how to configure SSR here. But I do understand and support the importance of having an SSR template available for .NET Core. I was just reading up on GatsbyJS and really liked how simple it was to create static sites but that's not true SSR. I think trying to integrate SSR as a template could be very messy if we want to use CRA. I think if we want to provide a good developer UX then it might be necessary to abstract things away into its own framework but then that's just repeating the work of ReactJS.NET.

Look at the React Core Boilerplate. It includes SSR, React, Redux and many other things. It's not a CRA, but it is used in many projects in production.


Detect Ruby code generated by protobuf compiler

Linguist doesn't detect Ruby code generated by the protobuf compiler. This PR changes that.
Checklist:

[ ] I am associating a language with a new file extension.

[ ] The new extension is used in hundreds of repositories on GitHub.com

Search results for each extension:

https://github.com/search?utf8=✓&type=Code&ref=searchresults&q=extension%3AFOOBAR+KEYWORDS+NOT+nothack

[ ] I have included a real-world usage sample for all extensions added in this PR:

Sample source(s):

[URL to each sample source, if applicable]

Sample license(s):

[ ] I have included a change to the heuristics to distinguish my language from others using the same extension.

[ ] I am adding a new language.

[ ] The extension of the new language is used in hundreds of repositories on GitHub.com.

Search results for each extension:

https://github.com/search?utf8=✓&type=Code&ref=searchresults&q=extension%3AFOOBAR+KEYWORDS+NOT+nothack

[ ] I have included a real-world usage sample for all extensions added in this PR:

Sample source(s):

[URL to each sample source, if applicable]

Sample license(s):

[ ] I have included a syntax highlighting grammar.
[ ] I have included a change to the heuristics to distinguish my language from others using the same extension.

[ ] I am fixing a misclassified language

[ ] I have included a new sample for the misclassified language:

Sample source(s):

[URL to each sample source, if applicable]

Sample license(s):

[ ] I have included a change to the heuristics to distinguish my language from others using the same extension.

[ ] I am changing the source of a syntax highlighting grammar

Old: https://github-lightshow.herokuapp.com/
New: https://github-lightshow.herokuapp.com/

[ ] I am adding new or changing current functionality

[ ] I have added or updated the tests for the new or changed functionality.

@pchaigno Thanks for the review. Should I just merge this?

We only require one review and a :+1: from GitHub's staff, so I guess you can :-)


Update setup-elixir action to fix CI

Pull Request Test Coverage Report for Build 555477a8973b380fa04eadabb4a6898e24b44980-PR-741

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 97.095%

Totals

Change from base Build 9363a7a23ad43c83b54bb6f5266f1b11eff6bf61:
0.0%

Covered Lines:
1504

Relevant Lines:
1549

💛  - Coveralls


Fix resolving relative path of Get-ScpFolder

This is related to issue #172. I applied the same code that is implemented for Get-SCPFile which I haven't had a problem with.

@soul4soul Could this pull request be closed given the fix was implemented?

From what I'm seeing its not fixed.

can you resolve the conflict? I will merge it in this weekend


SSAO doesn't work on integrated Intel GPU

Describe the bug
Run gltf_viewer with FlightHelmet on a MacBook Pro with integrated Intel GPU (make sure you are not using the discrete GPU if testing on a 15"). SSAO creates a lot of artifacts on screen. It looks like self shadowing.
Screenshots

Desktop (please complete the following information):

OS: macOS Mojave 10.14.5
GPU: Intel iris Plus  Graphics 655
Backend: OpenGL

I also have this issue (or similar), however I'm experiencing it on two NVIDIA GPU's. It only happends when using the OpenGL backend.

Thanks @nitronoid. Are those discrete GPUs?

@romainguy Sorry I'm not familiar with the term, but yes I think so. Quadro M3000M and Quadro K2200.

@romainguy Just saw the tags on this issue and wanted to clarify, I'm not using macos. I get these artifacts on RHEL-7 and Fedora 29.

Thanks, good to know it's not OS specific.


Removes need for QueryField custom Apex class per financialforcedev/f…

…flib-apex-common#105
#105

@daveespo  I think we can close this PR now, yes?

@daveespo I think we can also close this one as well now, yes?


add delete interface

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?


Adds a new abstract class table_type.py

Contains records creation and insertion methods for different types of tables.
Different table require different record types. The methods are common - string generation, long generation but we call them in different combinations which are defined in these methods. (separate for each variety of table)
/gcbrun

Closed as the branch is merged with PR #73


Demo 1 wrong link for Contoso Sales Assistant Example

The link to the Contoso Sales Assistant demo 1 documentation (the option developed with Chainlit) added into demo readme file is broken.  Correct link is https://github.com/microsoft/aitour-azure-openai-assistants

any update on this issue @gloveboxes @softchris ?
cc @mbaiza27

@blackgrey0010 addressed in my last PR https://github.com/microsoft/aitour-azure-openai-assistants/pull/11

PR merged https://github.com/microsoft/aitour-azure-openai-assistants/pull/11, closing issue


A trivial fix for supporting volume size in KB

Currently our driver is silently ignoring the value passed in KB and coverts into MB due a bug in covert.py: this PR is proposed to handle -o size parameter correctly while passing the input in KB.
Testing done: Yes (see below)
tested the proposed code with following volume creation commands:
root@photon-EHiCjpUvt [ ~ ]# docker volume create --driver=vmdk --name=MyVol8 -o size=4000kb
MyVol8
root@photon-EHiCjpUvt [ ~ ]# docker volume create --driver=vmdk --name=MyVol9 -o size=2000kb
MyVol9
root@photon-EHiCjpUvt [ ~ ]# docker volume create --driver=vmdk --name=MyVol10 -o size=1100kb
MyVol10

[root@promc-2n-dhcp105-97:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume   Datastore       Created By VM  Created                   Attached To VM  Policy  Capacity   Used      Disk Format  Filesystem Type  Access      Attach As
-------  --------------  -------------  ------------------------  --------------  ------  ---------  --------  -----------  ---------------  ----------  ----------------------
MyVol8   TestDatastore1  photon2        Fri Dec 16 23:03:49 2016  detached        N/A     3.00MB     3.00MB    thin         ext4             read-write  independent_persistent
MyVol9   TestDatastore1  photon2        Fri Dec 16 23:05:01 2016  detached        N/A     1.00MB     2.00MB    thin         ext4             read-write  independent_persistent
MyVol10  TestDatastore1  photon2        Fri Dec 16 23:05:26 2016  detached        N/A     1.00MB     2.00MB    thin         ext4             read-write  independent_persistent

while working on this PR came to know about the limitation from the ESX server that we can't create vmdk < 1024 KB going to file a separate issue to initiate the discussion for the same.
/CC: @ashahi1

@msterin: we can do that, by looking at the code (https://github.com/vmware/docker-volume-vsphere/blob/0e08691def491e7f36a0a03b8c013fe7cf3a5c37/esx_service/vmdk_ops.py#L386) thought to fix the issue at moment.
It needs some extra validation required if we down the route to handle KB is a valid unit, see below:
root@photon-EHiCjpUvt [ ~ ]# docker volume create --driver=vmdk --name=MyVol11 -o size=1011kb
Error response from daemon: create MyVol11: VolumeDriver.Create: Server returned an error: (vmodl.fault.InvalidArgument) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'A specified parameter was not correct: ',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [
      (vmodl.LocalizableMessage) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         key = 'vim.hostd.vdisksvc.diskSizeTooSmall',
         arg = (vmodl.KeyAnyValue) [],
         message = 'The virtual disk size is too small.'
      }
   ],
   invalidProperty = <unset>
}

If we are OK to limit the unit with MB, GB, TB (http://vmware.github.io/docker-volume-vsphere/documentation/user-guide/docker-volume-cli/#size)  then I need to revert my changes and need to cleanup the tests if there are any.
Please share your opinion.

LGTM after you fix the CI errors, I can see why supporting kb is not interesting but given that it is there we can fix it instead of removing it. A customer might want to specify it in kb :-)

@kerneltime

A customer might want to specify it in kb :-)

The point is - why do we need to allow this ? do you expect people to actually need 512KB disks :-) ?

@kerneltime @msterin

A customer might want to specify it in kb :-)

The point is - why do we need to allow this ? do you expect people to actually need 512KB disks :-) ?

while working on this PR came to know about the limitation from the ESX server that we can't create vmdk < 1024 KB; we need to accommodate such restriction by validating user input.

@kerneltime

after you fix the CI errors

Tested with ESX 6.0U2 & ESX 6.5:
ESX 6.5:
[root@sc-rdops-vm06-dhcp-183-137:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py status
Version: <IP_ADDRESS>36235-0.0.1
Status: Running
Pid:<PHONE_NUMBER>
Port: 1019
LogConfigFile: /etc/vmware/vmdkops/log_config.json
LogFile: /var/log/vmware/vmdk_ops.log
LogLevel: INFO
[root@sc-rdops-vm06-dhcp-183-137:~] python --version
Python 3.5.1+
[root@sc-rdops-vm06-dhcp-183-137:~]

ESX 6.0U2:
[root@promc-2n-dhcp105-97:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py status
Version: <IP_ADDRESS>36235-0.0.1
Status: Running
Pid: 3169410
Port: 1019
LogConfigFile: /etc/vmware/vmdkops/log_config.json
LogFile: /var/log/vmware/vmdk_ops.log
LogLevel: INFO
[root@promc-2n-dhcp105-97:~] python --version
Python 2.7.9
[root@promc-2n-dhcp105-97:~]

Great, so let's drop KB feom allowed suffixes, and the problem will be gone

@msterin
Comment has been addressed to drop support of 'KB', please have a look at the latest changes.
Thanks!

@shuklanirdesh82 - thanks for taking care of this.
@kerneltime  - same :-)

Would have been great to have this check in the plugin code instead, since
size is a plugin option.
On Tue, Dec 20, 2016 at 2:39 AM, Mark Sterin<EMAIL_ADDRESS>wrote:

@shuklanirdesh82 https://github.com/shuklanirdesh82 - thanks for taking
care of this.
@kerneltime https://github.com/kerneltime - same :-)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/vmware/docker-volume-vsphere/pull/829#issuecomment-268079090,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APHseJ1G9lNGOiv4RWKkMFUTnaBQbydlks5rJvJzgaJpZM4LPrzm
.

Would have been great to have this check in the plugin code instead, since
size is a plugin option.

@govint : That's good point. By looking at the code (https://github.com/vmware/docker-volume-vsphere/blob/master/vmdk_plugin/vmdkops/vmdkops.go#L50) it seems plugin is just a pass through for all kind of options and driver is the one where all validation code relies.
Can someone ( @msterin / @kerneltime ) shed some idea on keeping validation code at driver level instead of plugin?
Thanks!

We cannot skip validation in esx service in my opinion. Adding validation on plugin side is an optimization to quickly reject what is not supported but the support really originates from the esx service. Going forward we will have to support mixing of versions for ESX and the Docker plugin.. in that case I can see why it would be clean to keep all validation on the ESX side and let Docker only deal with the docker API not the extensions via -o flag

I agree with @kerneltime  here - ESX side should do validation. Plugin side may do it to shortcut the round trip, but I do not think does not really save much ; so  I think having all validation on ESX side is perfectly reasonable


8337: add test cases for guest pull images

add test cases for guest pull images
need revist after we use container2.0 with 'image pull per runtime class' feature

for #8337 and #8407

This pr only include the test cases for guest pull with nydus-snapshotter
I tested those cases with my kcli cluster with nydus-snapshotter be installed and config by hand.
A good result looks like:
root@liudali-x86-libvirt:~/kata-containers/tests/integration/kubernetes# bats k8s-guest-pull-image.bats
 ✓ Test can pull an unencrypted image outside the guest with runc
 ✓ Test can pull an unencrypted image inside the guest
 ✓ Test can pull an unencrypted image inside the guest again
 ✓ Test can pull an unencrypted image outside the guest
 ✓ Test pull an other unencrypted image outside the guest
 ✓ Test pull an other unencrypted image outside the guest again
 ✓ Test pull an other unencrypted image inside the guest

7 tests, 0 failures

Hey DaLi, I've made a few comments for possible ways to make the code a bit easier to read, which @wainersm might also have input on.
The other thing that we might want to consider is the reverse case of issue 8337 (unless you have it and I've missed it), where we pull without a cri-runtime handler annotation, then pull with it and verify that it's pulled on the guest properly. You seem to have the reverse, pulled with annotation, on guest and then without and (TODO) check that it isn't pulled on the guest, but maybe both ways would be useful unless you think the runc case covers that?

Both case are reversed, there are two TODO items, Test we can pull an other unencrypted image (alpine) outside the guest and then inside the guest successfully is for the case we pull without a cri-runtime handler annotation, then pull with it and verify that it's pulled on the guest properly

I also think that for test like this one that we know will fail it's fine to write them as they should work, but then add a skip, and add a note in kata-containers#8337 to unskip them in future.
Thanks for the great work!

@stevenhorsman @ChengyuZhu6 Many thanks for your review comments, I updated the codes, please help to take a look when you are free.

Both case are reversed, there are two TODO items

Sorry, I'm not sure how I missed this last night.
Just to ensure I completely understand the Test we can pull an other unencrypted image (alpine) outside the guest and then inside the guest successfully scenario - it is first doing a host pull (but still kata-containers, not runc) using the vanilla kata logic, as it doesn't set the cri-handler annotation, then it does a second pull with the annotation and (in future) checks that it is still pulled on the guest?
If so that works great for me, sorry I got distracted with the first two tests.
I guess something to think about here, and maybe it needs wider discussion is whether having TODOs and changing the expected behaviour is the correct thing to do. There is an argument that this is a pretty serious limitation, so having the tests correct and initially failing might drive useful conversations in the community and potentially get any container 2.0 fixes backported to 1.7 maybe, so we can start using this as soon as possible? I'm a bit worried that what you've done, although it passes the test, hides the problem a bit?

Sorry, I'm not sure how I missed this last night. Just to ensure I completely understand the Test we can pull an other unencrypted image (alpine) outside the guest and then inside the guest successfully scenario - it is first doing a host pull (but still kata-containers, not runc) using the vanilla kata logic, as it doesn't set the cri-handler annotation, then it does a second pull with the annotation and (in future) checks that it is still pulled on the guest? If so that works great for me, sorry I got distracted with the first two tests.

Yeah, that case try to pull image on host(without annotation) first and then try to pull image on guest(with annotation)

I guess something to think about here, and maybe it needs wider discussion is whether having TODOs and changing the expected behaviour is the correct thing to do. There is an argument that this is a pretty serious limitation, so having the tests correct and initially failing might drive useful conversations in the community and potentially get any container 2.0 fixes backported to 1.7 maybe, so we can start using this as soon as possible? I'm a bit worried that what you've done, although it passes the test, hides the problem a bit?

Yeah, agree with you @stevenhorsman that current way hides the problem.
So we should update the test case to expect all correct count of rootfs from now, and then:

option1: failed the test case
option2: skip the test case, enable it after we have containerd 2.0
Which options we should use here @stevenhorsman @ChengyuZhu6 ?

So we should update the test case to expect all correct count of rootfs from now, and then:

option1: failed the test case
option2: skip the test case, enable it after we have containerd 2.0
Which options we should use here @stevenhorsman @ChengyuZhu6 ?

So this might be kicking the can down the road a bit, but for this PR I'd recommend option 1 where we let the test case fail and then as part of the main PR 8484 (IIRC) we can test it out and have the wider discussion with the community to highlight that it is still an issue and try and get broader agreement on a path forward.

So this might be kicking the can down the road a bit, but for this PR I'd recommend option 1 where we let the test case fail and then as part of the main PR 8484 (IIRC) we can test it out and have the wider discussion with the community to highlight that it is still an issue and try and get broader agreement on a path forward.

Updated the test cases to use option 1 with comments provided.

Thanks @liudalibj  LGTM!


[Twig] throw error if twig-bundle is not installed

Q
A

Bug fix?
no

New feature?
no

Tickets
n/a

License
MIT

Ran into an issue with ContainerBuilder::willBeAvailable() and ux-live-component's test suite when requiring symfony/twig-bundle directly. Not perfect but throwing an exception on container build if twig-bundle is not available seemed like the easiest solution.

Thanks Kevin!


How to run a whole js file multiple times using duk_pcall? (question)

I try to continuously call a js file, using duk_pcall to do it, here is what I tried:
Firstly, I load the script file into a char* string.
Secondly, use duk_pcompile_string(ctx, 0, programBody) to compile it.
Then, I can use duk_pcall(ctx, 0) to call it once, and call duk_pop(ctx).
But when I try to use duk_pcall a second time, I failed.
Can you give me some advice about how to do it the right way?

The stack behavior in your example is as follows. After compilation:
[ func ]

Once you duk_pcall(ctx, 0), the stack will become
[ result ]

Note that duk_pcall() pops the function and replaces it with the return value (see stack description in https://duktape.org/api.html#duk_pcall).
If you want to keep the compiled function and call it again without recompiling, you'd need to duk_dup() first to get:
[ func func ]

Then duk_pcall(ctx, 0) to get:
[ func result ]

And then duk_pop() to remove the result:
[ func ]

Now you'd be ready to duk_dup() again.


QR reading library?

This comment on the introductory Reddit post indicates there's a lack of QR reading libraries. In my own searching, I've found plenty of libraries to WRITE QR codes, but READING this seems to be a legitimate hole in the ecosystem.
However, I'm not sure what this exactly would entail, or even the original use case of this user. It seems that there's a QR decoder in this crate; it's the only crate that speaks of decoding that I can see, and it has a C dependency, so there may not be a pure-Rust QR decoding implementation either. I understand that a computer vision library would be required to read a QR code from an image in order to extract QR code to decode in the first place. Perhaps that's all there is?
I'll wait for somebody who knows more than I do to confirm this.

I think it's fairly safe to say that there seems to be a lack of interest in further defining use cases for this domain, so I'll use the most recent commit to close this issue.


Get strange error on webkit

Describe the Bug
I'm not sure this is webkit issue or something
This example code success most time but often failed with strange error
Steps to Reproduce

Clone https://github.com/Riey/yaml-error-test
Build with wasm-pack
Serve page with some server
Try it on webkit browser I've been tested in epiphany and webview(https://github.com/zserge/webview)
And show console for check error you may retry several times to see it

Expected Behavior
Get 36

Actual Behavior

Thanks for the report! I'd be curious to test this out locally, but it looks like your example repo is missing the JSON file?

@alexcrichton Thanks you I just add it sorry

Thanks!
Can you confirm what wasm-bindgen/rustc versions you're using? I checked out your repo and built it and it works in Firefox/Chrome for me.

Yes this code work fine in firefox or chrome but not on epiphany

Well epiphany itself it not matter for me but I trying to browse rust wasm in webview(https://github.com/Boscop/web-view)
but it also fail since it use webkit for linux

Oh sorry I see, but in that case this seems like it's likely a bug in the browser and/or wasm engine?

Then I'd like to report this in webkit bug report can I left this issue for others who like me?

Sure!


fix(components): adjusted import location of scss theme file

Description
Fixing the import path of the theme file for the components storybook preview.
Note: see the broken css here https://teradata.github.io/covalent/docs/components/
Test Steps

[ ] npm run storybook
[ ] then wait for the browser window to pop open
[ ] then observe that css now looks correct

General Tests for Every PR

[ ] npm run start still works.
[ ] npm run lint passes.
[ ] npm run stylelint passes.
[ ] npm test passes and code coverage is not lower.
[ ] npm run build still works.

:tada: This PR is included in version 5.0.2 :tada:
The release is available on:

v5.0.2
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 5.0.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Bug: @eslint/compat fixupPluginRules() does not work with rule options

Which packages are affected?

[x] @eslint/compat
[ ] @eslint/config-array
[ ] @eslint/core
[ ] @eslint/migrate-config
[ ] @eslint/object-schema
[ ] @eslint/plugin-kit

Environment
Node version: 23.1.0
npm version: 10.9.0
ESLint version: 9.13.0
Operating System: Ubuntu 24.04
What did you do?
// eslint.config.js
import {fixupPluginRules} from '@eslint/compat';
import fp from 'eslint-plugin-fp';

export default [
  {
    plugins: {
      fp: fixupPluginRules(fp),
    },
    rules: {
      'fp/no-mutating-methods': ['error', {allowedObjects: ['state']}]
    },
  },
];

// index.js
const state = [];
state.push(3);

$ eslint index.js

Oops! Something went wrong! :(

ESLint: 9.13.0

Error: Key "rules": Key "fp/no-mutating-methods":
	Value [{"allowedObjects":["state"]}] should NOT have more than 0 items.

    at RuleValidator.validate (/home/ether/Desktop/repos/eslint-rewrite-bug/node_modules/eslint/lib/config/rule-validator.js:170:27)
    at new Config (/home/ether/Desktop/repos/eslint-rewrite-bug/node_modules/eslint/lib/config/config.js:243:27)
    at [finalizeConfig] (/home/ether/Desktop/repos/eslint-rewrite-bug/node_modules/eslint/lib/config/flat-config-array.js:216:16)
    at FlatConfigArray.getConfigWithStatus (/home/ether/Desktop/repos/eslint-rewrite-bug/node_modules/@eslint/config-array/dist/cjs/index.cjs:1102:55)
    at FlatConfigArray.getConfig (/home/ether/Desktop/repos/eslint-rewrite-bug/node_modules/@eslint/config-array/dist/cjs/index.cjs:1120:15)
    at /home/ether/Desktop/repos/eslint-rewrite-bug/node_modules/eslint/lib/eslint/eslint.js:764:40
    at async Promise.all (index 0)
    at async ESLint.lintFiles (/home/ether/Desktop/repos/eslint-rewrite-bug/node_modules/eslint/lib/eslint/eslint.js:759:25)
    at async Object.execute (/home/ether/Desktop/repos/eslint-rewrite-bug/node_modules/eslint/lib/cli.js:498:23)
    at async main (/home/ether/Desktop/repos/eslint-rewrite-bug/node_modules/eslint/bin/eslint.js:158:22)

What did you expect to happen?
The fixupPluginRules() from @eslint/compat should fix the fp/no-mutating-methods rule.
What actually happened?
It crashed it instead.
Link to Minimal Reproducible Example
https://github.com/ehmicky/eslint-rewrite-bug
Participation

[ ] I am willing to submit a pull request for this issue.

Additional comments
eslint-plugin-fp is unmaintained, so I cannot expect an upgrade to ESLint 9, and need to rely on @eslint/compat instead.
However, their rule seems to be correct: https://github.com/jfmengels/eslint-plugin-fp/blob/205861e874a4db8caedfbe2cc8af599d8d45b475/rules/no-mutating-methods.js#L70-L80. The above configuration worked fine with ESLint 8.
Please note that:

If I pass an empty object as options to that rule, it fails with the same error message
But if I pass no rule options, the rule works

The problem is that fp/no-mutating-methods specifies schema as a top-level property instead of a property of the meta object:
https://github.com/jfmengels/eslint-plugin-fp/blob/205861e874a4db8caedfbe2cc8af599d8d45b475/rules/no-mutating-methods.js#L82-L92
Top-level schema property in object-style rules was never officially supported, I believe, but was working in ESLint v8 by chance because it was picked up by code that was supposed to handle schema property of function-style rules.
I think we could update @eslint/compat to copy top-level schema into meta.schema. @nzakas what do you think?

I think we could update @eslint/compat to copy top-level schema into meta.schema. @nzakas what do you think?

That seems like the easiest solution to me. :+1:

I can work on this.

Thanks @mdjermanovic!
I was having the same problem with eslint-plugin-filenames, and it turns out it also incorrectly sets module.exports.schema.

@ehmicky yes, that was the original format of rules that we deprecated and removed in v9.


Incorrect area calculation in beams3d_distnorm.f90

In beams3d_distnorm we find the volume of a (s, u, v) cell by approximating it as a polyhedron, where the end sides are trapezoids.
The formula for (poloidal) area is incorrect, by unclear amounts.
 ! Assume parallel piped
area = abs( (rt(2) - rt(1)) *(zt(3) - zt(1)) - (rt(3) - rt(1)) *(zt(2) - zt(1))) + &
       abs( (rt(3) - rt(2)) *(zt(4) - zt(2)) - (rt(4) - rt(2)) *(zt(3) - zt(2)))

I know the (1/2) has been folded into the 8 a few lines below.

I see why we want the abs there, but it's not clear that you can separately take abs of two parts of the formula and them sum them. Better to take abs once.

I'm not sure where this formula came from.

It looks like a variant of the "Trapezoid formula" for polygon area (See the "Trapezoid formula" heading under https://en.wikipedia.org/wiki/Shoelace_formula) but with only subtraction rather than addition and subtraction?

Since I'm not sure where it came from I'm not sure of an easy fix. I would implement the Shoelace formula or Trapezoid Formula from scratch probably. The latter has fewer multiplications.

As an example of how it's incorrect, [putting back in the leading factor of 1/2]
trying it on the simple square ((r,z)) = ((0,0), (1,0), (1,1), (0,1)) gives an area of 1/2 rather than 1.
trying it on the set of points ((r,z)) = ((0,0), (1,0), (3,1), (0,1)) gives an area of 3/2 rather than 2.

@kudav Can you take a look at this?

I also stumbled over this formula when going over how we handle the distribution, and I believe it is correct. We still should document it better, or change to the Shoelace formula anyway as that would make the code more readable.
What we're calculating is the area of the two triangles comprising the quadrilateral given their coordinates, and then taking the absolute value of each of their surfaces to avoid checking the orientation of the points (not sure this is even necessary given how the points are set up).
As for your examples, when I plugged in your coordinates I arrived at the correct area values:
1/2*(|1-0|+|0+1|)=1
1/2*(|1-0|+|2+1|)=2

Sorry about this false bug report! I see now where I made a typo in my own calculations. Thanks for the two-triangles explanation; this makes sense to me now. Clever.

I'm reopening this because the area formula is in fact incorrect - in some cases by a factor of 2.
The first three of these are simple test cases where the Shoelace formula agrees with the one implemented.
For the testPoly4 points, the formula gives 8/3 instead of 2.
For the testPoly5 points the formula gives 0.000098 instead of 0.000049. Note that this last case is a bit degenerate as the first point equals the last point.
testPoly1 = {{0, 0}, {1, 0}, {1, 1}, {0, 1}};
testPoly2 = {{0, 0}, {0, 0}, {1, 1}, {0, 1}};
testPoly3 = {{1, 0}, {1, 0}, {1, 1}, {0, 1}};
testPoly4 = {{0, 0}, {2, -1/2}, {4/3, 3/2}, {2/3, 1}};
testPoly5 = {{5.933545022471559, 0.12823973812240586},
                     {5.942822499927604, 0.12512101909504394},
                      {5.940232620280285, 0.13658430586585163},
                       {5.933545022471559, 0.12823973812240586}};

I apologize again and withdraw my objection. I hadn't realized that the points (1,2,3,4) in the beams3d_distnorm code are not defined in either clockwise or counterclockwise order, which was leading to factors of 2 different in my re-implementation.


Change Proxy Function Failing

I'm getting an error when inputting this code:
async function changeProxyAdmin(
  proxyAddress: string,
  newAdmin: string,
): Promise<void>

Error:
  proxyAddress: '0xstring',
              ^

SyntaxError: Unexpected token ':'
    at Object.compileFunction (node:vm:352:18)
    at wrapSafe (node:internal/modules/cjs/loader:1032:15)
    at Module._compile (node:internal/modules/cjs/loader:1067:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1155:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
    at node:internal/main/run_main_module:17:47

I changed out the code of the contract address, but you get the point. The code seems to be missing something. I am using Node.JS 17. I am running this code in a hardhat upgrade.js file.
Any insight?

I didn't understand I had to call upgrades.admin.changeProxyAdmin(proxyAddress, newAdmin);


Retry nanosleep on EINTR

If nanosleep is interrupted, retry with the "remaining" time for sleeping the requested time.
Signed-off-by: Ponnuvel Palaniyappan<EMAIL_ADDRESS>

Thanks!


Collision Presets

I was looking through the "Snippets for Static and Skeletal Meshes" example file and was wondering if it is even possible to access and change the collision presets in the static mesh editor.

Yes, you can access both the KAggregateGeom as well as kdop stuff
https://github.com/20tab/UnrealEnginePython/blob/master/examples/kdop.py

Oh ok, sorry, those data are in the BodyInstance property (it is a struct, so you need to call .fields() on it to get its members)

Oh I didn't know about the "fields()" Thanks for the tip!
staticmesh.BodySetup.DefaultInstance.CollisionProfileName = "NoCollision"


Problem using thick borders

Since showybox passed from v0.2.0 to v0.2.1, now when the user chooses thick borders (like 15pt borders), the showybox's body looses their internal radius and, also, its interal inset. Compared with v0.2.0, this looks kind of rough.
Here's 15pt border width in v0.2.0

And here's 15pt border width in v0.2.1

Maybe this is just esthetic, and probably no one would use thicker borders than 5pt, but it would be nice to find a "beautiful way" to use thick borders

Solved adding a fill property in the body, so inset is respected. Images of the solutioned version is below

Changing that property makes 1pt borders look thiner than 1pt. So, in order to preserve coherence, this issue must be ignored.


Replace CURDIR in Makefile with the absolute path of the open-match directory

Some of the targets in the Makefile are based on the "CURDIR" variables (e.g. the proto path argument in "all-protos" target), which makes the build prone to fail if users invoke the makefile from any other directories or does not follow the same workspace setup as the README suggest. It would make more sense if we could use the absolute path of the Makefile instead if considering the above cases.

Resolved by #189