[BUG] CostManagment: Deserialization error "ResourceType is null"

Library name and version
Azure.ResourceManager.CostManagement 1.0.1
Describe the bug
When retrieving exports, the runHistory objects specifies null values for "name", "type", and "eTag".
This causes the deserializer to fail, bceause the ResourceType is null.
at Azure.Core.Argument.AssertNotNullOrWhiteSpace(String value, String name)
at Azure.Core.ResourceType..ctor(String resourceType)
at Azure.ResourceManager.CostManagement.Models.ExportRun.DeserializeExportRun(JsonElement element)
at Azure.ResourceManager.CostManagement.Models.ExportExecutionListResult.DeserializeExportExecutionListResult(JsonElement element)
at Azure.ResourceManager.CostManagement.CostManagementExportData.DeserializeCostManagementExportData(JsonElement element)
at Azure.ResourceManager.CostManagement.CostManagementExportCollection.<>c__DisplayClass8_0.b__1(JsonElement e)
at Azure.Core.PageableHelpers.PageableImplementation1.<GetAsyncEnumerator>d__15.MoveNext()    at Azure.Core.PageableHelpers.PageableImplementation1.d__15.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
at System.Linq.AsyncEnumerable.<g__Core|424_0>d1.MoveNext() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36    at System.Linq.AsyncEnumerable.<<ToListAsync>g__Core|424_0>d1.MoveNext() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
at System.Threading.Tasks.ValueTask`1.get_Result() in System.Threading.Tasks\ValueTask.cs:line 484
at Company.Azure.Utilities.Azure.BillingManagementClient.d__15.MoveNext()
Example payload from the portal, also has nulls for runHistory entries

Expected behavior
The payload should not contain null for type. I should be able to use sdk to retrieve data and deserialize without exceptions.
Actual behavior
GET https://management.azure.com/providers/Microsoft.Billing/billingAccounts/:accountId/billingProfiles/:profile-id/providers/Microsoft.CostManagement/exports/:export-name/runHistory?api-version=2023-07-01-preview
is returning a payload where the name, type and etag for the runHistory object is null. type must not be null, and be a valid ResourceType so that https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/costmanagement/Azure.ResourceManager.CostManagement/src/Generated/Models/ExportRun.Serialization.cs#L153 does not throw an exception.
{
  "value": [
    {
      "id": "/providers/Microsoft.Billing/billingAccounts/billing-account-id/billingProfiles/billing-profile-id/providers/Microsoft.CostManagement/exports/companyName-pricing/Run/964686a1-e832-4170-8899-0191e50559f2",
      "name": null,
      "type": null,
      "eTag": null,
      "properties": {
        "runSettings": {
          "format": "Parquet",
          "deliveryInfo": {
            "destination": {
              "resourceId": "/subscriptions/b5ac0f6f-7a51-4461-a2e5-53add7ed0d3d/resourceGroups/companyName/providers/Microsoft.Storage/storageAccounts/companyNameb5ac0f6f",
              "container": "companyName",
              "rootFolderPath": "exports/billing-account-id/billing-profile-id"
            }
          },
          "definition": {
            "type": "PriceSheet",
            "timeframe": "TheCurrentMonth",
            "dataSet": {
              "configuration": {
                "columns": [],
                "dataVersion": "2023-05-01",
                "filters": []
              },
              "granularity": null
            }
          },
          "partitionData": true,
          "dataOverwriteBehavior": "OverwritePreviousReport",
          "compressionMode": "snappy",
          "exportDescription": ""
        },
        "executionType": "OnDemand",
        "status": "Completed",
        "submittedBy": "",
        "submittedTime": "2024-10-09T12:01:19.2523149Z",
        "processingStartTime": "2024-10-09T12:11:22.2227377Z",
        "processingEndTime": "2024-10-09T13:03:33.0343405Z",
        "fileName": null,
        "manifestFile": "exports/billing-account-id/billing-profile-id/companyName-pricing/20241001-20241031/964686a1-e832-4170-8899-0191e50559f2",
        "startDate": "2024-10-01T00:00:00",
        "endDate": "2024-10-31T23:59:59.999"
      }
    }
   ]
}

Reproduction Steps
var armClientOptions = new ArmClientOptions();
armClientOptions.SetApiVersion(CostManagementExportResource.ResourceType, "2023-07-01-preview"); // same version used by portal...
var armclient = new ArmClient(credential, defaultSubscriptionId: null, options: armClientOptions);
var scope = ResourceIdentifier.Parse("/providers/Microsoft.Billing/billingAccounts/accountid/billingProfiles/profileid")
var export = await armclient.GetCostManagementExportAsync(scope, "my-export-name", "runHistory"); // exception!

Environment
Microsoft Windows [Version 10.0.22631.4169]
dotnet --info
.NET SDK:
Version:           8.0.306
Commit:            7f060a8005
Workload version:  8.0.300-manifests.86714225
MSBuild version:   17.10.4+10fbfbf2e
Runtime Environment:
OS Name:     Windows
OS Version:  10.0.22631
OS Platform: Windows
RID:         win-x64
Base Path:   C:\Program Files\dotnet\sdk\8.0.306\
.NET workloads installed:
There are no installed workloads to display.
Host:
Version:      8.0.10
Architecture: x64
Commit:       81cabf2857
.NET SDKs installed:
6.0.321 [C:\Program Files\dotnet\sdk]
7.0.317 [C:\Program Files\dotnet\sdk]
8.0.204 [C:\Program Files\dotnet\sdk]
8.0.206 [C:\Program Files\dotnet\sdk]
8.0.306 [C:\Program Files\dotnet\sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 6.0.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.26 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.35 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.26 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.35 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.18 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Other architectures found:
x86   [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]
Environment variables:
Not set
global.json file:
Not found
Learn more:
https://aka.ms/dotnet/info
Download .NET:
https://aka.ms/dotnet/download
visual studio
Microsoft Visual Studio Professional 2022
Version 17.9.6
VisualStudio.17.Release/17.9.6+34728.123
Microsoft .NET Framework
Version 4.8.09032
Installed Version: Professional
ASP.NET and Web Tools   <IP_ADDRESS>61
ASP.NET and Web Tools
Azure App Service Tools v3.0.0   <IP_ADDRESS>61
Azure App Service Tools v3.0.0
Azure Functions and Web Jobs Tools   <IP_ADDRESS>61
Azure Functions and Web Jobs Tools
C# Tools   4.9.0-3.24121.1+a98c90d56455379836dd5c845b35fa932b00cfa3
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.
Common Azure Tools   1.10
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.
Git Web Links   2.10.0
Copy links to files in their online Git repositories.
GitHub Copilot   <IP_ADDRESS> (v<IP_ADDRESS>@218a5ecac)
GitHub Copilot is an AI pair programmer that helps you write code faster and with less work.
GitHub Copilot Agent   1.147.0
Microsoft JVM Debugger   1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines
NuGet Package Manager   6.9.2
NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/
Razor (ASP.NET Core)   <IP_ADDRESS>2004+684740676513b3b208d5bd5b7f8786dab8d287b7
Provides languages services for ASP.NET Core Razor.
SQL Server Data Tools   <IP_ADDRESS>
Microsoft SQL Server Data Tools
TypeScript Tools   17.0.30103.2001
TypeScript Tools for Microsoft Visual Studio
Visual Basic Tools   4.9.0-3.24121.1+a98c90d56455379836dd5c845b35fa932b00cfa3
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.
Visual F# Tools   17.9.0-beta.23614.3+a521e1cd420beb56c15faf6836184fadd2b7937a
Microsoft Visual F# Tools
Visual Studio IntelliCode   2.2
AI-assisted development for Visual Studio.

@jsquire thx for the quick triage. It's pretty surprising to see the API return nulls for the name and type. Those are pretty fundamental fields in the ArmResource model.

@jsquire
I wanted to share a work-around I made. I was able to use the HttpPipeline to add a rewriting mechanism to fix the type. It's pretty great this capability exists in the SDK, but obviously the issue is the payload is invalid with regards to the SDK's validation rules.
var armClientOptions = new ArmClientOptions();
var policy = new ModifyResponseBodyPolicy();
policy.Modifications.Add(new GetExportListWithRunHistory());
policy.Modifications.Add(new GetExportWithRunHistory());
armClientOptions.AddPolicy(policy, HttpPipelinePosition.PerCall);
armClientOptions.SetApiVersion(CostManagementExportResource.ResourceType, _costManagementExportsApiVersion); // would have used "2023-03-01"
var armclient = new ArmClient(credential, defaultSubscriptionId: null, options: armClientOptions);

public class ModifyResponseBodyPolicy : HttpPipelinePolicy
{
    public List<ResponseBodyModification> Modifications { get; init; } = new List<ResponseBodyModification>();

    public override void Process(HttpMessage message, ReadOnlyMemory<HttpPipelinePolicy> pipeline)
    {
        // Call the next policy in the pipeline.
        ProcessNext(message, pipeline);

        // Modify the response body here.
        foreach (var modification in Modifications)
        {
            if (modification.IsMatch(message))
            {
                ModifyResponseBody(message, modification);
            }
        }
    }

    public override async ValueTask ProcessAsync(HttpMessage message, ReadOnlyMemory<HttpPipelinePolicy> pipeline)
    {
        // Call the next policy in the pipeline.
        await ProcessNextAsync(message, pipeline);

        // Modify the response body here.
        foreach (var modification in Modifications)
        {
            if (modification.IsMatch(message))
            {
                ModifyResponseBody(message, modification);
            }
        }
    }

    private void ModifyResponseBody(HttpMessage message, ResponseBodyModification modification)
    {
        // Check if the response content is not null.
        if (message.Response.ContentStream != null)
        {
            // Read the response content stream.
            using var reader = new StreamReader(message.Response.ContentStream);
            var content = reader.ReadToEnd();

            var modifiedContent = modification.RewriteResponseBody(content);

            // Update the response content stream with the modified content.
            var modifiedStream = new MemoryStream(Encoding.UTF8.GetBytes(modifiedContent));
            message.Response.ContentStream = modifiedStream;
        }
    }
}

public static class RequestExtensions
{
    public static bool QueryStringHasKeyValue(this Request request, string key, string value)
    {
        var queryCollection = AzureHttpUtility.ParseQueryString(request.Uri.Query);
        return queryCollection.TryGetValue(key, out string? queryValue) && queryValue == value;
    }
}

public abstract class ResponseBodyModification
{
    public abstract bool IsMatch(HttpMessage message);
    public abstract string RewriteResponseBody(string responseBody);
}

public class GetExportListWithRunHistory : ResponseBodyModificationRule
{
    public override bool IsMatch(HttpMessage message)
    {
        var request = message.Request;

        return request.Method == RequestMethod.Get
               && Regex.IsMatch(request.Uri.Path, "/providers/Microsoft.CostManagement/exports$")
               && message.Request.QueryStringHasKeyValue("$expand", "runHistory")
               && message.Request.QueryStringHasKeyValue("api-version", "2023-07-01-preview");
    }

    public override string RewriteResponseBody(string json)
    {
        // Parse the JSON into a JObject
        var jObject = JObject.Parse(json);
        var tokens =<EMAIL_ADDRESS>== null)]");
        foreach (var t in tokens)
        {
            t["type"] = "Microsoft.CostManagement/run";
        }
        // Convert the modified JObject back to a JSON string
        return jObject.ToString(Formatting.Indented);
    }
}

public class GetExportWithRunHistory : ResponseBodyModificationRule
{
    public override bool IsMatch(HttpMessage message)
    {
        var request = message.Request;

        return request.Method == RequestMethod.Get
               && Regex.IsMatch(request.Uri.Path, "^.+/providers/Microsoft\\.CostManagement/exports/[a-z0-9-]+$")
               && message.Request.QueryStringHasKeyValue("$expand", "runHistory")
               && message.Request.QueryStringHasKeyValue("api-version", "2023-07-01-preview");
    }

    public override string RewriteResponseBody(string json)
    {
        // Parse the JSON object
        var jsonObject = JObject.Parse(json);

        var tokens =<EMAIL_ADDRESS>== null)]");
        foreach (var t in tokens)
        {
            t["type"] = "Microsoft.CostManagement/run";
        }

        // Serialize the modified JSON object back to a string
        return jsonObject.ToString(Formatting.Indented);
    }
}

@jeremysimmons, kudos on the great use of pipeline policies to work around this issue! 🎉
@jsquire, I'd love to look at some point at how we handle nulls in deserialization routines ... when I was looking at generating clients will nullable reference types annotations, I spotted some of these.  I'm not sure it would solve the issue of services sending invalid payloads, but it would be easier to understand where some of these issues are and communicate optionality of field values to client users via public APIs ... might be worth spending some cycles on if we can get to it.

@annelo-msft thanks! that's high praise coming from you...
Please let me know if I can provide more details / context of what's going on.

Hey @jeremysimmons, as you mentioned, this issue is a service side problem that the return value of a ArmResource doesn't follow the ARM definition. You can open an Azure support request to ask service team fix it.
In the meantime, we can attempt a workaround using customization code. @HarveyLink , could you please assist with this?

@jeremysimmons, kudos on the great use of pipeline policies to work around this issue! 🎉
@jsquire, I'd love to look at some point at how we handle nulls in deserialization routines ... when I was looking at generating clients with nullable reference type annotations, I spotted some things we might be able to address there. I'm not sure it would solve the issue of services sending invalid payloads, but it would be easier to understand where some of these issues are and communicate optionality of field values to client users via public APIs ... might be worth spending some cycles on if we can get to it.

Hey @annelo-msft , we don't need to handle this issue since type and name are required properties of a ArmResource model, and their values should never be null.

@ArthurMa1978 do you mean some other "customization code" different than what I suggested?
@HarveyLink do you have suggestions?

@annelo-msft Can you confirm that @ArthurMa1978 's suggestion to open an Azure support request is going to actually route to the team that owns the API service. My initial response from them seems to be someone supporting the Azure Portal (UI), and not the actual CostManagement API Service that the Azure.ResourceManager.CostManagement sdk package is acting as a client for. As a developer, it's frustrating to find a specific bug with the API response, and not be able to get in touch with the team that owns the API quickly...

@jeremysimmons :  Unfortunately, that's not something that the maintainers of the Azure SDK have insight into nor influence over.   Support tickets generally start with the Microsoft support team.  I'd let them know that you're looking for the team that owns the ARM Resource Provider for Cost Management and that there's an error in their REST API.   You're welcome to tell them to reach out to me internally if they need additional context.

@jsquire THANK YOU! Knowing the "magic words" to speak in the support ticket is the first step. I really appreciate having people to engage via github. Thanks for everything and the rest of the team do.

Unfortunately, that's not something that the maintainers of the Azure SDK have insight into nor influence over.

I wouldn't have expected that the teams would be so dramatically separated. It's big company though.
I think at this point, ya'll on the sdk team have done as much as possible, so I'll close the ticket.

I wouldn't have expected that the teams would be so dramatically separated. It's big company though.

Yeah, I understand how that would be confusing and frustrating.  Unfortunately, that gets accentuated for open-source products like the Azure SDK, where we're able to talk directly with you while service teams are required to go through the support channel.

@ArthurMa1978 do you mean some other "customization code" different than what I suggested? @HarveyLink do you have suggestions?

It’s different. Your code provides an excellent workaround from a usage perspective. The customization code within the Azure SDK serves as a mechanism to handle some unexpected behaviors.

@ArthurMa1978 is the "customization code" something I'm expected to write? Do you have an example of the code?

@ArthurMa1978 is the "customization code" something I'm expected to write? Do you have an example of the code?

@HarveyLink will give a sample pr

@jsquire @ArthurMa1978 sorry to keep pinging you on a closed ticket. I got a reply from the azure service provider team, ticket #2410100040012460, but this makes no sense to me.

Kindly note that the Engineering Team has reached indicating that our API always returns these properties as null which is being made by design. In this case, the Engineering Team suggested to please adjust the script to ignore these properties completely.

Can you please provide me some advice on how to tell them that these properties are Required by the Azure Resource model?
From what I can tell looking at the rest api specs, they should be included. They're not explicitly marked required, only "readonly". The Export entity uses the same base as ExportRun which is CostManagementProxyResource. It doesn't make sense that Export includes the properties but ExportRun does not...
runHistory is a property of CommonExportProperties
https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cost-management/resource-manager/Microsoft.CostManagement/stable/2023-08-01/costmanagement.exports.json#L476
ExportExecutionListResult has items that are ExportRun
https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cost-management/resource-manager/Microsoft.CostManagement/stable/2023-08-01/costmanagement.exports.json#L695
ExportRun includes allOf CostManagementProxyResource
https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cost-management/resource-manager/Microsoft.CostManagement/stable/2023-08-01/costmanagement.exports.json#L714
CostManagementProxyResource defines 4 properties, id, name and type. They are all defined as "readonly".
https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cost-management/resource-manager/Microsoft.CostManagement/stable/2023-08-01/common-types.json#L165
The Export entities is also defined as having allOf CostManagementProxyResource
https://github.com/Azure/azure-rest-api-specs/blob/6e95d1f68657c6b1abfae889b6a12fe12ce67eaa/specification/cost-management/resource-manager/Microsoft.CostManagement/stable/2023-08-01/costmanagement.exports.json#L415

Hi @jeremysimmons.   No worries at all.   I'm sorry this has been such a frustrating experience.  The friction in support process for these areas is certainly regrettable.
@ArthurMa1978:  If I understand correctly, it looks like we've got an input-output type where the output is not returning data, but it is required for input so it would have to be artificially populated.   Is that a mismatch in the spec or in the generation?   It seems these should be symmetrical - either they're round-trip and always returned or they're not required for updates (which would require PATCH support, I believe.).
I don't have a good feel for ARM contracts and would appreciate your thoughts.  Likewise, do we have an escalation path or contact that we can pursue?  I'd like to get our PM team following up if we've got an issue with the spec that blocks client scenarios.

Thank you Jesse. I think the annotation of "readonly" is correct. I don't expect to be able to change either the name or the type.
I think the service team is mistaken in saying that the values are "null by design". I don't see how that could be valid.

They should define 2 models for input and output instead of use one model.


Suggest a Revision in Makefile Section clean-amazon-cluster

The Makefile  is not effective enough in the section clean-amazon-cluster. Several components remained undeleted after make clean-launch and make clean-amazon-cluster. I suggest to add a clean-amazon-launch section with the following script
clean-launch
clean-amazon-cluster
kubectl delete secrets amazon-secret
kubectl delete persistentvolumes rethink-volume
kubectl delete persistentvolumeclaims rethink-volume-claim

The script has been merged in my program locally and it works to deploy amazon.

Hi @ShengjieLuo thanks for the issues this seems like a great change. Btw when you have a change this specific in mind you can submit it as a Pull Request. If you've made the change to a forked copy then I think all you have to do is push it to GH and go to the repo and you'll get a big PR button. As a bonus you get your name in our contributors list 😸.
If that's not feasible for some reason we'll happily add this, but latency on this stuff will invariably bet a bit slower than if there's just a big green merge button I can hit.

FYI here is a tutorial on pull requests: https://help.github.com/articles/using-pull-requests/

merged


Fix | rendering problem by adding a single space

See https://github.com/vmg/redcarpet/issues/546 for more on how redcarpet cares about spaces. There's a lax_spacing setting that might be part of the problem or solution, but as is right now just remembering to add some extra space now and again is a good enough quick fix.

This change is

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/dcc0a24f63e1c7f61c2b0f9f7b02a2feea8daa14/

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/db78cbdbf7db91ae9c13b45562d5fd15158ebc0b/

@sesam, unfortunately, this doesn't render any differently. See http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/db78cbdbf7db91ae9c13b45562d5fd15158ebc0b/stable/use-the-built-in-sql-client.html#sql-shell-commands.

Oh, strange. Maybe something "normalizes" the markdown source before
rendering? I noticed my added spaces get removed when I reopen the page
source. How about adding space   space?
On Sep 9, 2017 1:33 PM, "Jesse Seldess"<EMAIL_ADDRESS>wrote:

@sesam https://github.com/sesam, unfortunately, this doesn't render any
differently. See http://cockroach-docs-review.s3-website-us-east-1.
amazonaws.com/db78cbdbf7db91ae9c13b45562d5fd15158ebc0b/stable/use-the-
built-in-sql-client.html#sql-shell-commands.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cockroachdb/docs/pull/1895#issuecomment-328271547,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAi2YWL9Wnc9FXtKkmMR42NC5MX-wGkks5sgneBgaJpZM4PRlpT
.

Think I figured this out. See https://github.com/cockroachdb/docs/pull/1896. Thanks for your efforts, @sesam!


Support Notebook comments on Markup cells

I would like code review comments in notebooks to also work on non-code cells. Currently the code explicitly filters to only code cells, I am not sure why: https://github.com/microsoft/vscode/blob/dee8eaf52679a71ee790e3c09c1160e1ab037b99/src/vs/workbench/contrib/notebook/browser/view/cellParts/cellComments.ts#L157-L164
Is there a reason why comments on markup cells should be hidden, or would you accept a PR to add support for comments on Markup Cells?
@alexr00

@rehmsen that was a pretty early exploration and we don't have support for the notebook commenting officially. With that said, I'm good with PRs for enabling it for Markup cell.

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

I took a look at this today and wanted to propose a rough plan for how to implement this:

Add a new property commentOffset to the CodeCellLayoutInfo, so that cellComments does not need to know that the comments should go below the outputs - that computation would move into CodeCellViewModel.layoutChange. This is the first step to making cellComments.ts agnostic of the type of cell.
Extract shared interfaces CellLayoutInfo and CellLayoutChangeEvent from CodeCellLayoutInfo/MarkupCellLayoutInfo and CodeCellLayoutChangeEvent/MarkupCellLayoutChangeEvent. They already share a bunch of fields, and actually putting them into a parent interface means that cellComments.ts can read and write the relevant fields (commentHeightandcommentOffset`) without needing to distinguish between Code and Markup cells.
(Optional) share some of the logic of how these shared layout values are computed. That is not strictly necessary and in the case of offsets might not be possible because they depend on which UI elements are above them pushing them down. But those things that are the same could benefit from some deduping. That code could live in BaseCellViewModel.
Make MarkupCellViewModel compute commentOffset based on its view hierarchy.
Make cellComments use the commentHeight setter and the cellOffset getter from the ICellViewModel instead of casting to CodeCellViewModel.

@alexr00 @rebornix Please let me know if that sounds good to you.

@rehmsen could I get some verification steps for this feature request?
I haven't been able to find a way to add comments to a notebook.
The diff editor is also editable on the right side, which seems off to me.

@rebornix Can you help?
I also do not know how to add a code review comment for a notebook via the UI. We have a closed source extension which registers a comment provider that creates notebook comments, and I verified that those work on Markdown cells with the fix.
Open source I know that the Github Pull Requests extension provides Github comments, but those are always left on the raw file, not on notebook cells, see #214017 - I have suggested some ideas how one might fix that, but we have not reached agreement yet.
I there some other way to create a notebook comment @rebornix? I guess you had some way to verify this when you build the original comment support?
Regarding the diff editor, editing the right side is expected I think - the left side should not be editable. Is it? Is that caused by my change somehow? I tried to repro, but when I run ./scripts/code-web.sh, I cannot install any notebook extension (like jupyter) in the first place.

Marking this issue as verified considering the fix works for you.
Also, the left side of the diff editor is not modifiable. I'll check whether the diff editor is modifiable on the right side on Stable.


Font size doesn't change for some ePubs

The font size doesn't change for this ePub, and some others:
http://www.desiringgod.org/books/the-pilgrim-s-progress.epub
Any idea why?

Fixed


JdkOpenSslEngineInteroptTest.testMutualAuthSameCerts() needs investigation

We disabled JdkOpenSslEngineInteroptTest.testMutualAuthSameCerts() as the test was quite flaky. We need to investigate.
See also https://github.com/netty/netty/pull/11488

I think we should disable the entire JdkOpenSslEngineInteroptTest class test because of repetitive test failures.
Should I do a PR?

@hyperxpro see https://github.com/netty/netty/pull/11513


Create frontend layout

Figma Link:
https://www.figma.com/design/0bLAmyRfjGEnX46mOAlq4G/Ghana-Hazard-Report?node-id=395-2783&node-type=frame&t=rjJy2tfyVjD6vzIY-0

i will like to work on the frontend layout please


[Feature] How can i follow logs of subgraph ?

Description
My idea: I want to know where the store logs are and use the cloudwatch agent follow that
Are you aware of any blockers that must be resolved before implementing this feature? If so, which? Link to any relevant GitHub issues.
No response
Some information to help us out

[X] Tick this box if you plan on implementing this feature yourself.
[X] I have searched the issue tracker to make sure this issue is not a duplicate.

@Jannis


[UE4] Phonon Probe Volume and Phonon Geometry not working on Instanced Static Meshes

We've just released Steam Audio 4.1.0, which includes an updated Unreal Engine plugin: https://github.com/ValveSoftware/steam-audio/releases/tag/v4.1.0.
Try it out and let us know if you continue to face issues in the new version.

Hello
Instanced static meshes are still not supported I believe.


Cached node on ubuntu not set up correctly

Describe the bug
I think this commit introduced a bug and isn't setting up node modules correctly.
Area for Triage: JavaScript and Node.js
Question, Bug, or Feature?: Bug
Virtual environments affected

[ ] macOS 10.15
[x] Ubuntu 16.04 LTS
[x] Ubuntu 18.04 LTS
[ ] Windows Server 2016 R2
[ ] Windows Server 2019

Expected behavior
We should be able to install a different version of npm after installing node.
Actual behavior
Trying to do this results in an error: Error: Cannot find module 'semver'
You can see this behaving correctly on a downloaded version of node and incorrectly on the cached version. In this example, I use Node 8.16.2 since that's the highest version that isn't the cached version (8.17.0) - https://dev.azure.com/vstscix/damccormTest/_build/results?buildId=115122&view=logs&j=4021d9ab-6287-5638-9fd6-4c6c049820ed&t=b5b3e3e4-1b78-582b-1ea4-c44df544cec9. Uses the following yaml:
trigger:
- master

jobs:
- job: cachedNode
  pool:
    vmImage: 'ubuntu-16.04'

  steps:
  # Use node 8, npm 5
  - task: NodeTool@0
    displayName: Use node 8
    inputs:
      versionSpec: "8.x"

  - script: npm i -g<EMAIL_ADDRESS>--force
    displayName: Use npm version 6.9.0

- job: downloadedNode
  pool:
    vmImage: 'ubuntu-16.04'

  steps:
  - task: NodeTool@0
    displayName: Use node 8
    inputs:
      versionSpec: "8.16.2"

  - script: npm i -g<EMAIL_ADDRESS>--force
    displayName: Use npm version 6.9.0

cc the following people who created or reviewed the pr: @dmitry-shibanov @sergey-akhalkov @AlenaSviridenko @maxim-lobanov

Hello @damccorm , Thank you for the report.
We are aware about this issue and it affects only Ring 0 currently.
We have done fix on Provisioner side and it should be deployed to Ring 0 today later

@damccorm should be fixed now. Could you check, please?

Looks like its working now, thanks for the quick turnaround!
Successfully build - https://dev.azure.com/vstscix/damccormTest/_build/results?buildId=115149&view=logs&j=09020321-c96b-5abf-ada0-57823a408a7d&t=09020321-c96b-5abf-ada0-57823a408a7d


Add key combination shortcuts to BrList for Primary+Home/End to select/scroll first/last item

Add key combination shortcuts to BrList for Primary+Home/End to select/scroll first/last item

https://github.com/feenkcom/Brick/commit/6187b32f809911982fc73733a2b34935b2a92a68


Support dynamic presence of properties

Prerequisites

[x] I have read the documentation;
[x] In the case of a bug report, I understand that providing a SSCCE example is tremendously useful to the maintainers.

Description
I would like a way to define a schema where some properties should be present if and only if another property has some value(s) and not present otherwise.
Steps to Reproduce

Define a JSON Schema defining properties that sometimes should be present and sometimes should not be (e.g. using dependencies with oneOf, not, and required).
Render the schema on a react-jsonschema-form and attempt and try it out.

https://jsfiddle.net/07fm4n1b/
Expected behavior
react-jsonschema-form reacts to the "oneOf" array in the "dependencies" and intelligently displays only the relevant properties and when values change and some properties become relevant or no longer relevant they are shown or hidden appropriately.
Actual behavior
All of the properties are displayed requiring the user to understand which properties they need to fill and which ones they must leave blank.
Version
0.44.0

That seems very complicated to implement properly and efficiently, no mention of maintainability. We usually recommend to process the schema when onChange is triggered to contextually augment the initial one with new definitions. This has worked great for us so far, so I'm having a hard time justifying spending a possibly huge amount of time implementing what you suggest.
I'm closing for now, though if you're willing to work on this, feel free to reopen.

I've recently implemented something like this using a custom field template. I've added additional entries to the uiSchema:
{
  "ConditionallyEnabledfield": {
    "ui:options": {
      "enabledIf": "SomeOtherField",
      "hasValue": "yes"
    }
}

and then check them in the custom field rendering:
export class CustomFieldTemplate extends React.PureComponent {

  render() {

    const {id, label, displayLabel, rawHelp, description, rawErrors, children, schema, uiSchema} = this.props

    let childrenProps = children.props
    let disabled = false

    if(schema && schema.type === "object") {
      const { formData, uiSchema } = children.props
      childrenProps = {
        ...childrenProps,
        uiSchema: Object.keys(uiSchema).reduce((newUiSchema, key) => {
          const uiOptions = (uiSchema[key] || {})['ui:options']
          if(uiOptions
            && uiOptions.enabledIf
            && uiOptions.hasValue
            && formData[uiOptions.enabledIf] !== uiOptions.hasValue) {
            newUiSchema[key] = {
              ...uiSchema[key],
              'ui:options': {
                ...uiOptions,
                disabled: true,
              }
            }
          } else {
            newUiSchema[key] = uiSchema[key]
          }
          return newUiSchema
        }, {})
      }
    } else {
      if(uiSchema && uiSchema['ui:options'] && uiSchema['ui:options'].disabled) {
        disabled = true
        childrenProps = {
          ...childrenProps,
          disabled,
        }
      }
    }

    return (
      <Form.Field disabled={disabled} >
...
        {{...children, props: childrenProps}}
      </Form.Field>
    )
  }
}

I apologize for the code, it really looks ugly so any improvements are more than welcome.

This also looks related to #430, which is also closed until someone wants to implement it.

@misto Where is the Form.Field component defined?  I'm not seeing it in this package.  Is it your own?


Fix render pipeline for frames completely outside full image.

This fixes an uninitialized memeory access discovered by fuzzers.

Isn't it rollback of https://github.com/libjxl/libjxl/pull/1357?


jupyterlab-favorites fails to load on Jupyterlab 4.0.0a33

Description
On Jupyterlab 4.0.0a33 the jupyterlab-favorites extension fails to activate with the following error.

Looks like this comes from filebrowser being undefined in the following lines.
https://github.com/jupyterlab-contrib/jupyterlab-favorites/blob/09b69bb050059dc18bb992cfb5a98438176da4fa/src/index.ts#L39-L40
This looks to be because the defaultBrowser property of IFileBrowserFactory being removed in the following commit.
https://github.com/jupyterlab/jupyterlab/commit/7227449db8db478dc5e31bbc1ffdfb741766a35d#
Expected behavior
jupyterlab-favorites extension activates.
Context

Python package version: 3.8.1

Extension version: v3.1.1

Operating System and its version: MacOS
Browser and its version: 109.0.5414.119

Hi! Thanks for your plugin, been using it for a while. Any updates on making it compatible with v4? Thanks!


Using DuckDNS to overcome dynamic IP addresses

In this pull request, I explain why and how to use duckdns to circumvent a server IP address that always changes. This will be invaluable to 301 students who want to maintain a constant production IP address for their projects, especially if they choose to self host their own server.

I have made the requested changes now if you could take a look


ros-installer calls non-existing cos-install

At https://github.com/rancher-sandbox/os2/blob/391ea8a56549f93839373e0b1ba87204ecb72e3f/pkg/install/install.go#L110

This is caused by https://github.com/rancher-sandbox/os2/issues/16, as we are not yet aligned with cOS master and we weren't pinning packer templates previously. Closing it


Exclude non-matching localities in subset LB

We're using the subset lb with locality weights, and ran into an issue where where requests would fail when a subset excluded an entire locality. For example:
L1: (50% weight)
host1: metadata: A
host2: metadata A
L2: (50% weight)
host3: metadata B
host4: metadata B

The subset matching A results in:
L1: (50% weight)
host1
host2
L2: (50% weight)
empty

and as a result 50% of requests fail because they get routed to L2.
A thing to note is that in the case where a subset ends up with an empty LB, this assert triggers: https://github.com/envoyproxy/envoy/blob/master/source/common/upstream/upstream_impl.cc#L276
This is in line with the original warnings about using subset lb and locality weighting together, but in our case we know that the entire LocalityLbEndpoint will have the same metadata. As such I suggest one of two solutions:

Add a metadata field to LocalityLbEndpoint and have the subset LB exclude the entire locality if the metadata does not match for a given subset. This will per-locality metadata to exclude localities that do not match.

Not sure how easy this would be to implement, but wanted to see if this sounds reasonable before digging into the code too much.

Another option would be to special case (in the LB implementation perhaps?)  the case where a locality ends up with 0 weight and simply redistribute the locality weight of the empty locality between the remaining ones, proportional to their current weight. Without being super familiar with the EDF scheduler, this might just mean we exclude localities with 0 hosts from being inserted into the scheduler.

A thing to note is that in the case where a subset ends up with an empty LB, this assert triggers: https://github.com/envoyproxy/envoy/blob/master/source/common/upstream/upstream_impl.cc#L276
The ASSERT makes me think that the better solution might be to adjust the locality load (my second suggestion in the original post) so that won't be accidentally running into this ASSERT.

A thing to note is that in the case where a subset ends up with an empty LB, this assert triggers: https://github.com/envoyproxy/envoy/blob/master/source/common/upstream/upstream_impl.cc#L276
This makes me think that the better solution might be to adjust the locality load in this case (maybe behind a flag?) instead of having people accidentally running into this ASSERT.

Put up https://github.com/envoyproxy/envoy/pull/4840 as an example of how 2) can be implemented

@snowp I'm thinking (2) sounds sane, but I wonder about edge cases. For example, imagine a locality L2 in your above example that has zero hosts in one EDS update, and then gains a single host in the next. We go from 100% traffic to L1 to 50% of traffic to L1 and 50% to L2, which seems a bit step function. Maybe this corner case isn't super interesting in your case, but just throwing it out there.

Yeah, we're somewhat safe from that my aligning localities with subsets. While someone could set up a locality with only one host in it, that's not specific to the subset lb.
Would you be more comfortable if I put this behind a flag?

One option that comes to mind in order to smooth out the change in LB is to scale the weight of each locality with (number of hosts in filtered locality / number of hosts in original locality). Then, in the case you mentioned you get:
L1 weight =  50 * (2/2) = 50
L2 weight =  50 * (1/2) = 25

which gets you a 66/33 split between the two localities. This would work for our case since we'd always have 0 as the numerator and would help even out the load in cases where subsets do not align with localities.
This would still require the change to support handle zero weight localities to support our use case, so I'm thinking this could probably be a separate change.

@snowp maybe the system assigning weights should also be aware of the number of healthy endpoints in the locality and do this calculation itself? I think that would also avoid the adverse situation I described by not maintaining 50/50. Of course, that doesn't work when you have Envoys themselves HCing vs. a centralized HC and health status via EDS. Which situation do you have?

We have Envoys health checking.
Don't we already account for % of health hosts in a locality here https://github.com/envoyproxy/envoy/blob/master/source/common/upstream/upstream_impl.cc#L271 ?
Sounds like we're talking about two different issues: scaling locality weight based on host health, and scaling locality weight based on host presence due to being filtered out by the subset LB.
I think the health case is dealt with in Envoy already by the code I linked (unless I'm misunderstanding). It might be nice to match this logic for subsets (maybe make it opt-in?), though I'm not sure if including an overprovisioning factor makes sense for subsets.

@snowp yeah, that actually seems pretty nice, I misunderstood in the above reply. Do any other @envoyproxy/maintainers or other community members want to opine before we commit?

Happy to do the work for this if we're happy with the design, should be fairly straightforward. Would we want to guard this behind a configuration flag? It sounds like a sane default, but there might people relying on the existing behavior?

AFAIK, you're the only folks who have been specifically interesting in the intersection of subset LB and priorities, @zuercher do you know who else might be using this?

cc @rgs1, who is using subset lb iirc

@zuercher thanks for tagging, looking.

@htuch we (Pinterest) are interested too

@rgs1 can you comment on how the above proposal would affect you?

cc: @fishcakez as well

@htuch @snowp so scaling locality weight based on host presence due to subset LB matching sounds reasonable. Making it configurable also sounds reasonable (we could start with off by default, and then switch if over once we've gained confidence).

@snowp how would the default_subset policy affect this?

The default_subset itself is just another subset, so I'd imagine it would work exactly the same way as the subsets targeted using route metadata (ie it would also have its weights scaled by host presence)

@snowp sorry, i meant the fallback_policy when using ANY_ENDPOINT — would that mean disabling scaling locality based on subset LB matching?

@rgs1 ANY_ENDPOINT is just a subset with a predicate that matches anything, so the resulting weights should be unaffected by the proposal. Each locality will retain all its hosts, so no adjustment is made.

Sounds like the consensus here is that this seems like we should make this configurable (for now at least). I'll try to find the time to get this done sometime this week.


Fix multiple cases of invalid windows line endings handling

This is a good change, but for consistency I think we should use SourceFile.lines/1 wherever we can.

I wasn't sure if it wouldn't be misleading as those are not source files


Deprecate any_raw and Invariant

Description of changes:
Basically invariant type was only safe for T that had no invalid value independent on its binary values. All other types had some sort of UB involved. Thus, deprecate this for now.
I also removed nondet which has been deprecated for quite some time.
Resolved issues:
Related to #1394
Call-outs:
We will likely want the Invariant type reintroduced one day, but we might need to either change its form or ensure that any_raw() at least respect primitives and std type invariants to avoid users having to explicitly check for any potential UB (which in itself is UB).
Testing:

How is this change tested? New performance tests + old tests

Is this a refactor change? No

Checklist

[ ] Each commit message has a non-empty body, explaining why the change was made
[ ] Methods or procedures are documented
[ ] Regression or unit tests are included, or existing tests cover the modified code
[ ] My PR is restricted to a single feature or bugfix

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

Looks good. A much needed change!
General clarification: now, if a user wants to call kani::any on a custom type, do they manually have to implement kani::any? And their implementation would in-turn call kani::any on the sub-fields?

Yes. Users will have to call any() for each field of their struct. We provide the Arbitrary trait that allows user to implement any() for their type once and reuse that for things like array, Option, Result and others.
We are also planning to add a #[derive(Arbitrary)] macro to make this easier for the user (https://github.com/model-checking/kani/issues/726).


Crashes (includes fix)

Since I can't commit to your project, here are two fixes that I had to made in order to get the scraper to run:
In mirror_spider.py, line 50, there is no check whether the output path is valid. The URL can contain ? characters which causes the script to crash.
Here's my solution, it's just a quick fix and may require elaboration for different characters and Linux/Windows compatibility:
url_parts = response.url.split('://')[1].split('/')
parent_directory = os.path.join(self.directory, *url_parts)
parent_directory = parent_directory.replace("?", "_q_")  # normalize path

os.makedirs(parent_directory, exist_ok=True)

There is another bug in your other project scrapy_wayback_machine which is imported here, that causes a crash.
It's in init.py, line 91:
cdx_url = self.cdx_url_template.format(url=pathname2url(request.url))
At this point, request.url is something like http://website.com.
But pathname2url will look for a colon : and require that anything before that is only one letter (since we are dealing with regular paths here, like C:\mypath.
When I removed the call to pathname2url, it worked for me, but I don't know which other cases may break:
cdx_url = self.cdx_url_template.format(url=request.url)

I feel like the author should really apply all these fixes that make his trash program absolutely useless...

This should be resolved between #14 and sangaline/scrapy-wayback-machine#2. Please let me know if you're still running into any issues.


MjSpec: Deleting a body unexpectedly resets keyframe values

Intro
Hi!
I am a MuJoCo user working on manipulation.
My setup
MuJoCo:
(built from https://github.com/google-deepmind/mujoco/commit/3a17be2d24ad8b89272115ef5027b799f233ba19)
$ python -c "import mujoco; print(mujoco.__version__)"
3.2.3

API:
Python
OS:
$ python -c "import platform; print(f'{platform.system()=}, {platform.release()=}, {platform.machine()=}')"
platform.system()='Darwin', platform.release()='24.0.0', platform.machine()='arm64'

What's happening? What did you expect?
I think the test case and its comments below describes the issue quite clearly. To summarize: deleting a body that previously had a joint under it, resets keyframe values. I would expect the keyframe values to not be modified when deleting the body without joints.
Steps for reproduction
  def test_body_deletion_should_not_delete_key(self):
    spec = mujoco.MjSpec()
    spec.from_string(textwrap.dedent("""
      <mujoco model="MuJoCo Model">
        <worldbody>
          <body name="body-1">
            <joint name="joint-1" pos="0 0 0" axis="1 0 0" limited="false" />
            <geom size="1"/>
          </body>
          <body name="body-2">
            <joint name="joint-2" pos="0 0 0" axis="0 1 0" limited="false" />
            <geom size="1"/>
          </body>
        </worldbody>
      </mujoco>
    """))

    # 1. We delete the `joint-1`.
    joint_1 = spec.find_body("body-1").first_joint()
    self.assertEqual(joint_1.name, "joint-1")
    joint_1.delete()

    # 2. Add key for `joint-2`
    home_key = spec.add_key()
    home_key.name = "home"
    home_key.qpos = np.array([16.0])

    # 3. Now, detach `body-1`, which has nothing to do with `body-2` or `joint-2` and
    # the key gets reset!
    spec.detach_body(spec.find_body("body-1"))
    model = spec.compile()
    np.testing.assert_array_equal(model.key("home").qpos, 16.0)  # Is actually 0.0
    np.testing.assert_array_equal(spec.key[0].qpos, 16.0)  # Is actually 0.0

    # Here's what the XML looks like. Note that the `spec.key[0].qpos` still exists,
    # even though it's not written in the xml. The value is just incorrect (`0.0`
    # instead of `16.0`).
    self.assertEqual(textwrap.dedent("""
      <mujoco model="MuJoCo Model">
        <compiler angle="radian"/>

        <size nkey="1"/>

        <worldbody>
          <body name="body-2">
            <joint name="joint-2" pos="0 0 0" axis="0 1 0" limited="false"/>
            <geom size="1"/>
          </body>
        </worldbody>

        <keyframe>
          <key name="home"/>
        </keyframe>
      </mujoco>
    """).strip(), spec.to_xml().strip())

Minimal model for reproduction
See above.
Code required for reproduction
See above.
Confirmations

[X] I searched the latest documentation thoroughly before posting.
[X] I searched previous Issues and Discussions, I am certain this has not been raised before.

I was actually able to simplify this even further. The key gets reset even without joint-1's presence:
  def test_body_deletion_should_not_delete_key(self):
    spec = mujoco.MjSpec()
    spec.from_string(textwrap.dedent("""
      <mujoco model="MuJoCo Model">
        <worldbody>
          <body name="body-1"/>
          <body name="body-2">
            <joint name="joint" />
            <geom size="1"/>
          </body>
        </worldbody>
      </mujoco>
    """))

    # 1. Add key for `joint`
    home_key = spec.add_key()
    home_key.name = "home"
    home_key.qpos = np.array([16.0])
    model = spec.compile()

    np.testing.assert_array_equal(model.key("home").qpos, 16.0)  # Is actually 0.0
    np.testing.assert_array_equal(spec.key[0].qpos, 16.0)  # Is actually 0.0

    # 2. Detach `body-1`, which has nothing to do with `body-2` or `joint` and
    # the key gets reset!
    spec.detach_body(spec.find_body("body-1"))
    model = spec.compile()

    np.testing.assert_array_equal(model.key("home").qpos, 16.0)  # Is actually 0.0
    np.testing.assert_array_equal(spec.key[0].qpos, 16.0)  # Is actually 0.0

    # Here's what the XML looks like. Note that the `spec.key[0].qpos` still exists,
    # even though it's not written in the xml. The value is just incorrect (`0.0`
    # instead of `16.0`).
    self.assertEqual(textwrap.dedent("""
      <mujoco model="MuJoCo Model">
        <compiler angle="radian"/>

        <size nkey="1"/>

        <worldbody>
          <body name="body-2">
            <joint name="joint" />
            <geom size="1"/>
          </body>
        </worldbody>

        <keyframe>
          <key name="home"/>
        </keyframe>
      </mujoco>
    """).strip(), spec.to_xml().strip())


Upgraded library to work with specs-v1

fixes #32
fixes #50
fixes #53
fixes #24
fixes #31
fixes #63
fixes #55

@amercader
please take a look
Because of brithis holidays I've decided to put all changes into one PR to not be blocked. So there are many changes. I suppose the easiest way to review will be going commit by commit. Describing main steps:

added specs-v1 jsonschemas using starndard to the stack way with up-to-date tests - https://github.com/frictionlessdata/datapackage-js/commit/93dfd6fb3d4761b94bf9ff7e15de1c57f7526ff6
merged node/browser tests into one codebase because it was splited into two mostly identical copies making changing the actual codebase really difficult - https://github.com/frictionlessdata/datapackage-js/commit/3696312ec2ce04c8ade2905e599018237b470abf
introduced Profile class based on Python API - https://github.com/frictionlessdata/datapackage-js/commit/e5162ba00e504f9b9e980ed13ab4fda3f582cbce
updated Resource class based on Python implementation - https://github.com/frictionlessdata/datapackage-js/commit/b7d68a2b848f0874c026ad4bc1867d6bead9dd1c
update DataPackage class based on Python implementation - https://github.com/frictionlessdata/datapackage-js/commit/aa4b3a9bc9171db17b39ebbb354fa971f2052bcc

I would say the codebase state before this PR was more like a prototype so I've just ported many things from Python codebase. It was easier than twaeking existent ones. Especially it's related to tests - we had a really small amount of them. So now it's covered much better and also it uses many Python test cases so we aim here cross-language consistency in behaviour.
During this work I've tweaked API based on previosly raised issues and improved readme - https://github.com/frictionlessdata/datapackage-js/tree/specs-v1#datapackage-js (could be a good review start).
That the last PR for core libs to specs-v1 update.

@amercader
Thanks!


Kicad does not appear in the GNOME 3 application menu

Describe the bug
After installing Kicad, I am unable to locate the application in the gnome menu.
I can see that it's present on the system from bash

But the applications menu knows nothing about "kicad" (should be between I and M apps)

Nor any "k*" type name when searched for

To Reproduce
Steps to reproduce the behavior:

Add "kicad" to list of applications in home-manager
home-manager switch
alt+f2 > restart or logoff/on

Expected behavior
Kicad to have an application icon
Notify maintainers

@evils @kiwi @berce
Metadata
$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.72, NixOS, 20.09.1500.edb26126d98 (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.7`
 - channels(root): `"home-manager-20.09, nixos-20.09.1500.edb26126d98, unstable-21.03pre249162.1dc37370c48"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute: kicad
# a list of nixos modules affected by the problem
module:

as i don't use a desktop environment, i never got around to linking through the ${base}/share/applications/*.desktop files
this would probably be done by #88923 implicitly if i ever got around to finishing that...
i think if you can try installing that PR (maybe rebase onto current nixos-unstable to avoid rebuilding the base) it should work, but i'm not currently set up to try that

Sounds like you're right, I'll close this.


Add '#file' to /help

Fix microsoft/vscode-copilot-release#1092

@roblourens Is the branch name randomly generated? 🤔


Why the onChange is required if works without it?

Works, but whats the point using this package without the onChange?

I released https://pub.dev/packages/flutter_awesome_select with fixed null safety.
Could you check that everything works for you?


Remove implementation of and references to InternalParser

System.Text.Internal.InternalParser contains many prototype implementations of parsing methods. Most of these implementations have been replaced by the more standardized APIs in System.Text.PrimitiveParser, although many concepts from these prototypes have been brought over. This change removes the implementation and all references to InternalParser, as part of the effort to productize the parsing APIs ( #1070).

@KrzysztofCwalina @yizhang82 Please take a look.

thanks!


Webpack compilation warning in 0.29.0 with DOMPurify

Hi,
I just installed version 0.29.0 with<EMAIL_ADDRESS>and when I start my application with webpack-dev-server I start seeing the following warnings:
Compiled with problems:

WARNING in ../../node_modules/monaco-editor/esm/vs/base/browser/markdownRenderer.js 250:4-21

export 'addHook' (imported as 'dompurify') was not found in './dompurify/dompurify.js' (possible exports: default)

WARNING in ../../node_modules/monaco-editor/esm/vs/base/browser/markdownRenderer.js 269:4-21

export 'addHook' (imported as 'dompurify') was not found in './dompurify/dompurify.js' (possible exports: default)

WARNING in ../../node_modules/monaco-editor/esm/vs/base/browser/markdownRenderer.js 281:15-33

export 'sanitize' (imported as 'dompurify') was not found in './dompurify/dompurify.js' (possible exports: default)

WARNING in ../../node_modules/monaco-editor/esm/vs/base/browser/markdownRenderer.js 284:8-28

export 'removeHook' (imported as 'dompurify') was not found in './dompurify/dompurify.js' (possible exports: default)

WARNING in ../../node_modules/monaco-editor/esm/vs/base/browser/markdownRenderer.js 285:8-28

export 'removeHook' (imported as 'dompurify') was not found in './dompurify/dompurify.js' (possible exports: default)

I used version 0.28.0 and never bumped into this error so I reckon this must come from the latest release, this is a warning so it's not a blocker, but it's quite inconvenient for developers using webpack and webpack-dev-server. Do you have plan to fix this (I'm not so sure, since this seems like code of DOMPurify and not vscode) or suggest any workaround for us?
Thank you so much!

This ends up as a build error for me. I used create-react-app. Downgraded to<EMAIL_ADDRESS>+<EMAIL_ADDRESS>then a build succeeded.


(Converted from comment) Imagebam not working

Converted from: https://github.com/4pr0n/ripme/issues/502#issuecomment-306664381 by @Flip32

Hi, It seems Imagebam it's not working! And when does, the pics come smaller.
Can you help me?
ex.: http://www.imagebam.com/gallery/72c5utvqv44wcqnkc9vzapm5yck8j1cr

I can't repo this. Ripping http://www.imagebam.com/gallery/72c5utvqv44wcqnkc9vzapm5yck8j1cr seems to rip the entire album from all pages and all the images are full sized

Thanks for investigating. I'll try it also a bit later and if I can't repro I'll close this.

Seems to work for me (tried a quick rip urls-only). Verifying whether the images are full size or not will take a bit closer of a look.

I've tested this again and I am able to rip the whole album and all the images are downloaded in their full size. I'm closing this as can't repo


K8SPG-354 pgo 2.2.0 for dbaas production

Test name
Status

api-tests
failed

Test name
Status

api-tests
failed

Test name
Status

api-tests
failed

Test name
Status

api-tests
failed


Add mod Freight Forwarding

Context: https://mods.factorio.com/mod/FreightForwarding
This is a naive import via FactorioLab Exporter.
Known issues:

Airplanes do not show up as cargo wagons.
The exporter cannot deal with the looping nature of ferrous/cupric nodules and doesn't know where cobalt crust is coming from.
Titansteel smelting/heating misses its input ingredients (steel, titanium plate, cobalt ingot) as well.
Discharged batteries seem to be discarded on export, but that's a KI from the exporter and I don't think matters too much. (They are used as energy sources for trains and discharging stations, which doesn't matter to the calculator, and as raw inputs, which seems to work fine.)

Superseded by #1065, data set rebuilt using new data generator

Thanks!


Doesn't Convert the Text Node in the Input SVG

I was trying to convert the SVG To JSX and found one weird bug. This doesn't support the conversion of Text Node in SVG.
For E.g. below SVG Doesn't translate to JSX.
It just says [Object Object]
<svg xmlns="http://www.w3.org/2000/svg" width="18" height="18">   <g fill="none" fill-rule="evenodd" stroke="#333" stroke-linecap="round" stroke-linejoin="round" transform="translate(1 1)">     <circle cx="5.565217" cy="6.06087" r="25.565217"/>     <path d="M9 10.495652L15.504348 17"/>     <text id="Track-job-sheet-stat" font-family="sans-serif" font-size="18" font-weight="bold" fill="#000000">         <tspan x="44.5080652" y="59.286875">Track job sheet status</tspan>     </text>   </g> </svg>

+1

Fixed.


Look at orjson as replacement for all json calls

Export requires a number of JSON calls as does --exiftool.  Consider orjson as replacement.

See also msgspec


symbol_properties.minimum_price_variation incorrect for Australian Dollar Futures (6A) Futures

Expected Behavior

Future: Australian Dollar Futures
Exchange Ticker:  6A
Minimum Price Variation: 0.00005
References:
https://www.cmegroup.com/markets/fx/g10/australian-dollar.contractSpecs.html
Actual Behavior

Future: Australian Dollar Futures
Exchange Ticker:  6A
Minimum Price Variation: 0.0001
Minimum price variation is incorrect for Australian Dollar Futures (6A)
Potential Solution

Adjust the corresponding values in symbol properties database (line 226).
Reproducing the Problem

qb = QuantBook()
print(qb.AddFuture("6A").SymbolProperties.MinimumPriceVariation))
System Information

QC Cloud
Checklist

[x] I have completely filled out this template
[x] I have confirmed that this issue exists on the current master branch
[x] I have confirmed that this is not a duplicate issue by searching issues

[x] I have provided detailed steps to reproduce the issue

Hi @Martin-Molinero , I would like to work on this.


Include angular-material

Index Page

Whiteboard Page

(y)

This is merged to develop branch, thanks:)


Implemented an initial version of fishing

More details in commit messages.

This has been merged into the v7.0.0 branch.


[Help] Publish don't trigger any error

func sendRBMQMessage(workflow []byte) error {
	err := ch.Publish(
		rmbqExchange,
		rmbqRoutingKey,
		true,
		false,
		amqp.Publishing {
			DeliveryMode: amqp.Persistent,
			Timestamp:    time.Now(),
			ContentType:  "application/json",
			Body:         message,
		})
	return err
}

If queue does not exist, err is nil.
I assume that if any error appears during the publish, err will still be nil.
How can I specify an acknowledgement during the publish?
I read somewhere that I could use some kind of listeners but I don't really understand how it works.
Thanks

The documentation states that:

...
Since publishings are asynchronous, any undeliverable message will get returned by the server. Add a listener with Channel.NotifyReturn to handle any undeliverable message when calling publish with either the mandatory or immediate parameters as true.
...
This can return an error when the channel, connection or socket is closed. The error or lack of an error does not indicate whether the server has received this publishing.
...

So there are "listeners" like NotifyReturn and NotifyPublish which is channels you can read from, you can read more about them in the documentation as well.
Regarding how they work, you simply pass these functions a channel and whenever a notification occurs you can read from that channel.
Example
// ch is the same channel you use in your code to publish.
returnErrChan := ch.NotifyReturn(make(chan amqp.Return))

go func() {
    notification, ok := <- returnErrChan
    if !ok {
        // Channel was closed.
    }

    log.Printf("A message got returned: %s %s\n", notification.RoutingKey, notification.ReplyText)
}()

You might want to do that in a loop and take proper action based on the result and if the channel is closed etcetera.
Hope this helps!

Hi @bombsimon
Thanks for your answer and example.
It helps a lot.
About your example, does it mean the go func will be called each time a message is returned? Or only once?
Why do I need a loop here?

The func() is not some kind of callback, it's just an anonymous function spawned in a separate go routine. The way it works is with the channel you create. I'm not sure how familiar you are with go routines and channels so sorry if this is too much details.
The channel created will receive the notification each time any request published will be returned so it might be more than once. This means that if you wan't to keep tracking the result of your publishings for more than one request you need to read from the channel as long as your application is running.
Since amqp.Return which is sent on the channel contains the CorrelationId you could use that field to identify which of your requests didn't get delivered. That means that you need to keep track of what request was sent with what ID. The correlation id is set on the amqp.Publishing just like your body and content type.
A more proper way and why I suggest a loop would look like this:
// Spawn a go routine taking care of all requests returned
go func() {
    for {
        notification, ok := <- returnErrChan
        if !ok {
            // Channel closed, probably also the channel or connection.
            return
        }

        messageFailing, ok := myMessageMap[notification.CorrelationId]
        if !ok {
            fmt.Println("No message mapped to ID: ", notification.CorrelationId)
        }

        // Do what you need with messageFailing.
    }
}()

Thanks a lot man.
Indeed I was not very familiar with routines and channels (just started with go). I am more now. :-)
Working just fine. 😬
Thanks for your help, really appreciate! 👍


Adding command line capabilties

This is a simple wrapper to use dreamjs from the command line, it detects a for a schema.js in the current directory and provides an interface to generate data from those schemas

Thanks for the good work


Add file upload to flask app

Currently, files on the IATI registry can be tested. It would be useful to add a direct file upload as well (this was in an earlier version, but disappeared!)

Fixed some time ago!


feat(category_theory/limits): constructions

This extends the first PR on (co)limits. It

Shows that C ⥤ D has limits if D does.
Constructs equalizers and products from limits.
Constucts limits from equalizers and products.
Constructs pullback from equalizers and binary products.

We don't yet have anything contructing colimits from limits in the opposite category, so these constructions don't immediately give all the corresponding dual constructions.

173 simps...

Closing for now, as a redesign is in the works.


[Platforms] Managing service bindings and secrets

:exclamation:  If you're interested in supporting and driving this work please respond to this issue. Once we have enough interest we'll schedule an initial meeting to review current ideas and proposals.

Background
The Platforms group is going deep into capability domains for platforms to seek ways to reduce complexity and increase synergies and standardization across related projects, vendors and users.
One of the domains which many find hard to rationalize today is how to seamlessly bind workloads to the capabilities or services they rely on. That is, workloads must be able to authenticate and connect to databases, object stores, message brokers, observability systems and more and there are many different ways to accomplish this.
At a high level this binding is typically accomplished by 1) getting a credential and URL from the service; 2) securely storing that info in a secure store like Hashicorp Vault; and 3) injecting that info into workloads in specific formats and locations. However, it seems every service, every platform, every secure store and every workload type has a different way to accomplish these steps.
This leads to difficulty, confusion and delay for cloud developers and lots of discomfort and concern to cloud security practitioners about how secrets are managed. For example, many folks may not realize that some projects write secret data in Kubernetes secrets as it's transferred to the workload, a practice some enterprises don't allow.
We've jumpstarted discussion about projects and ideas to address this in this Google doc. I'll copy the two main sections here to continue discussion.
Please share other projects and ideas with us and if/how you can contribute. Thanks!
Related Projects and Services

External Secrets Operator (ESO) - synchronizes secrets from an external store into Kubernetes' secret store (i.e., etcd)
SOPS ("Secrets Ops") - secrets are encrypted and the encrypted strings are stored in plaintext. A master secret is maintained elsewhere and decrypts secrets before use.
Service binding runtime and spec - a relationship between a service and a workload is declared; on reconciliation credential is retrieved from service and injected into workload
Hashicorp Vault - static secrets are stored in an encrypted database; and/or dynamic secrets are generated on demand for specific service types. Secrets are retrieved by direct API call or injected into workloads by a sidecar.
Azure Passwordless - automatically generate secrets for services and inject into workloads (need verification as it's closed source)
Dapr secrets management - provides a standard API for retrieving secrets from any store, bypassing the Kubernetes store
SPIFFE/SPIRE - inject a workload identity
GCP Cloud SQL Auth Proxy
vault-config-operator
Tekton auth: https://github.com/tektoncd/pipeline/blob/main/docs/auth.md
Cloud providers: Azure Key Vault, AWS KMS, GCP Key Management
ArgoCD’s doc on secrets integration: https://argo-cd.readthedocs.io/en/stable/operator-manual/secret-management/

How Argo manages secrets in its rendering engine: https://argo-cd.readthedocs.io/en/stable/operator-manual/secret-management/#mitigating-risks-of-secret-injection-plugins

kube-bind - binding CRD based APIs from providers to consumer clusters, with the ability to exchange secrets and other auxiliary objects, e.g. services.
Vault Secrets Operator - syncs secrets from Hashicorp Vault into Kubernetes secrets

Ideas

Focus on dynamic generation of short-lived credentials and transparent injection of those credentials into workloads (pods).  Contrast with static secrets, i.e., simple key/value pairs that must be kept secret, like an API key from a service provider.
Explore conventions for requesting short-lived credentials and locators from services on demand
Could we have a standard API that given a CR* returns fresh credentials?

Compare Vault dynamic secrets engines
Compare knative SinkBinding and Binding duck type
Compare service binding "Provisioned Service"
How to avoid transiting a K8s secret? (some enterprises insist on this)

Explore conventions for injecting credentials and locators into workloads in specific locations, e.g. as files at a specific path

Hypothesis: this will enable library authors to seamlessly consume secrets, reducing load on app developers to template secrets in specific files at specific locations.
Establish a convention on where to put binding info in workloads, e.g. as files in a well-known folder. Could be used in Vault agent and SBO.

Establish a library of dynamic credential generators for different service types, a la Hashi Vault's dynamic secrets engines.

Use these in service binding runtime and other secret generators.

Encourage user community to treat all secrets as short-lived and ephemeral. Encourage folks to avoid static secrets persisted in a store. If we get a library going, ask service and cloud providers to contribute "generators" for their services.
Add SecretClass to Kubernetes secrets and enable alternate implementations, e.g. served from an external store rather than etcd. Compare to ClusterClass and StorageClass.

cc @monadic @raffaelespazzoli @sttts @sabre1041 @hiddeco @cmoulliard

What about bitnami sealed-secret?

I'm interested in potentially helping out with this. I'm a member of TAG Security and a maintainer of Conjur Open Source.

In trying to get alignment on current WG priorities, I am going to suggest we close this topic. But before doing so, I want to give 1 week for anyone who is interested to revive the focus on this, else we can reopen in the future when we have the right people/energies to commit to this topic.
Thanks!

Per my last message, I am closing for now but am happy to reopen this conversation if it becomes higher priority / has a sponsor to drive it!

Instead of closing it, it should be better to move the request/issue to a future milestone


Fix #10 - skip functions that pyd can't wrap

Codecov Report

Merging #46 into master will not change coverage.
The diff coverage is n/a.

@@         Coverage Diff          @@
##           master   #46   +/-   ##
====================================
  Coverage      20%   20%
====================================
  Files           6     6
  Lines          25    25
====================================
  Hits            5     5
  Misses         20    20

Impacted Files
Coverage Δ

python/source/autowrap/python/wrap.d
0% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 761e6be...72e94de. Read the comment docs.


Prevent overwriting of defaultDependency detekt-cli by additional detekt dependencies

When including an additional dependency in the detekt configuration it would overwrite the defaultdependency we define in DetektPlugin. This behavior is also described in the Gradle documentation: https://docs.gradle.org/current/javadoc/org/gradle/api/artifacts/Configuration.html#defaultDependencies-org.gradle.api.Action-

Execute the given action if the configuration has no defined dependencies when it first participates in dependency resolution.

In case of an extra detekt dependency the default dependency the DetektPlugin defines is omitted.
However we require detekt-cli to be able to run detekt at all. To prevent the overwriting we simply define detekt-cli as a regular dependency in the DetektPlugin.
The added test fails with the master version of DetektPlugin and passes with the changes.
fixes #1143

Instead of changing the detekt-cli dependency from being a default to forcing it to be applied, consider instead leaving it as a default, and creating a separate configuration for any detekt plugins that are to be added to the classpath.
This is the approach taken by the FindBugs plugin and should work well here.
There would then be two configurations in the Gradle plugin: detekt, and detektPlugins. The detekt configuration would remain the same (i.e. don't merge this PR). The new detektPlugins configuration would be combined with the detekt configuration's classpath in DetektInvoker.

@3flex That is something I have considered. However what would the benefit of the two configurations vs this one be?

It's the most flexible setup and follows conventions used in other Gradle code quality plugins. People can use the detekt configuration when overriding the default detekt-cli dependency (like we need for #1124, unless using dependency substitution which is a hassle) but still allows adding plugins using detektPlugins configuration. It's a nice separation of responsibilities.

That makes sense. I split the configurations into detekt and detektPlugins.

Actually I did it the other way around. detekt is still the "external" configuration. The plugin uses detektPlugins internally. That way we don't have another breaking change to the API.
We could also go with detekt and detektInternal to make it clearer.

detektInternal sounds better if it's meant to be only for internal usage!

@3flex As long as we have one "default" configuration and one "public" configuration similarly to what FindBugs does I think we should be fine. The names of which we can still discuss/adjust. I just kept detekt as the public configuration as for the breaking API change.
I see your point about the current API not working correctly. At this point I don't have a strong opinion on the naming of the configurations.

Mhh, I do not quite understand if this is rdy?
I've also noticed that if I use detekt-formatting I need also to add detekt-cli to the detekt configuration. Does this fix this issue too?

Just read the issue. Will merge this now, thanks!

@Mauin you misunderstood what the FindBugs plugin configurations do, sorry if I didn't explain properly. FindBugs doesn't have a "default" and a "public" configuration. Both "findbugs" and "findbugsPlugins" are part of the public configuration DSL. Neither is private or internal.
From the Gradle docs, the FindBugs plugin:

Declares a findbugs configuration which needs to be configured with the FindBugs library to be used. Additional plugins can be added to the findbugsPlugins configuration.

For detekt, the detekt configuration would be used to configure which detekt library to use (detekt-cli). Additional plugins (i.e. detekt-formatting, others) would be added with detektPlugins.
Isn't it better to do it according to Gradle conventions while version is still RC instead of publishing 1.0.0 with a DSL that isn't aligned with other Gradle plugins?


Basic support for $currentDate operation

Timestamp is not supported so far, hence throwing NotImplementedError when typeSpecification is exactly {'$type': 'timestamp'}.
Not sure if typeSpecification should be restricted to True or {'$type': 'date'} in mongomock. Since I've noticed some weird behavior in mongo, e.g. it allows $currentDate: {foo: False} and treats it exatly the same as  $currentDate: {foo: True} while prohibiting $currentDate: {foo: 1}

There are some merge conflicts in the test__collection_api.py file, after I merged your last PR. Could you please rebase this PR to the develop HEAD?

Build failed with gpgkeys: key 7F0CEB10 not found on keyserver


Preprocessing

Preprocessing for the images.

now config has two parameters: PRE_PROCESS =True/False and CROPPING
if PRE_PROCESS=True then openCV needs to be installed, otherwise app crashes
should work without OpenCV if PRE_PROCESS = False

Requires to create new Car-folder:
donkey createcar --path ~/yourcar

changes all over the place. TODO test properly.

also added the markku.ai image augmentation scripts


Video can't Play

Only thumbnail displayed Video can't plays

Having same issue. I have checked its log. it says, trying to load empty source. I could not find way out. Any suggestion?

anyone have?
seek = (time, tolerance = 100) => {
|     if (isNaN(time)) throw new Error('Specified time is not a number');


Please add all possible errors and possible solutions

A verified publisher cannot be added to this application. Please contact your administrator for assistance. [VFSMWUDu8KWgEx8CQ8Djg7]
I am admin of this Azure instance and same a/c is having MPN ID
Why am I still getting this error?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f83478f0-0faa-2e10-37f2-819ad2351445
Version Independent ID: 3ffd5b35-a3d7-5dca-dc8b-acec1d85ec3c
Content: Publisher verification overview - Microsoft identity platform
Content Source: articles/active-directory/develop/publisher-verification-overview.md
Service: active-directory
Sub-service: develop
GitHub Login: @rwike77
Microsoft Alias: ryanwi

The MPN ID you provided (XXXXXX) does not exist, or you do not have access to it. Please provide a valid MPN ID and try again. [VFSMWUDu8KWgEx8CQ8Djg7]
A verified publisher cannot be added to this application. Please contact your administrator for assistance. [VtDMuVrKrDh71B4CpEGuzK]
You must use multi-factor authentication to proceed. Please ensure MFA is enforced for your account, and then sign in again using MFA. [VjpDekCBS9tULdQpOdAQQe]

I called up support, and couple of times we tried. Out of the blue MPN ID got accepted. Please update detailed steps of what is required when such errors come up.
You must use multi-factor authentication to proceed. Please ensure MFA is enforced for your account, and then sign in again using MFA. [VjpDekCBS9tULdQpOdAQQe]
--- My Azure AD tenent was different one
The MPN ID you provided (XXXXXX) does not exist, or you do not have access to it. Please provide a valid MPN ID and try again. [VFSMWUDu8KWgEx8CQ8Djg7]
--- I was using Location MPN and not Global MPN
A verified publisher cannot be added to this application. Please contact your administrator for assistance. [VtDMuVrKrDh71B4CpEGuzK]
--- This error came many a times and all of a sudden it got accepted. So keep trying it couple of times

A verified publisher cannot be added to this application. Please contact your administrator for assistance. [VtDMuVrKrDh71B4CpEGuzK]
This one is really frustrating-- no idea how many times we have to try to get right :(

Thanks for bringing this to our attention. Your feedback has been shared with the content owner for further review.
@rwike77  - FYI


Set custom serializer for includes file

HI,
I've been wondering is there any chance that i can use above class at includes?
For example:

class OrderTransformer extends TransformerAbstract
{
    public $availableIncludes = ['user', 'items'];

    /**
     * A Fractal transformer.
     *
     * @param $resource
     * @return array
     */
    public function transform($resource)
    {
        return [
            'id'         => $resource->id,
            'receiver'   => $resource->receiver,
            'totalPrice' => $resource->totalPrice,
            'address'    => $resource->address,
            'phone'      => $resource->phone,
            'orderedAt'  => $resource->createdAt->format('Y-m-d H:i:s'),
        ];
    }

    public function includeUser($resource)
    {
        $resource = $resource->user;

        return $this->item($resource, new UserTransformer);
    }

    public function includeItems($resource)
    {
        $resource = $resource->items;

        return $this->collection($resource, new OrderItemTransformer); // I want to use custom serializer here to remove "data" key.
    }
}

I need to use those fractal to remove the additional data key on the result.
Here is the result I get:
{
    "orders": [
        {
            "id": "C170620PVJM6",
            "receiver": "Example",
            "totalPrice": 10,
            "address": "Anywhere near the horizon",
            "phone": "081234567",
            "orderedAt": "2017-06-20 22:42:43",
            "items": {
                "data": [
                    {
                        "price": 2,
                        "menu": {
                            "name": "VERY tired of.",
                            "image": {
                                "web": "https://lorempixel.com/413/413/?87962",
                                "mobile": "https://lorempixel.com/413/413/?87962",
                                "square": "https://lorempixel.com/413/413/?87962"
                            }
                        }
                    },
                    {
                        "price": 6,
                        "menu": {
                            "name": "Queen jumped up in.",
                            "image": {
                                "web": "https://lorempixel.com/413/413/?77046",
                                "mobile": "https://lorempixel.com/413/413/?77046",
                                "square": "https://lorempixel.com/413/413/?77046"
                            }
                        }
                    },
                    {
                        "price": 2,
                        "menu": {
                            "name": "And oh, I wish you.",
                            "image": {
                                "web": "https://lorempixel.com/413/413/?22822",
                                "mobile": "https://lorempixel.com/413/413/?22822",
                                "square": "https://lorempixel.com/413/413/?22822"
                            }
                        }
                    }
                ]
            }
        }
    ],
    "meta": {
        "pagination": {
            "total": 5,
            "count": 1,
            "per_page": 1,
            "current_page": 1,
            "total_pages": 5,
            "links": {
                "next": "https://api.butterfish.dev/v1/user/orders?page=2"
            }
        }
    }
}

Expected result:
{
    "orders": [
        {
            "id": "C170620PVJM6",
            "receiver": "Example",
            "totalPrice": 10,
            "address": "Anywhere near the horizon",
            "phone": "0123123123",
            "orderedAt": "2017-06-20 22:42:43",
            "items": [
                    {
                        "price": 2,
                        "menu": {
                            "name": "VERY tired of.",
                            "image": {
                                "web": "https://lorempixel.com/413/413/?87962",
                                "mobile": "https://lorempixel.com/413/413/?87962",
                                "square": "https://lorempixel.com/413/413/?87962"
                            }
                        }
                    },
                    {
                        "price": 6,
                        "menu": {
                            "name": "Queen jumped up in.",
                            "image": {
                                "web": "https://lorempixel.com/413/413/?77046",
                                "mobile": "https://lorempixel.com/413/413/?77046",
                                "square": "https://lorempixel.com/413/413/?77046"
                            }
                        }
                    },
                    {
                        "price": 2,
                        "menu": {
                            "name": "And oh, I wish you.",
                            "image": {
                                "web": "https://lorempixel.com/413/413/?22822",
                                "mobile": "https://lorempixel.com/413/413/?22822",
                                "square": "https://lorempixel.com/413/413/?22822"
                            }
                        }
                    }
                ]
        }
    ],
    "meta": {
        "pagination": {
            "total": 5,
            "count": 1,
            "per_page": 1,
            "current_page": 1,
            "total_pages": 5,
            "links": {
                "next": "https://api.butterfish.dev/v1/user/orders?page=2"
            }
        }
    }
}

Can I include custom serializer on that includes?

Faced the same issue. It would be great to get any feedback from package maintainers.

@DeRain @stefensuhat
i think that's why:

Collections keep the 'data' namespace to avoid confusing JSON when meta data is added.

See here: http://fractal.thephpleague.com/serializers/

See https://github.com/thephpleague/fractal/issues/315#issuecomment-248789771 for a work around

@JayBizzle is there any reason why this workaround would no longer be working? Our app was working great for the longest time, then we updated to Laravel 5.5 and all of a sudden I can't get that data key to go away. I've tried setting the default_serializer in the config file and using the manager to set a custom serializer.

You must change the default serializer in config/fractal.php
default_serializer' => 'League\Fractal\Serializer\ArraySerializer',


Miningcore -c config.json  Startup failure

Hello, I have been struggling to start MiningCore for several days. How can I get in touch with you for assistance? I am willing to compensate for your help. Thank you!

docker also reports errors

Hello,
Sorry to hear about your troubles but i am not familiar with Docker, i actually never used it for running miningcore. I'm afraid i can't really offer much assistance in these circumstances.

@panbo248 After looking at your screenshot, i think the issue is libzmq5n, edit the file and change it to libzmq5, then try again.

https://github.com/blackmennewstyle/miningcore/commit/df5c3c51059b095aba8b77fe6677fb49aaa4571b updated Dockerfile. Give it a try.


Mount method for sdkv2 is not working

Package Name = sdk-v2:
Python Version = 3.10:

Describe the bug
Mount method is not working as it was working 2 months ago. I an using it in my compute instance in my Azure ML ws
To Reproduce
Steps to reproduce the behaviour:

I have created the following function to mount data from the datalake:

def mount_data_asset(
    mlclient: MLClient,
    container_name: Literal["raw", "curated"],
    path_to_data: str,
    folder_name: str,
) -> str:
    """
    NOTE: Make sure to pip install azure-ai-ml[mount] to use the mount method.

    Mounts a data asset from the specified container and path to the specified folder name.

    Args:
    mlclient (MLClient)                       : The MLClient object.
    container_name (Literal["raw", "curated"]): The container name where the data asset is stored.
    path_to_data (str)                        : The path to the data asset.
    folder_name (str)                         : The name of the folder where the data asset will be mounted.

    Returns:
    str: The local mount path where the data asset is mounted.

    Usage:
    1. Datafiles (csv and parquet files)
    lcl_path = mount_data_asset(
    mlclient=ml_client_dev,
    container_name='raw',
    path_to_data='customers/music/music_tracks/',
    folder_name='datafiles')

    2. Deltatables
    lcl_path = mount_data_asset(
    mlclient=ml_client_dev,
    container_name='curated',
    path_to_data='customers/data_more/music/music_tracks/',
    folder_name='deltadata')

    Official documentation:
    https://learn.microsoft.com/en-us/python/api/azure-ai-ml/azure.ai.ml.operations.datastoreoperations?view=azure-python
    """
    path =<EMAIL_ADDRESS>    local_mount_path = f"./mounted_data/{folder_name}"

    try:
        mlclient.data.mount(
            path=path,
            mount_point=local_mount_path,
            persistent=True,
            mode="ro_mount",
            debug=True,
        )
        print(f"Data asset mounted at {local_mount_path}.")
    except Exception as e:
        print(f"The following error occurred: {e}")

    return local_mount_path

And in my compute instance I am running:

# 1. Datafiles (csv and parquet files)
# Mount the data folder (csv or parquet files) from the Datalake
credential = DefaultAzureCredential()
ml_client_dev = MLClient.from_config(credential=credential)
lcl_path = mount_data_asset(
    mlclient =ml_client_dev,
    container_name='raw',
    path_to_data='customers/music/music_tracks/',
    folder_name='datafiles')

Expected behavior
I was authenticating with the default credentials (it was working perfectly) and I expected to return a string the (local path the data) and the data to be mounted in the directory local_mount_path.
Additional context
The function was working perfectly 2 months ago. It was working like a charm.
Now we want to use this function into production, and apparently is not working.
Errors
The function runs fine but nothing is mounted.
The output is the following:
Method mount: This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Mount requested [name: unified_mount_356a63fba30c4f099899f83fdb6e8281]. Waiting for completion ...
Mounted [name: unified_mount_356a63fba30c4f099899f83fdb6e8281].
Data asset mounted at ./mounted_data/datafiles.
Also if I try to rerun the function I get the error:
Method mount: This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
The following error occurred: Operation returned an invalid status 'Only creator or assigned user can execute this action.'
Content: {
  "error": {
    "code": "UserError",
    "severity": null,
    "message": "Only creator or assigned user can execute this action.",
    "messageFormat": null,
    "messageParameters": null,
    "referenceCode": null,
    "detailsUri": null,
    "target": null,
    "details": [],
    "innerError": null,
    "debugInfo": null,
    "additionalInfo": null
  },
  "correlation": {
    "operation": "c09e468ffaa396245bcac860cd079323",
    "request": "144f23f2fb050056"
  },
  "environment": "westeurope",
  "location": "westeurope",
  "time": "2024-10-29T10:33:15.0397135+00:00",
  "componentName": "machinelearningcompute",
  "statusCode": 400
}

Can you please have a look at this issue?

Thanks for the feedback, we’ll investigate asap.

@ChenxiJiang333 Please help on this issue.

got it

Hi @achauhan-scc, could you help take a look at this issue?

Any updates?

Hi @pdhotems, could you help take a look at this issue?

Any luck with the mount method?


Implement a PDF text extraction solution with Python

I started with pycld2 as it seems to be the fastest (Polyglot is built on it) and simplest to start with. However the module was not updated since 2 years and installation was a bit tricky (I documented in the readme.md).
There is also pycld3 library but installation looks much more complex. The size is bigger and it's slower.
I didn't look into other options.

es, i would stay in Python Eco-system for language detection as well. The pycld2 indeed seems to be outdated. PyCLD3 would be a better choice for future.
During my quick research, i bumped into these libraries / frameworks. In particular, Spacy and TextBlob seem to be promising...

TextBlob
Spacy
FastText
Polyglot
Langdetect
Langid
PyCLD3
Guesslang

But at this stage, you can just one library of your preference and implement a quick prototype.. We can improve or select a different at later stages of the project, when we gain more insights on what we really need.

Hey guys, great progress! Your choices are understandable. Thanks for the
documentation!
Seems like we're on a good path regarding text extraction.
What I wanted to add: is it possible to implement a REST API service so we
could call on the microservice from the nocode test platform?
And: extracted text definitely needs to be cleaned up. I think I could be
done with spacy, among many others. I suppose it might depend in some cases
on the NLP model we are using on how big an impact the pre-prpcessing of
the data might have, but I suspect stopword removal, punctuation removal,
stemming and lemmiatization should always be performed. Inspired by this:
https://stackoverflow.com/questions/45605946/how-to-do-text-pre-processing-using-spacy
At any rate, I think this issue should be looked into before we deal with
the question of how to process tables in a PDF.
Talk soon!
catilgan @.***> schrieb am Mo., 4. Okt. 2021, 21:51:

es, i would stay in Python Eco-system for language detection as well. The
pycld2 indeed seems to be outdated. PyCLD3 would be a better choice for
future.
During my quick research, i bumped into these libraries / frameworks. In
particular, Spacy and TextBlob seem to be promising...

TextBlob
Spacy
FastText
Polyglot
Langdetect
Langid
PyCLD3
Guesslang

But at this stage, you can just one library of your preference and
implement a quick prototype.. We can improve or select a different at later
stages of the project, when we gain more insights on what we really need.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/bizres/core-team/issues/25#issuecomment-933803509,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACO4TIS7M5QG6CPADKY2LKDUFIASTANCNFSM5DL7Z2SQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

and jsut to clarify: language detection will probably be necessary, but even more fundamental is text cleanup. it seems to me therefore this should the the issue to tackle next.

Hi @dev-ng
Are there any updates concerning this issue. I have deployed a version of text extraction tool into Hidora cloud, which can be used.
The extracted texts and the uploaded pdfs are stored in NFS data storage.

https://extractor.bizres.ch/docs/index.html

Is there any updates from you and @JohannesHool ?
Cheers, Cahit

Service "extract" is done. See PR 9.


21 filtering capability

Filtering Capability
Turned the informational badges into clickable filters

New Expandable Card Component
Used Alistair's project component to design these expandable cards for stardrops and museum items (thanks @alii)

yeah true