📋 November 2024 (v0.7) release checklist

Complete the following tasks to publish a monthly release.

✅ Monthly updates

[x] Update open data files (learn more).

[x] Pricing units

Run the extraction query.
Replace the contents of the corresponding open data files.
Compare changes and revert any invalid updates.
Re-gen and test PowerShell function: <root>/src/scripts/Build-OpenData.ps1 -PowerShell -Test
Review changes and revert unrelated files or unintended changes.
Document the added/updated rows in the changelog.
Publish a PR for updates

[x] Regions

Run the extraction query.
Replace the contents of the corresponding open data files.
Compare changes and revert any invalid updates.
Re-gen and test PowerShell function: <root>/src/scripts/Build-OpenData.ps1 -PowerShell -Test
Review changes and revert unrelated files or unintended changes.
Document the added/updated rows in the changelog.
Publish a PR for updates

[x] Services

Run the extraction query.
Replace the contents of the corresponding open data files.
Compare changes and revert any invalid updates.
Re-gen and test PowerShell function: <root>/src/scripts/Build-OpenData.ps1 -PowerShell -Test
Review changes and revert unrelated files or unintended changes.
Document the added/updated rows in the changelog.
Publish a PR for updates

[x] Update resource types and re-gen internal PowerShell functions: <root>/src/scripts/Build-OpenData.ps1 -PowerShell -Test
[x] Consider automating this process in Build-OpenData.

[x] Update Bicep CLI (docs): az bicep upgrade

🔄 Core features

[x] Review issues and PRs assigned to the target milestone.

[x] Create a new milestone for the next monthly release.
[x] Move issues that cannot be resolved in time to a new milestone.
[x] Submit PRs for issues that can be resolved.
[x] Complete open PRs that are ready to be resolved.

[x] Confirm there are no pending changes in dev.cd "<root>"
git checkout dev
git status

[ ] Update feature branches and confirm if they're ready for release.

[x] Auto-merge feature branch with dev: <root>/src/scripts/Merge-DevBranch.ps1 *
[ ] Manually update remaining feature branches: <root>/src/scripts/Merge-DevBranch.ps1 features/<name>

[ ] features/TODO

[x] Build and deploy templates: <root>/src/scripts/Build-Toolkit.ps1.
[x] Confirm all tests pass: <root>/src/scripts/Test-PowerShell.ps1 -Unit -Integration -Build.
[x] Confirm if all features are code complete and not missing any functionality required for release.

Once in dev, the feature is considered part of the next release and can be pushed out at any time. Any broken features will be reverted.

[x] Confirm new or updated functionality is documented in the changelog.

See Changelog guidance for details about changelog requirements.

[x] Confirm new or updated functionality must be documented in the documentation.
[ ] If adding a new tool, update the list of available tools on the documentation home page.

[ ] Merge any feature branches that are ready to dev.

[ ] Create a PR to merge the feature branch into dev.
[ ] Follow the normal PR process to merge the PR.

🔜 Finalize release

[ ] Review the changelog to ensure it encapsulates all changes.

Move all released changes to an official numbered version section.
If there are committed changes in a feature branch that you want to mention, add them to an "Unreleased" section.

[ ] Update the version: <root>/src/scripts/Update-Version [-Major|Minor|Patch]

[ ] Build all toolkit templates and resolve any issues: <root>/src/scripts/Build-Toolkit

This step is optional, but can catch issues earlier. You can also add the -Build parameter to the publish command in the next step.

[ ] Ensure all tests pass: <root>/src/scripts/Test-PowerShell -Unit -Integration

[ ] Package all release files

[ ] Run Package-Toolkit -Build -PowerBI script.

[ ] For the Cost summary report:

[ ] Save the file as a PBIX file to the release folder.
[ ] Change the sensitivity to Public. If the option is disabled, close the file and reopen it.

⚠ Power BI does not remember the sensitivity setting for Power BI projects so this needs to be done for each release. If not done, the report will not open for anyone outside of Microsoft.

[ ] Remove all but the following queries from Transform data: CostDetails, Prices.
[ ] Remove the Populate missing prices parameter and its reference from the CostDetails query.
[ ] Remove the Advisor for disks and Compliance calculation calculated tables.
[ ] Verify each page has data.
[ ] Select the Get started page and save the PBIX again in the release folder.

⚠ DO NOT save the above changes back to the Power BI project files!

[ ] Copy the first paragraph from the Get started page and export a template (PBIT file) in the release folder. Use the copied text for the description and add "Learn more at https://aka.ms/ftk/pbi/{report-name}" as a separate paragraph in the description.

[ ] For the Data ingestion report:

[ ] Save the file as a PBIX file to the release folder.
[ ] Change the sensitivity to Public. If the option is disabled, close the file and reopen it.

⚠ Power BI does not remember the sensitivity setting for Power BI projects so this needs to be done for each release. If not done, the report will not open for anyone outside of Microsoft.

[ ] Remove all but the following queries from Transform data: CostDetails, Hubs*, Storage*.
[ ] Remove the Populate missing prices parameter and its reference from the CostDetails query.
[ ] Remove the Advisor for disks and Compliance calculation calculated tables.
[ ] Verify each page has data.
[ ] Select the Get started page and save the PBIX again in the release folder.

⚠ DO NOT save the above changes back to the Power BI project files!

[ ] Copy the first paragraph from the Get started page and export a template (PBIT file) in the release folder. Use the copied text for the description and add "Learn more at https://aka.ms/ftk/pbi/{report-name}" as a separate paragraph in the description.

[ ] For the Governance report:

[ ] Save the file as a PBIX file to the release folder.
[ ] Change the sensitivity to Public. If the option is disabled, close the file and reopen it.

⚠ Power BI does not remember the sensitivity setting for Power BI projects so this needs to be done for each release. If not done, the report will not open for anyone outside of Microsoft.

[ ] Remove all but the following queries from Transform data: CostDetails and everything under Resource Graph.
[ ] Remove the Populate missing prices parameter and its reference from the CostDetails query.
[ ] Verify each page has data.
[ ] Select the Get started page and save the PBIX again in the release folder.

⚠ DO NOT save the above changes back to the Power BI project files!

[ ] Copy the first paragraph from the Get started page and export a template (PBIT file) in the release folder. Use the copied text for the description and add "Learn more at https://aka.ms/ftk/pbi/{report-name}" as a separate paragraph in the description.

[ ] For the Rate optimization report:

[ ] Save the file as a PBIX file to the release folder.
[ ] Change the sensitivity to Public. If the option is disabled, close the file and reopen it.

⚠ Power BI does not remember the sensitivity setting for Power BI projects so this needs to be done for each release. If not done, the report will not open for anyone outside of Microsoft.

[ ] Remove all but the following queries from Transform data: CostDetails, InstanceSizeFlexibility, Prices, Reservation* and all Resource Graph queries.
[ ] Remove the Populate missing prices parameter and its reference from the CostDetails query.
[ ] Remove the Advisor for disks and Compliance calculation calculated tables.
[ ] Verify each page has data.
[ ] Select the Get started page and save the PBIX again in the release folder.

⚠ DO NOT save the above changes back to the Power BI project files!

[ ] Copy the first paragraph from the Get started page and export a template (PBIT file) in the release folder. Use the copied text for the description and add "Learn more at https://aka.ms/ftk/pbi/{report-name}" as a separate paragraph in the description.

[ ] For the Workload optimization report:

[ ] Save the file as a PBIX file to the release folder.
[ ] Change the sensitivity to Public. If the option is disabled, close the file and reopen it.

⚠ Power BI does not remember the sensitivity setting for Power BI projects so this needs to be done for each release. If not done, the report will not open for anyone outside of Microsoft.

[ ] Remove all but the following queries from Transform data: CostDetails, Prices, AdvisorRecommendations, Disks, NetworkInterfaces, PublicIPAddresses, Resources, Subscriptions, VirtualMachines.
[ ] Remove the Populate missing prices parameter and its reference from the CostDetails query.
[ ] Remove the Advisor for disks and Compliance calculation calculated tables.
[ ] Verify each page has data.
[ ] Select the Get started page and save the PBIX again in the release folder.

⚠ DO NOT save the above changes back to the Power BI project files!

[ ] Copy the first paragraph from the Get started page and export a template (PBIT file) in the release folder. Use the copied text for the description and add "Learn more at https://aka.ms/ftk/pbi/{report-name}" as a separate paragraph in the description.

[ ] Check the docs for broken links:

[ ] Create a personal fork of the main repo.
[ ] If you already have one, update it to the latest.
[ ] Enable GitHub pages in your fork to use the dev branch docs folder.
[ ] Verify the pages build and deployment action completes successfully.
[ ] Open a dead link checker (e.g., deadlinkchecker.com) and check https://{your-username}.github.io/finops-toolkit.

Ignore link/href errors for new pages that have not been released in the official main branch.
Ignore any Download errors for the new release or any new files that haven't been released in the official main branch.
Ignore the Full changelog error for the new release, which hasn't been released in the official main branch.
Ignore the 403 error for learn.finops.org.
Ignore any 429 errors from GitHub. These are caused due to all the contributor links that are checked.

[ ] Fix any broken links, push changes, and rerun the tool.

🔜 Publish release

[ ] Publish the PowerShell module by running the Publish PowerShell action.
[ ] Submit PRs to publish bicep modules in the Bicep Registry.

[ ] Update your fork and local clone of the Bicep Registry.

Make sure you're in the main branch of the target repo when publishing a new template. If in another folder, the script will assume you're updating that branch and not create a new one.

[ ] Publish the scheduled actions bicep modules:

Start PR: <root>/src/scripts/Publish-Toolkit.ps1 "scheduledactions" -Commit
Complete PR template requirements and set the name to FinOps toolkit v#.# – scheduledactions.
Review the code that's changed in the PR, verify that the changelog covers everything, and update as needed.
If you make changes, re-publish without committing: <root>/src/scripts/Publish-Toolkit.ps1 "scheduledactions" -Build
Switch to the target repo folder and verify your changes were applied correctly.
Commit and push your changes.
Return to the PR URL and publish the PR.

[ ] Submit PRs to publish templates to the Azure Quickstart Templates repo.

[ ] Update your fork and local clone of the Azure Quickstart Templates.

Make sure you're in the master branch of the target repo when publishing a new template. If in another folder, the script will assume you're updating that branch and not create a new one.

[ ] Publish a FinOps hub PR:

Start PR: <root>/src/scripts/Publish-Toolkit.ps1 "finops-hub" -Commit
Complete PR template requirements and set the name to FinOps toolkit v#.# – FinOps hub.
Review the code that's changed in the PR, verify that the changelog covers everything, and update as needed.
If you make changes, re-publish without committing: <root>/src/scripts/Publish-Toolkit.ps1 "finops-hub" -Build
Switch to the target repo folder and verify your changes were applied correctly.
Commit and push your changes.
Return to the PR URL and publish the PR.

[ ] Publish a governance workbook PR:

Start PR: <root>/src/scripts/Publish-Toolkit.ps1 "governance-workbook" -Commit
Complete PR template requirements and set the name to FinOps toolkit v#.# – Governance workbook.
Review the code that's changed in the PR, verify that the changelog covers everything, and update as needed.
If you make changes, re-publish without committing: <root>/src/scripts/Publish-Toolkit.ps1 "governance-workbook" -Build
Switch to the target repo folder and verify your changes were applied correctly.
Commit and push your changes.
Return to the PR URL and publish the PR.

[ ] Publish a optimization workbook PR:

Start PR: <root>/src/scripts/Publish-Toolkit.ps1 "optimization-workbook" -Commit
Complete PR template requirements and set the name to FinOps toolkit v#.# – Optimization workbook.
Review the code that's changed in the PR, verify that the changelog covers everything, and update as needed.
If you make changes, re-publish without committing: <root>/src/scripts/Publish-Toolkit.ps1 "optimization-workbook" -Build
Switch to the target repo folder and verify your changes were applied correctly.
Commit and push your changes.
Return to the PR URL and publish the PR.

[ ] Check back after 1 hour to see if there any failed checks across all PRs (AQT).

If there are errors, click the Details link to understand what needs to be fixed.
Fix any issues in the FinOps toolkit codebase (not in the target repo).
Repeat publishing steps until all issues have been resolved.

[ ] Check back after 2-3 days to see if all your PRs were merged (AQT).

Bicep Registry and Azure Quickstart Template repos are manually reviewed and closed 2-3x per week. If you don't see an update, contact the ARM team.

[ ] Merge to main: <root>/src/scripts/Merge-DevBranch.ps1 main
[ ] Update the milestone.

Review all issues in the milestone, move anything that needs to be pushed, and close any completed items.
Close the milestone when all issues have been closed or moved.

[ ] Verify documentation updated correctly.

The documentation site may take 5 minutes to update after the merge is committed. If not updated, look at GitHub actions to see if there are any failures.

[ ] Tag and publish a new release:

[ ] Create a tag on publish using the "vX.X" format.
[ ] Set the Target to main.
[ ] Set the Previous tag to the previous release tag.
[ ] Set the name to Mmm yyyy (v#.#).
[ ] Copy the body from the previous release to use as a template.
[ ] Change the "New in" header to use the new version number.
[ ] Summarize changes from the changelog in the New in and Updated sections.

Simplify to only include one line per tool.
Each tool should be linked to its doc page.
Don't link features to their respective pages (e.g., PowerShell commands).
Don't list every small change. Use "various bug fixes and improvements" to keep it simple.

[ ] Update the list of direct and indirect contributors.

Use the "Generate release notes" feature to get a list of all code contributors.
Carefully review the list to ensure everyone is covered since feature branch PRs get merged, which can hide contributors.
If they made a code change, add them to the contributor list.
If they filed an issue, reviewed a PR, or participated in on- or offline discussions, add them to the list of supporters.

[ ] Update the discussion and changelog links in the footer. Comment out the AQT link if not ready.
[ ] Upload all files from the release folder:

ZIP files for templates like hubs and workbooks.
Power BI PBIX and PBIT files.
Open data CSV and JSON files.
ZIP file for sample data files.
DO NOT copy Bicep, PowerShell, PBIP, or image files.

[ ] Update the related discussion with the same text as the release.
[ ] Update all issues to Status: Released.

🔜 Post-release clean-up

[ ] Update the Toolkit / Should return all known releases PowerShell integration test based on the latest version.

See src/powershell/Tests/Integration/Toolkit.Tests.ps1 > Get-FinOpsToolkitVersion > Should return all known releases

[ ] Add the latest public version to the $expected variable.
[ ] Update the file checks to include/exclude any new/removed files.

[ ] Update the minor version and add the "dev" label: <root>/src/scripts/Update-Version.ps1 -Minor -Label dev
[ ] Update remaining branches that were not merged with dev

[ ] TODO: Add branches
[ ] features/adx
[ ] features/alerts
[ ] features/hack24
[ ] features/mslearn
[ ] features/private
[ ] features/ux

[ ] Confirm the following branches are needed or should be deleted:

[ ] features/governance
[ ] features/hourly
[ ] features/levelup
[ ] features/powershell
[ ] features/privateendpoint
[ ] features/savings
[ ] features/services

Branches that did not sync:

[ ] features/adx (local changes)
[ ] features/aoe (conflicts)
[ ] features/exports (conflicts)
[ ] features/hack24 (conflicts)
[ ] features/hourly (conflicts)
[ ] features/mslearn (conflicts)
[ ] features/private (conflicts)
[ ] features/savings (local changes)
[ ] features/ux (local changes)
[ ] features/workbookv2 (conflicts)

Documentation that has not been covered:

FinOps hubs:

[ ] Remote hubs
[ ] Private endpoints

Power BI:

[ ] Setup instructions (first-run experience for PBIT)


Add: AnanteshG Profile

Description
fixes #296
Checklist

[x] I have read and followed the Contribution Guidelines.
[x] All new and existing tests passed.
[x] I have updated the documentation to reflect the changes I've made.
[x] My code follows the code style of this project.
[x] The title of my pull request is a short description of the requested changes.

Hi @AnanteshG   thanks for your interest in contributing to DevDisplay!
We noticed you adding your profile on #DevDisplay but forgot to star the repo. It's like making an amazing sandwich and forgetting to take a bite! -- I think our repo's feeling a little lonely without your 🤗
So, be sure to star the DevDisplay GitHub Repository. 🌟
💌 This is a mandatory step for all contributors. & It's is the first priority of contribution in DevDisplay 💌
🚀 We look forward to your contribution once these steps are completed. 🚀
Thanks 👍🏻


[113] - Handle batches with no gt, Add images with no gt to the dataset

This PR addresses the following issue:
https://github.com/WongKinYiu/YOLO/issues/113

Hi,
Would you be willing to push to the MODEL branch?
DATASET branch on GitHub
Best regards,
Henry Tsui

Hey,
sure, opened a new pull request for the MODEL branch.
https://github.com/WongKinYiu/YOLO/pull/120
Closing this one as duplicate then.
Best regards,
Dmitrii


Work around/suppress around 95 warnings produced on JDK-17 with -Xlint:all

Work around/suppress around 95 warnings produced on JDK-17 with -Xlint:all flag enabled.
Fix a couple of warnings dotted around in tests to do with unchecked operations
Fix incorrect javadoc reference in Floats_AssertLessThanOrEqualTo_test

Some suppressions are being marked as being redundant by IntelliJ when they are not. For these, I have added an IntelliJ-specific suppression about redundant suppressions to prevent a risk of the original suppression being accidentally removed in the future due to the false positive.
Xlint:all was run by adding
<compilerArgs>
  <arg>-Xlint:all</arg>
</compilerArgs>

...to the <configuration /> block for maven-compiler-plugin in the POM.

Integrated thanks @ascopes!


net/http: panic: runtime error: index out of range (when using a faulty proxy server)

What version of Go are you using (go version)?
go1.9, problem exists in the master branch
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
not related
What did you do?
use a faulty proxy server
What did you expect to see?
no panic, error is returned
What did you see instead?
a panic
The problem is in this line: https://github.com/golang/go/blob/master/src/net/http/transport.go#L1133
	if resp.StatusCode != 200 {
		f := strings.SplitN(resp.Status, " ", 2)
		conn.Close()
		return nil, errors.New(f[1])
	}

The slice size of f is not checked. Therefore, if resp.Status has no spaces, the following panic is generated instead of the error:
   panic: runtime error: index out of range
   goroutine 101523 [running]:
   net/http.(*Transport).dialConn(0xc4236d3860, 0xd6a780, 0xc42007e030, 0xc4231c1400, 0xa5ed93, 0x5, 0xc420b9dee0, 0xd, 0xc4238a8380, 0x0, ...)
           /usr/lib64/go/src/net/http/transport.go:1128 +0x2090
   net/http.(*Transport).getConn.func4(0xc4236d3860, 0xd6a780, 0xc42007e030, 0xc421372300, 0xc4209a40c0)
           /usr/lib64/go/src/net/http/transport.go:943 +0x78
   created by net/http.(*Transport).getConn
           /usr/lib64/go/src/net/http/transport.go:942 +0x393

Thank you for the report @derlaft. This issue is a duplicate of https://github.com/golang/go/issues/21701 and there is a CL and test ready for it, we just need to test out the code on Windows and then we'll be ready to merge and fix it. /cc @tombergan

Thank you. Sorry for the dupe -- I checked the issues before posting, but not carefully enough.

That's alright @derlaft, it wasn't an obvious duplicate at all, I too had to double check and it is only by familiarity of that issue that I recalled it :)

@derlaft please get the latest from master, this bug was fixed by CL https://go-review.googlesource.com/c/59990/ that was merged already. Thank you for your report.


(Contacts) Find A Contact and Take Action

There currently is no skill that will allow a user to find a contact and take action on that Contact.
We would need to add this with the following utterance support:
Find "Contact Name"
Call "Contact Name"
Call "Contact Name" on "Mobile / Home / Cell / Work"
Email "Contact Name" (May already be a function of Email)

tracked with #929


containers with extraModules in host

Basic info
To make sure that we are on the same page:

Kernel: Linux Rizilab 4.3.4 #1-NixOS SMP Thu Jan 1 00:00:01 UTC 1970 x86_64 GNU/Linux
System: 15.09.1076.9220f03 (Dingo)
Nix version: nix-env (Nix) 1.10
Nixpkgs version:"15.09.1076.9220f03"

Describe your issue here
I create new containers called beHaskell with this configuration
{ config, lib, pkgs, ... }:

with lib;

{ boot.isContainer = true;
  networking = {
    hostName = mkDefault "behaskell";
    useDHCP = false;
    firewall.enable = false;
    firewall.allowedTCPPorts = [ 80 443 ];

  };

  services = {
    #openssh.enable = true;
  };

  users = {
    mutableUsers = true;
    extraUsers.Rizilab = {
      createHome = true;
      uid = 1001;
      extraGroups = [ "wheel" ];
      home = "/home/Rizilab";
      description = "Development Account";
      useDefaultShell = true;
    };
  };

}

when i tried to do nix-channel --update i cannot connect to the internet, i tried to ping google.com in my containers and it cannot connect.
so, i add containers configuration in my host /etc/nixos/configuration.nix like this:
 containers = {
    # behaskell is containers for learning haskell
    behaskell = {
      config =
        { config, pkgs, ... }:
        {
          boot = {
            kernelPackages = pkgs.linuxPackages_latest;
            kernelModules = ["e1000e"];
          };
          #privateNetwork = true;

      };
    };
  };

i add the kernelModules = [ "e1000e"], then nixos-rebuild switch in my host. It build perfectly and nothing happened. I can connect to my beHaskell container and do nix-channel --update and nixos-rebuild switch
the problem occurs when i tried to do nixos-container show-ip beHaskell from my host, the error says
/run/current-system/sw/bin/nixos-container: cannot get IP address
then i tried to add the following code:
hostAddress = "<IP_ADDRESS>";
localAddress = "<IP_ADDRESS>";

to the container attribute in my host configuration.nix, and when i tried to do "nixos-rebuild switch" it got an error :
The optioncontainers.beHaskell.hostAddress' defined in <unknown-file>' does not exist.
Expected result
<IP_ADDRESS>
Actual result
/run/current-system/sw/bin/nixos-container: cannot get IP address
and after add the localAddress and hostAddress:
The optioncontainers.beHaskell.hostAddress' defined in <unknown-file>' does not exist.
Steps to reproduce

nixos-container create beHaskell
nixos-container start beHaskell
nixos-container root-login beHaskell
nano /etc/nixos/configuration.nix
add the configuration above
in host, nixos-rebuild switch

Is your container declarative or imperative? It looks like you've mixed these two configuration schemes. When created as described in your steps to reproduce everything should be fine because your container will be assigned an IPv4 during creation.

@hrdinka , i mix declarative and imperative. At first, i did imperative and it has IPv4 when i did show-ip. But, i cannot connect to the internet, because my "imperative" way does not recognize my ethernet card.
That's when i remember that i added extraModules to my host configuration.nix. So, I added these extraModules to the containers via "declarative" way, and suddenly, the IPv4 is missing and produce the above error.

Looks like you've got several things mixed up here:

Mixing declarative and imperative containers does not work. Declarative containers are built together with your host configuration. This means after a host rebuild your container will be configured as defined in the host config. After a rebuild within the container it will obey your containers config file. There is no merging or mixing taking place between these config schemes.
Your containers config completely belongs to /etc/nixos/configuration.nix of your container. extraModules would belong there as well if there wasn't the following:
Containers aren't fully emulated virtual machines. They are just another process of your system and do not run their own kernel. If your host kernel understands you ethernet card your container does as well.
Containers use veth devices. Your containers network interface has nothing to do with your host interface. A veth device is a virtual ethernet device. These are created in pairs, this means your host sees one and your container the other one. These two device behave as they were two linked ethernet cards. This is a kernel feature and doesn't need any hardware drivers.

The reason why you didn't had network connectivity within your container is likely a different one. On a freshly created container you can examine your network interfaces with ip link. If you see an interface in there your interface is recognized and working. This means all drivers are loaded and if something does not work its due network configuration.
The problem is that your container is in a different subnet than your hosts wan and by default your host won't route between these two networks. If it does and your router is properly configured it will block all traffic from your container because it's not part of the routers subnet.
First check if ip forwarding is enabled on your host sysctl net.ipv4.ip_forward if not enable it via sysctl or by adding this to your host config:
kernel.sysctl = {
  "net.ipv4.ip_forward" = true;
  "net.ipv6.conf.all.forwarding" = true;
};

Next you can review (ip route) and setup your routing table to route between your container and your wan or use NAT to translate the containers IP range to the one used by your wan. The easiest and likely wanted solution for your problem is to use masquerading all traffic from and to your container via NAT. When masqueraded your host machine will rewrite all packages from your container to make them look like they are coming from your host machine. Your router will think it is talking to your host. On the way back packages are rewritten to make them target your containers ip. This is described in the nixos manual over here http://nixos.org/nixos/manual/#sec-container-networking.

Hi @hrdinka,
thank you for your suggestion, i think i did had misunderstanding about how containers work.
so in this case, if i do pure declarative configuration, i don't need to configure the configuration.nix on my containers?
thanks,
andika

Yes that's how it works. When using imperative containers your containers function like completely independent machines and won't be bothered by your host at all.
With declarative containers your containers are part of your hosts config. NixOS handles your host&containers as one entity and builds&updates these together.
Both behaviors are nice to have and come in handy depending on your projects needs. Decide for on cause you can't mix them.

I believe this issue can be closed.


Operator Overloading for Graphql Types

Library Version
1.0.0-RC3
Describe the bug
Overloaded operators currently produce functions in the Graphql Schema. This causes errors if an operator is overloaded for multiple types
i.e.
operator fun plus(x: MyInt): MyInt = this.value + x.value
operator fun plus(x: Int): MyInt = this.value + x
Would cause the error:
"Caused by: graphql.AssertException: All types within a GraphQL schema must have unique names. No two provided types may have the same name.
No provided type may have a name which conflicts with any built in types (including Scalar and Introspection types).
You have redefined the type 'MyInt' from being a 'GraphQLObjectType' to a 'GraphQLObjectType'"
To Reproduce
Will need a little bit of time to create a simple example but can provide if helpful.
Expected behavior
I would expect overloaded operator's not to produce nested functions in the graphql schema.

When generating the schemas we use reflections to find all public properties, fields and functions. Since operators are just functions that have some additional special treatment by Kotlin they are picked up as such.
Unsure how useful those functions would be as part of the Graph but you can expose them with different names using @GraphQLName or exclude them from the schema using @GraphQLIgnore.


[AKS] Upgrade api-version to 2022-01-01 for ACS module

Description

Update SDK "azure-mgmt-containerservice" to 17.0.0 and API version to v2022_01_01 in acs module.

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change.
[Component Name 2] az command b: Add some customer-facing feature.

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

AKS


[C++][Parquet] Print more helpful failure message when PARQUET_TEST_DATA environment variable is not set

Reporter: Wes McKinney / @wesm
Note: This issue was originally created as PARQUET-1021. Please see the migration documentation for further details.

Deepak Majeti / @majetideepak:
Fixed with PARQUET-1255

Antoine Pitrou / @pitrou:
This is the same issue as PARQUET-1255.


PR: Update Japanese translation for 3.2.5

Dear @ccordoba12
I'm sorry to be late.
I updated the Japanese translation for 3.2.5.

@sohnishi, don't worry about it. Thanks a lot for your help!


⚠ API - Lens has degraded performance

In 28654f8, API - Lens ($LENS_API_URL) experienced degraded performance:

HTTP code: 503
Response time: 410 ms

Resolved: API - Lens performance has improved in d29ca56 after 6 minutes.


use trim spaced string

Describe the PR
use trim spaced string.

Codecov Report

Merging #353 into master will not change coverage.
The diff coverage is 50%.

@@           Coverage Diff           @@
##           master     #353   +/-   ##
=======================================
  Coverage   85.43%   85.43%
=======================================
  Files           7        7
  Lines        1318     1318
=======================================
  Hits         1126     1126
  Misses        111      111
  Partials       81       81

Impacted Files
Coverage Δ

operation.go
86.7% <50%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3335bb4...821b19b. Read the comment docs.

@easonlin404 review plz.


checkpoint: wait checkpoint finished if exit before success

What problem does this PR solve?
Fix the bug that if lightning exit before finished, e.g. manually cancelled, some checkpoint update may not be written to db before exit.
We have encounter this bug several times in recent integration test's failure. Example:  https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/lightning_ghpr_test/detail/lightning_ghpr_test/1570/pipeline/47/
What is changed and how it works?
If lightning exit early, wait for checkpoint finish before exit.
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects
Related changes

/build


HBase connectivity tests 'hang'

HBase connectivity tests are hanging:
:compileJava UP-TO-DATE
:compileGroovy UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:Example
Connection request: [route: {s}->https://bi-hadoop-prod-xxxx.services.dal.bluemix.net:8443][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
Connection leased: [id: 0][route: {s}->https://bi-hadoop-prod-xxxx.services.dal.bluemix.net:8443][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
Connecting to bi-hadoop-prod-xxxx.services.dal.bluemix.net:8443
CookieSpec selected: best-match
Re-using cached 'basic' auth scheme for https://bi-hadoop-prod-xxxx.services.dal.bluemix.net:8443
Target auth state: CHALLENGED
Proxy auth state: UNCHALLENGED
Attempt 1 to execute request
Sending request: GET /gateway/default/hbase/version HTTP/1.1
 >> "GET /gateway/default/hbase/version HTTP/1.1[\r][\n]"
 >> "Accept: application/json[\r][\n]"
 >> "Host: bi-hadoop-prod-xxxx.services.dal.bluemix.net:8443[\r][\n]"
 >> "Connection: Keep-Alive[\r][\n]"
 >> "User-Agent: Apache-HttpClient/4.3.6 (java 1.5)[\r][\n]"
 >> "Authorization: Basic ***masked***[\r][\n]"
 >> "[\r][\n]"
>> GET /gateway/default/hbase/version HTTP/1.1
>> Accept: application/json
>> Host: bi-hadoop-prod-xxxx.services.dal.bluemix.net:8443
>> Connection: Keep-Alive
>> User-Agent: Apache-HttpClient/4.3.6 (java 1.5)
>> Authorization: Basic ***masked***
> Building 80%  <--  The tests are hanging at this point

As a workaround, I updated my properties file to use the IP address and the example worked ok.   I then reverted my properties file to use host name and the example worked ok.
Keeping this ticket open in case anyone encounters a similar issue.

The workaround worked on one cluster, but not on a different one :(

Have not seen this on my cluster. HBase test runs as expected.

They were working fine on mine, and then suddenly stopped.

Added some code to force timeout for HBase groovy example: https://github.com/snowch/biginsight-examples/commit/f1f4cc0dbdf0dd196b7631b078e78e4a9e420322

I've reverted my changes and the tests are running ok.  The flipside of this is that if there is a hang again, the only way to break out of ./gradlew tests will be CTRL+C.
I did try an alternative approach from build.gradle in the project root folder, however it seemed to conflict with gradle internals and resulted in crashes that I couldn't resolve.  The code I tried is here:
        // run the actual build in a separate thread so we can set a timeout
        def threadPool = Executors.newSingleThreadExecutor()
        def writer = new FileWriter("build/test/${proj}.txt")
        try {

            def buildClosure = { p, w ->
                GradleRunner.create()
                    .withProjectDir(file("./examples/${p}/"))
                    .forwardStdError(w)
                    .forwardStdOutput(w)
                    .withArguments(['Example', '-PdebugExample', '--debug'])
                    .build()
            }
            def future = threadPool.submit({ -> buildClosure proj, writer } as Callable)
            future.get(project.ext.TEST_TIMEOUT, TimeUnit.SECONDS)

        } catch (TimeoutException e) {
            e.printStackTrace()
            println "Test failed ${proj} (timed out)"
            return
        } catch (Exception e) {
            e.printStackTrace()
            println "Test failed ${proj}"
            return
        } finally {
            //threadPool.shutdown()
            writer.close()
        }

BTW I had problem with the previous version as well. Hang or connection issue. I ran latest version (no TO code) and it is working fine for me now.


Fix images in cirq docs

Move images to an images directory
Also use relative links to force docs to add them.

Where does _static(https://github.com/quantumlib/Cirq/tree/master/docs/_static) fit into this?

Changed the images to _static and moved the links back to absolute.


Configuration extensions for RedisServiceConnectorFactory creation

Resolves #9.

Added new IConfiguration extensions for creating a RedisServiceConnectorFactory that can be used outside of service registration.
Added support for splitting application configuration from Redis connector configuration. This means you don't have to store your redis node at the root of your app config anymore - you can pass in a child node with the connector config or you can generate the RedisCacheConnectorOptions yourself.

Note something that would potentially make contributions easier: It appears you're using a Gitflow-ish dev/master release format. If you set the "default branch" to "dev" in the repository options in GitHub, it'll automatically set all PRs to be based off "dev" by default rather than "master."

@tillig Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

@tillig Thank you for signing the Contributor License Agreement!

@tillig Thanks for the contribution!
You're right about the workflow, we're working on providing additional guidance about workflows and more and we hope to have that out soon!


Add pyGeo specific contribution information

Purpose
Add a link from the pyGeo docs to the general MACH contribution guide page as well as pyGeo specific information
Expected time until merged
1 day, very simple addition
Type of change

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (non-backwards-compatible fix or feature)
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no API changes)
[X] Documentation update
[ ] Maintenance update
[ ] Other (please describe)

Checklist

[ ] I have run flake8 and black to make sure the Python code adheres to PEP-8 and is consistently formatted
[ ] I have formatted the Fortran code with fprettify or C/C++ code with clang-format as applicable
[ ] I have run unit and regression tests which pass locally with my changes
[ ] I have added new tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation

approved


Add files via upload

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

[0xf09209cc5eae846ee2cc4a493e7b962ca7bcfbbb.png ] Filename in the lowercase format
[![reallifecointoken1](https://user-images.githubusercontent.com/47842040/55522843-76efd880-563b-11e9-9430-929f44193730.png]
[] Image size: 256 by 256 px

Please follow requirements

What needs to be fixed?

@cherrybomb420 Everything https://github.com/TrustWallet/tokens#requirements

Harsh lol...working on it now, my bad..
On Wed, Apr 3, 2019, 6:41 PM Mykola<EMAIL_ADDRESS>wrote:

@cherrybomb420 https://github.com/cherrybomb420 Everything
https://github.com/TrustWallet/tokens#requirements
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/TrustWallet/tokens/pull/1393#issuecomment-479717734,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AtoC-LdB2CLC58J2ppC-addvfzs1p95gks5vdVhTgaJpZM4cZ5cb
.


Typo in page 768

Under equation (20.1) there is a typo in defining the Dataset.
$$ \mathcal{D}  = \set{ vx_1, ... , \mathbf{x}_n } $$
Must be written to:
$$ \mathcal{D}  =  \set{ \mathbf{x}_1, ... , \mathbf{x}_n } $$

fixed


fix : fix wrong colors in AuthUI

Fix #502
@samtstern This should resolve the issue 🙂
Please let me know if there are any more issues.

Dang, accidentally pushed to master again. :/


docs: merge contribute.md and development.md

Type of change:

What this PR does / why we need it:

Resolves  #905
Pre-submission checklist:

[x] Did you explain what problem does this PR solve? Or what new features have been added?
[ ] Have you added corresponding test cases?
[x] Have you modified the corresponding document?
[x] Is this PR backward compatible? If it is not backward compatible, please discuss on the mailing list first

Please check the CI errors.
You can run make update-mdlint locally.

请检查 CI 错误。 您可以在make update-mdlint本地运行。

development.md after renaming, there are three files containing development.md. Can I modify it?(README.md line 92, CHANGELOG.md line 419, getting-started.md line 70)

development.md after renaming, there are three files containing development.md. Can I modify it?(README.md line 92, CHANGELOG.md line 419, getting-started.md line 70)

Sure, please!

Hi @bin-ya  , thank you for your contribution!
Here is the Contributor T-shirt form[1], if you're interested, kindly take a look :)
[1] https://github.com/apache/apisix/blob/master/CONTRIBUTING.md#contributor-t-shirt


Dep ensure/init seem to miss certain dependencies

When including github.com/Shopify/sarama dep ensure/init seem to miss the dependency github.com/stathat/go which is included in github.com/rcrowley/go-metrics/stathat/stathat.go.
Result of dep ensure -v
Root project is "kafkatest"
 1 transitively valid internal packages
 1 external packages imported from 1 projects
(0)   ✓ select (root)
(1)     ? attempt github.com/Shopify/sarama with 1 pkgs; at least 1 versions to try
(1)         try<EMAIL_ADDRESS>(1)     ✓ select<EMAIL_ADDRESS>w/1 pkgs
(2)     ? attempt github.com/davecgh/go-spew with 1 pkgs; at least 1 versions to try
(2)         try<EMAIL_ADDRESS>(2)     ✓ select<EMAIL_ADDRESS>w/1 pkgs
(3)     ? attempt github.com/eapache/go-xerial-snappy with 1 pkgs; at least 1 versions to try
(3)         try github.com/eapache/go-xerial-snappy@master
(3)     ✓ select github.com/eapache/go-xerial-snappy@master w/1 pkgs
(4)     ? attempt github.com/eapache/go-resiliency with 1 pkgs; at least 1 versions to try
(4)         try<EMAIL_ADDRESS>(4)     ✓ select<EMAIL_ADDRESS>w/1 pkgs
(5)     ? attempt github.com/rcrowley/go-metrics with 1 pkgs; at least 1 versions to try
(5)         try github.com/rcrowley/go-metrics@master
(5)     ✓ select github.com/rcrowley/go-metrics@master w/1 pkgs
(6)     ? attempt github.com/eapache/queue with 1 pkgs; at least 1 versions to try
(6)         try<EMAIL_ADDRESS>(6)     ✓ select<EMAIL_ADDRESS>w/1 pkgs
(7)     ? attempt github.com/klauspost/crc32 with 1 pkgs; at least 1 versions to try
(7)         try<EMAIL_ADDRESS>(7)     ✓ select<EMAIL_ADDRESS>w/1 pkgs
(8)     ? attempt github.com/pierrec/lz4 with 1 pkgs; at least 1 versions to try
(8)         try github.com/pierrec/lz4@master
(8)     ✓ select github.com/pierrec/lz4@master w/1 pkgs
(9)     ? attempt github.com/golang/snappy with 1 pkgs; at least 1 versions to try
(9)         try github.com/golang/snappy@master
(9)     ✓ select github.com/golang/snappy@master w/1 pkgs
(10)  ? attempt github.com/pierrec/xxHash with 1 pkgs; at least 1 versions to try
(10)      try<EMAIL_ADDRESS>(10)  ✓ select<EMAIL_ADDRESS>w/1 pkgs
  ✓ found solution with 10 packages from 10 projects

Solver wall times by segment:
         b-list-pkgs: 119.972212ms
              b-gmal:  76.130478ms
     b-source-exists:  31.391625ms
         select-atom:   2.789118ms
            new-atom:   2.643822ms
             satisfy:   1.781472ms
  b-deduce-proj-root:    810.731µs
               other:    196.282µs
         select-root:     173.27µs

  TOTAL: 235.88901ms

Hello, welcome! 🎉 🎉
We're working on adding a bit to these messages that would allow us to tell just by reading them, so I had to confirm, but...yep - as of this writing, at master, github.com/Shopify/sarama (and sub-packages) imports github.com/rcrowley/go-metrics, but none import github.com/rcrowley/go-metrics/stathat.
Which means, this is working as intended. Even though the project github.com/rcrowley/go-metrics is a dependency, the specific subpackage github.com/rcrowley/go-metrics/stathat is not, so dep doesn't pull in imports unique to that package - namely, github.com/stathat/go. If anything did start importing that subpackage, then stathat/go would show up in results.
This is a feature, not a bug (no, really!) - think about how awful it would be to depend on one of Kubernetes' ~1000 packages, but to pull in the dependencies for all of them.

Right, I was stupidly using go build kafkatest/... which threw an error (obviously) while I should have just use go build kafkatest. Sorry for the confusion.


Slow parsing performance

I've just started using this library to check that parsed JSON objects conform to the types I define. I'm really enjoying it, but I'm afraid I've hit a performance snag.
Everything works fine with small tests but when I ran it on a slightly bigger input file (~10MB) I noticed that it was really slow.
I've tried the latest version and the beta, and check vs parse, with similar results. The beta is faster, but I don't see a big difference between check and parse.
After profiling the check version in the beta, I'm seeing calls to .check taking in the 500ms-1100ms range per object.

Are those numbers typical, or am I doing something wrong with the schema definitions?
My schema definitions look like:
const EntryJsonSchema = z.object({
  a: z.string().optional().nullable(),
  b: z.string().optional().nullable(),
  id: z.string().optional().nullable(),
  creation: z.string().optional().nullable(),
  content: ContentJsonSchema.optional().nullable(),
  labels: z.string().array().optional().nullable(),
  answers: AnswerJsonSchema.array().optional().nullable(),
  results: ResultJsonSchema.array()
    .optional()
    .nullable(),
});

const ContentJsonSchema = z.object({
  id: z.string().optional().nullable(),
  title: z.string().optional().nullable(),
  version: z.union([z.number(), z.string()]).optional().nullable(),
});

const AnswerJsonSchema = z.object({
  key: z.string().optional().nullable(),
  value: z.any().optional().nullable(),
});

const ResultJsonSchema = z.object({
  key: z.string().optional().nullable(),
  value: z.any().optional().nullable(),
});

I'm really hoping that there is a way to speed it up, as this is too expensive for the use case where I'll have to process files with 100k+ objects.
Thanks!

Honestly I've never really stress tested Zod with payloads that large. I suspect there is lots of low-hanging fruit that would fastly improve performance.
I'm a little confused though. Zod is taking 500+ms to parse each object? How big is each object? Is it possible you're hitting memory limits of the Nodejs process? Without a sample data set it's hard for me to duplicate this.

Thanks for taking the time to think about this.
I was also very confused by the slowness I was seeing. I first managed to narrow it down to the parse or check call, and then confirmed with the profiler. I would not be surprised if there are other effects at play here.
I usually work with go and was ready for performance to be lower but as I don't have experience of what is considered "normal" I decided to ask.
I'm going to try to prepare a dataset and extract a code example that allows to reproduce.

Yeah all of Zod's complex logic is in .parse, and .check just delegates to .parse. I don't think this is normal.

@mgilbir Here is a github that tracks benchmarks for different validation libraries: https://github.com/moltar/typescript-runtime-type-benchmarks if performance is an issue.

@davidmdm Thanks for pointing to those benchmarks!
I took a look with the profiler and found that makeError was allocating, and doing a lot of work upfront, on every call to the parse function even when no error occurred. I'm new to typescript, so I'm not sure this is the best way to solve it, but it may point you towards some low-hanging fruit. Code is available in PR #218

My team was using zod to validate JSON messages coming from IoT devices via a queue. We received hundreds of messages per minute (so not exactly high volume).
Validating 150 messages with zod takes more than 6 seconds.
Using ajv we can validate the same amount of messages in 120 ms with a fraction of the CPU usage.
We liked the idea of zod, but we have to move away from it until we can get a decent performance. :-(

I saw this test results and it actually came after I had already looked into a issue with slow code where I discovered that zod was taking up 600 ms to parse 1000 json elements.
moltar/typescript-runtime-type-benchmarks
Zod is one of the slowest in the overview.
Normally speed is not a problem since I use zod on user input, but if you run a busy api endpoint this could be something important.

@dgg I maintain myzod, which is basically just a lightweight reimplementation of zod. It doesn't have a big community like zod does, but it is much much faster, and has an extremely similar api.

@dgg I maintain myzod, which is basically just a lightweight reimplementation of zod. It doesn't have a big community like zod does, but it is much much faster, and has an extremely similar api.

@mgilbir I just published a big rewrite of Zod's internal parsing logic. Would you be willing to upgrade to the latest v3 version (npm install zod@next) and let me know how performance has changed?

This is forcing me to abandon zod for the time being, unfortunately. I'm seeing it take seconds to parse even small chunks of data. I was hoping to see an improvement with v3 but it's actually worse for me. It's really unfortunate because I'm having a hard time finding a decent replacement. I was hoping to do a drop-in replacement with myzod, but that doesn't work quite the same and has some really unusual behaviors. Every other library I've tried has severe limitations as well, it seems. And none of them match the level of documentation found here. It's a really excellent library otherwise, and I appreciate it very much -- I hope I can use it again soon.

I have parsed 10k objects base on the schema provided by @mgilbir with version '3.0.0-alpha.29' and '3.1.0', the output is high improved by the team! @colinhacks
the result is:

24160ms on '3.0.0-alpha.29', which is 2.4s/parse
22137ms on '3.1.0', which is 2.2s/parse

`describe('Test suite for OfficePlanSchema', () => {
it('bigObject parse test', () => {
const bigObject = {
a: 'hello',
b: 'hello',
id: 'Welcome',
creation: 'Today is Tuesday',
content: {
id: 'id1234',
title: 'Software Engineer',
version: 100,
},
labels: ['2342l42342344', '24234234', '234243242345', '2342342342342'],
answers: [
{
key: 334242,
value: {
name: 'hello',
age: 20,
},
},
],
result: [
{
key: '324234324234',
value: {
name: 'hello',
age: 20,
a: 'hello',
b: 'hello',
id: 'Welcome',
creation: 'Today is Tuesday',
content: {
id: 'id1234',
title: 'Software Engineer',
version: 100,
},
},
},
],
};
const start = new Date().getTime();
console.log('start =', start);
for (let i = 0; i < 10000; i++) {
EntryJsonSchema.safeParse(bigObject);
}
const end = new Date().getTime();
    console.log('end =', end);
    console.log('takes = ', end - start);
})

});`

I still not do a test on the version before 3.0.0-alpha.29, will do that
today or tomorrow, and let you know the result.
myzod looks great!
On Fri, 11 Jun 2021 at 06:35, Joe Krill @.***> wrote:

@zhenhao18 https://github.com/zhenhao18 What sort of results were you
seeing before 3.0.0-alpha.29 that we can use as a comparison? Because
I'm still seeing very slow parsing performance in all versions,
including the latest (3.1.0).
Unfortunately I can't really share my dataset and schemas at the moment
because I'm using some real code and data to benchmark (if I have time,
I'll try to generate some mock data and schemas to share), but these are my
latest internal results:
library parse time (ms)
zod 3.1.0 2200.53
zod 3.0.0 2102.35
zod 1.11.17 656.82
myzod 1.7.4 19.19
We've switched to myzod https://github.com/davidmdm/myzod for now,
which is why it is included here. And as you can see it's significantly
faster (and the schemas used in each case are virtually identical - nothing
has been slimmed down or removed). But myzod is not without issues and
concerns in itself. Also interesting to note how much performance has
decreased in zod v3 from v1.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/colinhacks/zod/issues/205#issuecomment-859038556, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AT6BEI543N6N2UE2E4W7NBDTSEOZPANCNFSM4TECTSOQ
.

This morning I decided to check in on zod and see if there's been any changes (it's been a few weeks since last I checked), and WOW! HUGE improvements since v3.1!  Here is the results of my latest benchmark on our internal dataset adding Zod 3.5.1:

library
parse time (ms)

zod 3.5.1
76.74

zod 3.1.0
2453.06

zod 3.0.0
2080.56

zod 1.11.17
723.87

myzod 1.11.17
26.72

So just a massive improvement. This is so awesome. Thank you so much to @milankinen, @colinhacks, and everyone who has been contributing -  I really appreciate you all. This is really amazing and definitely makes zod a usable option for me again - I'll definitely be looking into switching back.
Side note: I did run into some minor tweaks I had to make to my schema where I was using passthrough with intersection which is causing an error in 3.5.1, but that was pretty straightforward to clear up.

Just went through versions on our end too. Seems like there's a huge performance regression from 3.9 -> 3.10. One of our functions went from 192ms to 660ms between the two

I really love this lib and anything typescript, but taking 3x to 4x to parse things compared to myzod. Performance should be a one of the highest priority as the ecosystem starting to build around zod.

@mmahalwy Do you have an example of something that has gone up in time? I think that would help debug and maybe a benchmark suite could be added

Recently migrated from ajv to zod then to myzod, so there is no baseline for me to compare between zod versions. Just compared between zod & myzod and zod overall making my task to take around 50% to 80% more time to complete. As the task itself has lot's mini validations individually it won't look much but they add up by the time task completes. I also suspect there could be some GC pressure too.

I've implemented #1023, #1021, and #1022 to explore different vectors at making zod faster. I think updating the target is the highest payoff for the least amount of change, but the other PRs also have some notable perf improvements. I'll keep looking - I think it's possible to make zod as fast as the alternatives.

@tmcw
Incredible work and thanks so much for digging in! I'll spend some time tomorrow and this weekend trying to think through any implications that need to be addressed.
The point about switching our target is an important one: we have mostly treated compatibility as a best-effort thing, but this would be a bit of a line in the sand. I am not involved in enough open source projects of this size to have a good feel for the general approach the community usually takes here, but I feel like that might warrant a major bump?

@tmcw I’m not sure if I understand the current caching correctly. Are you implying that the cache is permanent? This would explain why parsing a 150 MiB JSON with a complex schema uses up over 4 GiB memory (not sure how much over, just that it’s less than 8 as I had to manually bump node’s memory limit) on one of my side-projects.
Do you know if the current cache gets freed over time? Also, do you know how I could try out this change to see what the performance difference is with my real-world data? The schema is insanely complex as the data structure it has to parse is ancient and organically grew. There’s lots of .optional()s etc.. So, would be really interesting to see the impact of your PRs.
Also, thanks a lot for starting to work on zod performance. I’m really looking forward to this as it currently takes >4 GiB of memory and around 2 minutes to parse the data. So, let me know if there’s a way to try this out.
I originally commented this on the PR, but I think let’s keep the PR’s comments for the code itself

I've focused mostly on the primitive types because they've been pretty straightforward to tweak & get some little wins. Should probably take a break to work on my product 😆 but in the meantime, some thoughts:

A lot of real-world schemas look like they include .optional().nullable() on a lot of types. That has come with a performance penalty, which #1028 reduces, but could be reduced more by making optional & nullable more like 'modifiers' (like maxLength) than their own types. But that would make the codebase less elegant. Anyway, this is all at the micro-level with values and probably isn't the biggest performance drag.
I bet a bunch of realworld performance is heavily influenced by ZodObject, which is already pretty well-written but does require an array of objects be created and then merged back into each object. That seems like a worthwhile thing to optimize in the future.

Fwiw, if folks want to benchmark their things, what I've been relying on is the npm run benchmark script in this repo combined with
node --prof bench.js
node --prof-process --preprocess -j isolate*.log > profile.v8log.json

And then dumping the results into speedscope.

@tmcw  that is extremely cool!

I gotta say @tmcw, your work is seriously appreciated. I migrated away from Zod because its real-world performance was too slow but this focus on performance makes it viable again!

I think a fresh issue about performance should probably be more specific than this one, otherwise it's not really something that could ever be closed. Performance could always be improved, it's a theme not a task.
My guess in terms of further performance tweaks is that zod would need an option to turn off cloning, which would also need to disable methods like transform or require a new fancy implementation of those methods. Cloning is basically the main overhead, afaict, and the modules that do well in the moltar benchmarks are the ones that do not clone.

@tmcw That was insightful! There has only been one other ticket mentioning "cloning" which was closed.
https://github.com/colinhacks/zod/issues/922
What do you think? 😄

Yep, I think that's the direction (though obviously it'd be useful to do a prototype and early benchmarks to confirm that intuition). Parse, don't validate is a great principle and probably the right default, but I think it might be the cause for a sort of performance ceiling.

@tmcw I’m not sure if that fits into the scope of zod though. For me, this library has always been about type safety and security (e.g. by disallowing unexpected stuff in API parameters) by ensuring that data is always recreated rather than cloned or copied. I’m not sure if it’s possible to guarantee avoiding all the possible issues of prototype pollution and otherwise modified objects without recreating the data from scratch

Sure, so - a no-clone option, as far as I can see it, would mean:

No transforming would be possible, no coerce, objects will only support .strict or .passthrough, not .strip
Objects or arrays with prototype additions would leak through instead of being recreated as plain-old objects.

This is, fwiw, the same tradeoff you'll see in any of the faster modules on moltar's benchmarks. I would reckon that people are generally okay with these tradeoffs because those modules are heavily used, and if zod doesn't support a no-cloning mode, then it'll never get close to their performance.
Anyway, cloning would be a choice, and you could use the cloning (default) mode and get the guarantee of fresh object prototypes, or use the no-cloning mode and accept that you might get a modified object prototype. At least for my application, I think no-cloning would be very useful and would probably be a lot faster, and modified object prototypes - in my experience - are not especially concerning.

This affects array and object schemas mostly, correct? I could see adding different "no-clone" versions of those schema types as a potential optimization that you have to opt into explicitly. Definitely would feel a little unergonomic in some cases since if you have nested schemas you'd have to ensure they are also of this same type if you wanted maximum performance in that case.
Might be worth opening a Discussion about this.

I strongly agree with the idea about "parse, don't validate" semantics which is (imho) one of the biggest reasons to choose zod over those high performance alternatives. We need to keep in mind that those libraries have been developed from the scratch focusing explicitly on the performance, whereas zod has had more focus on developer friendliness and fail-proof semantics. Because of this, the "legacy" of the former design and implementation choices (e.g. decision to support asynchronous parsing) will have a moderate effect on the performance even though the API and semantics are changed. As a result, zod loses one of it's main advantages (developer experience) and gets a little bit better performance but is still not able to complete with the top ones in terms of performance.
Luckily there still seems to be plenty of room for performance improvements in zod's internals. 🙂 I quickly skimmed through the codebase and at least following parts could give some nice speed improvements without changing the existing API and semantics:
Parse context management
The current implementation seems to be using contexts with "lazy" parsing path where path is concatenated upon request. In theory, this is a nice idea in theory, but if we take look at the usage, path is read for each object property. That's O(k*n) complexity where k = length(path) and n = length(objectKeys). In addition, the whole parse context get "cloned" for each property because the path is different for each key. Same applies to arrays. This is a huge amount of internal cloning, object allocations and GC pressure that must be done regardless of the public API semantics.
I blame myself of this design choice. 😞 When I did my performance improvement PR, I left the path management in such state that the path is constructed prematurely which was definitely a mistake. If we think about where the path is really needed, the only places are issues that are returned to the user. By "returned", I mean issues that are visible to the user of zod APIs - as long as the parsing is in progress, the parse path isn't relevant. In practice this means that the path must be generated only if the (sub)-parser returns any issues. In pseudo:
ZodObject._parse = (ctx, object) =>
  const issues = []
  const parsed = {}
  shapeKeys.forEachKey key =>
    const keyResult = keyParser._parse(ctx, object[key])
    if (isInvalid(keyResult))
      keyResult.issues.forEach issue => issue.path.unshift(key)
      issues.push(...keyResult.issues)
    else
      parsed[key] = keyResult.value

  return issues.length > 0 ? INVALID(issues) : OK(parsed)

Even the ctx can be passed as it is because there are no dynamic parts in it anymore. The only required "cloning" is when pushing key issues to the object's issues. In fact, this could also be eliminated by using linked list (linking two lists is an O(1) operation). Compared to the current implementation, the difference should be significant.
Extra object keys parsing
The current implementation has roughly O(n^2)  complexity. And in addition those extra keys are not even needed in the default mode. Fortunately the fix for the default case (strip) should be trivial, just use shapeKeys and don't even try to find extras. For passthrough, strict and catchall, the most efficient implementation that comes to my mind is to merge both shape and data object into single work-in-progress clone and use it:
// same error handling as in default "strip" case
const issues = []
const wip = Object.assign({}, shapeObject, dataObject)
for const key of keys(wip)
  if key in shapeObject
    wip[key] = shapeObject[key]._parse(ctx, dataObject[key])
  else
    wip[key] = handleExtraKey(ctx, key, dataObject[key])

return issues.length > 0 ? INVALID(issues) : OK(wip)

Other improvements
There are also some other smaller improvements that could be implemented, e.g using function composition for number/string refinements and simplifying the ok/aborted/invalid result handling but I'm not sure whether they're worth changing or not.

the only places are issues that are returned to the user

Now that I looked the codebase again, it seems that the path is actually needed in the refinement context and that might be the reason why I left the implementation as it is... 😬
The .path property is not documented in the README or in the error handling documents at all, so how big deal it would be to remove it? Technically it's a breaking change but not to the official documented feature... 😇

@milankinen isn't path the only reasonable way to programmatically use zod errors? It'd be a pretty big deal if they're not machine-readable anymore

Path is definitely documented in multiple places including the error handling guide.

Yes, path indeed is an important property when processing the received parsing/validation issues and the issue contract should not be changed. 👍 But I'm not talking about issue paths. I'm talking about refinement context path property that is not officially document (or at least I didn't find any references to it) but still accessible through superRefine, e.g.
import { z } from "zod";

const notTsers = z.string().superRefine((s, ctx) => {
  console.log("ctx.path", ctx.path);     // <<< this
  if (s === "tsers") {
    ctx.addIssue({
      code: z.ZodIssueCode.custom,
      message: "can't be 'tsers'"
    });
  }
});

const person = z.object({
  name: z.object({
    first: notTsers,
    last: notTsers
  })
});

console.log(
  "result",
  person.safeParse({ name: { first: "matti", last: "tsers" } })
);

If you run the snippet, you can see that the ctx.path prints ['name', 'first'] to the console. However, the parse issue returns the same path although I don't explicitly set it to the added issue.
My question is: how important the read-only access to the ctx.path during the superRefine callback execution is, and could that be removed? That is the only functionality that forces the path to be created prematurely and preventing the performance optimizations I presented earlier. Removing the ctx.path doesn't affect the issues returned from safeParse, they'll continue to include the path like before.

I've implemented the suggested tweak to the default object parsing - not collecting extraKeys, in #1393. It's a very modest performance bump, seems like around 5% in the realworld testcase. Unfortunately I think that's what we're looking at with the cloning strategy - there's maybe another 10-20% optimization left, but systematically zod will be magnitudes slower than non-cloning modules.

Hello,
We at Twilio are considering using your library to validate messages sent over WebSocket. The API is probably the most fantastic thing I've seen over any validators/parses I've checked. The only question that remains is performance. Our WebSocket protocol requires to be as optimized as possible and Zod hasn't really been great in that regard. I've read this discussion about adding an option that disables cloning. Do you know whether that's feasible in near future and if that will have a significant impact on performance?

@InfiniteRain Would you be willing to contribute to the performance improvement effort :)?

@kibertoad If adding a no-clone option is something that the maintainers would want and if it brings significant performance improvements, then I would love to contribute.

Oh wow a toggle for no clone mode would be incredible!

I'm definitely open to this! Slightly concerned about bundle-size bloat, and I'm not sure how things like transforms will work. But ultimately it's worth it if perf is a blocker on adoption. @InfiniteRain don't hesistate to get in touch about any implementational questions - Twitter is probably best 👍👍

@colinhacks I'm very interested in working on this! I'm on vacation right now, but once I'm back I'd love to get in contact with you to discuss the details further.

@colinhacks if zod supports tree shaking, I wouldn’t worry much about bundle size. It hardly matters in that case

Just to add an additional data point, since Slonik adopted Zod for all query validation, our CPU usage doubled. Avoiding cloning would dramatically improve things.

Typeform.com is also considering using zod for form validations and API request payloads in comparison to yup 👍

I did a very fast experimental internals refactoring as I described in the previous comments last weekend (diff, very WIPWIP) and got approximately 50% performance improvement in "real world" and "object" suites and 100% in primitive suites. I'd be nice to compare this with the no-clone approach. 🙂

We are currently parsing API responses with Immutable.js Record and looking to move to zod. Anyone know how performance compares?

@colinhacks Is this something that you might still consider for v4?

For those following this thread, https://github.com/duplojs/duplojs-zod-accelerator has been a good solution for me. I'm noticing improvements of ~50x in parsing speed (similar to what is shown on their ReadMe Benchmark). I can't infer types from the accelerated schemas, so I've been writing a lot of code like this, but it has been worth it so far for the performance gains on my largest schemas.
const UnacceleratedIntegerSchema = z.number().int().nonnegative();
export type Integer = z.infer<typeof UnacceleratedIntegerSchema>;
export const IntegerSchema = ZodAccelerator.build(UnacceleratedIntegerSchema);


doc: replace lena image with baboon image

Resolves #5
I included the baboon image in the tree since the baboon-image package is broken.
The A/B test image in the readme has also been replaced.

Thanks! Merged and updated the demo with a new script that generates the LUT in software, which was fun to experiment with.

Great, thank you !

Could you also release the npm package ?


The example throwing

While trying to run the example I get this:
Invalid operands 'Array' and 'float' in operator '+'
    mutate()        @ line 22:
buffer.weights[i] += randf_range(-curiosity_rate, curiosity_rate)

I got the same Error when I tried using RLNET. When I added print(buffer.weights[i]) it printed one Array containing 16 Arrays.
My network structure was [16,32,32,3]

Same bug here. Attempted to account for nested arrays but it must be stuck in a loop because the debugger won't show more than the logo when run.

Just solved the issue, changed the loop in the mutate function to this.  I am aware of this problem and fixed it a long time ago, but have not released an update. The main reason for this is that RLNNET does not actually use RL, it is a simple mutation algorithm. I'll try to recreate the RLNNET class from scratch so that it uses real RL, though I'm not sure if I know enough to make it. I'll also optimise a lot of things. Unfortunately, I don’t have free time to do this now; a lot of exams appeared suddenly. I'll continue working on this repository over the summer holidays. The update will be released by June 20th.

If you are running into trouble with model-based reinforcement learning, you could try implementing Q-Learning to prototype for now if it clicks better and should be more robust than what you have. Academic responsibilities first, but it's good to see you still want to work on it and contribute something to Godot. I've just been using Markov Chains for brains with adaptive weights for scores and state decisions but I'd really like to try your approach and addon in a multi-population life-simulation project at some point.
https://en.wikipedia.org/wiki/Q-learning
https://www.baeldung.com/cs/reinforcement-learning-neural-network

Hey Zero! Happy to hear that there'll be more updates soon. I'm very thankful that you've taken the time to do something like this and share it with the community, it really helped me out with my capstone. I'm looking forward to the rework, but just thought I'd share a quick hotfix for the issue if it might help other people. Good luck with your academics!

Hi @ZeroLikviz,
It seems that the RLNN example is also diverging, even with the modification provided by @braxus32  , as is I'm only able to make it reach 0.0 around 33% of the time. The rest it diverge to 0.9999.
I've just read the comment that around the 20th an update should be released so I'll wait for your update.

@GeminiSquishGames, @Remi123, @braxus32, @haloraphi,
The addon has been recently updated. Currently, it includes only a fully rewritten version of NNET. I will be working on PPONNET until June 20th.
@Remi123, this is the reason why I am rewriting everything.

@ZeroLikviz  Any news on the update?

@JekSun97
I'm working on the project, very slowly though. I'm missing a lot of necessary mathematics, so I'm currently studying it. I don't expect any updates to come out soon. I hope to finish this project by the end of summer 2025, but it's not very likely.
Thanks for asking. ^-^
The project isn't abandoned.


Update Hessian Lite Version

What is the purpose of the change
Brief changelog
Verifying this change

Checklist

[x] Make sure there is a GitHub_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GitHub issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Check if is necessary to patch to Dubbo 3 if you are work on Dubbo 2.7
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Add some description to dubbo-website project if you are requesting to add a feature.
[ ] GitHub Actions works fine on your own branch.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #9326 (fe52af3) into 3.0 (16319c5) will decrease coverage by 0.11%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                3.0    #9326      +/-   ##
============================================
- Coverage     65.18%   65.07%   -0.12%
+ Complexity      342      339       -3
============================================
  Files          1201     1201
  Lines         51821    51764      -57
  Branches       7723     7670      -53
============================================
- Hits          33781    33686      -95
- Misses        14417    14439      +22
- Partials       3623     3639      +16

Impacted Files
Coverage Δ

...ache/dubbo/remoting/transport/AbstractChannel.java
75.00% <0.00%> (-12.50%)
:arrow_down:

...dubbo/common/status/support/LoadStatusChecker.java
50.00% <0.00%> (-11.54%)
:arrow_down:

...e/dubbo/remoting/transport/netty/NettyChannel.java
51.13% <0.00%> (-9.10%)
:arrow_down:

...mmon/threadpool/support/fixed/FixedThreadPool.java
80.00% <0.00%> (-7.50%)
:arrow_down:

.../common/threadpool/serial/SerializingExecutor.java
70.37% <0.00%> (-7.41%)
:arrow_down:

.../threadpool/support/limited/LimitedThreadPool.java
83.33% <0.00%> (-5.56%)
:arrow_down:

...mmon/threadpool/support/AbortPolicyWithReport.java
78.94% <0.00%> (-5.27%)
:arrow_down:

...o/rpc/protocol/injvm/DefaultParamDeepCopyUtil.java
57.89% <0.00%> (-5.27%)
:arrow_down:

.../apache/dubbo/remoting/transport/AbstractPeer.java
63.04% <0.00%> (-4.35%)
:arrow_down:

...pache/dubbo/remoting/transport/AbstractServer.java
57.14% <0.00%> (-4.29%)
:arrow_down:

... and 39 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 16319c5...fe52af3. Read the comment docs.


Remove gitignore's TODO comment.

CodeClimate find and judge it is error if gitignore has TODO comments.

Relate to #36


Automated cherry pick of #7834: fix: uniform cachedimage active status

Cherry pick of #7834 on release/3.4.
#7834: fix: uniform cachedimage active status

/lgtm
/approve


Light curve fit parameters applied in a cosmological fit

Hello again,
I would like to know if I could use light curve fit parameters given by sncosmo directly in a cosmological fit (aside from bias correction, and second order erros). I saw that the color factor used in sncosmo is a little different from salt2 (10**-.4cCl compared to exp(cCl)). I tried to do a cosmological fit and omega_m and omega_l minimized doesn't make sense.
Thanks,
João.

You should be able to do this. There are several groups that are using sncosmo for cosmological analyses. I believe that @kbarbary validated the sncosmo SALT2 fits against snfit, so you should get nearly identical results with either fitter.


How to save the network graph ploted by tensorwatch?

How to save the network graph ploted by tensorwatch? I want to save the graph. Is there any way to do that?

To save manually, click on the blue power botton on top-right and then save image by right clicking on it.

this should add in read.md


[Question] Is is possible to translate Django's Permission model (the name)

I know the issue tracker is not always the right place for a question, but we would like to migrate our app that uses django-modeltranslation to Django version 2.1. Since modeltranslations is currently unmaintained, and only support Django 1.11 (more or less), we are thinking about using this package.
I think it has everything we need (apart maybe from some glue code which is easy to create), except support for multi-language permissions (contrib.auth.model.Permission.name), am I correct?
Do you maybe have any idea how to circumvent this issue?

Hi!
Since the new release (0.8), the Permission.str = fields.translated_attrgetter("name") is giving me the error: TypeError: Translated_attrgetter() missing 1 required positional argument: 'field'.
Is there a quick fix for this?
I'm still using the old 0.7.2 until I have time to look into this, but maybe you know a fix so we can stay up-to-date. And it is also nice to mention it in this thread I think :)

Oh I'm sorry for breaking your code!
Something like the following should work:
Permission.__str__ = lambda self: getattr(
    self,
    to_attribute("name", language=get_language() or settings.LANGUAGES[0][0]
)

Thanks a lot! It is working :)

On another note:
    Permission.add_to_class(to_attribute("name", language_code), CharField(
        _("Name [" + language_code + "]"), max_length=255, blank=True, default=''))

adds a migration file every time I run manage.py makemigrations.
Is there a way to circumvent this that you know of?

The _("Name [" + language_code + "]") will probably not work the way you want it to. The translation strings "Name [en]", "Name [other]" will not be picked up by makemessages.
I'd recommend using only _("name") as verbose_name and maybe use the TranslatedFieldAdmin if you want to see the fields' language code in the administration interface.

Sorry for the late reply, I first wanted to test your solution thoroughly. However, it seems that it does not matter what I make for the verbose name. It seems that the for loop (so the 'dynamic' generation of the extra fields) is the culprit.
for language_code, dummy in settings.LANGUAGES:
    Permission.add_to_class(to_attribute("name", language_code), CharField(
        _("name"), max_length=255, blank=True, default=''))

The final code that does not generate migrations on every makemigration:
in models.py:

Permission.__str__ = lambda self: getattr(
    self,
    to_attribute("name", language_code=get_language() or settings.LANGUAGES[0][0]
                 ))

for language_code, dummy in settings.LANGUAGES:
    if not hasattr(Permission, to_attribute("name", language_code)):
        field = CharField(
            _("name") + ' [' + language_code + ']', max_length=255, blank=True, default='')
        Permission.add_to_class(
            to_attribute("name", language_code), field)

And in migrations/0001_initial.py:
import django.contrib.auth.models
from django.contrib.auth import validators
from django.db import migrations, models
from django.utils import timezone

TARGET_APP = 'auth'

class Migration(migrations.Migration):

    def __init__(self, name, app_label):
        # overriding application operated upon
        super(Migration, self).__init__(name, TARGET_APP)

    replaces = (
        (TARGET_APP, '0001_initial'),)

    dependencies = [
        ('contenttypes', '__first__'),
    ]

    operations = [
        migrations.CreateModel(
            name='Permission',
            fields=[
                ('id', models.AutoField(verbose_name='ID',
                                        serialize=False, auto_created=True, primary_key=True)),
                ('name', models.CharField(max_length=50, verbose_name='name')),
                ('content_type', models.ForeignKey(
                    to='contenttypes.ContentType',
                    on_delete=models.CASCADE,
                    to_field='id',
                    verbose_name='content type',
                )),
                ('codename', models.CharField(
                    max_length=100, verbose_name='codename')),
            ],
            options={
                'ordering': ('content_type__app_label', 'content_type__model', 'codename'),
                'unique_together': {('content_type', 'codename')},
                'verbose_name': 'permission',
                'verbose_name_plural': 'permissions',
            },
            managers=[
                ('objects', django.contrib.auth.models.PermissionManager()),
            ],
        ),
        migrations.CreateModel(
            name='Group',
            fields=[
                ('id', models.AutoField(verbose_name='ID',
                                        serialize=False, auto_created=True, primary_key=True)),
                ('name', models.CharField(unique=True,
                                          max_length=80, verbose_name='name')),
                ('permissions', models.ManyToManyField(
                    to='auth.Permission', verbose_name='permissions', blank=True)),
            ],
            options={
                'verbose_name': 'group',
                'verbose_name_plural': 'groups',
            },
            managers=[
                ('objects', django.contrib.auth.models.GroupManager()),
            ],
        ),
        migrations.CreateModel(
            name='User',
            fields=[
                ('id', models.AutoField(verbose_name='ID',
                                        serialize=False, auto_created=True, primary_key=True)),
                ('password', models.CharField(
                    max_length=128, verbose_name='password')),
                ('last_login', models.DateTimeField(
                    default=timezone.now, verbose_name='last login')),
                ('is_superuser', models.BooleanField(
                    default=False,
                    help_text='Designates that this user has all permissions without explicitly assigning them.',
                    verbose_name='superuser status'
                )),
                ('username', models.CharField(
                    help_text='Required. 30 characters or fewer. Letters, digits and @/./+/-/_ only.', unique=True,
                    max_length=30, verbose_name='username',
                    validators=[validators.UnicodeUsernameValidator()],
                )),
                ('first_name', models.CharField(max_length=30,
                                                verbose_name='first name', blank=True)),
                ('last_name', models.CharField(max_length=30,
                                               verbose_name='last name', blank=True)),
                ('email', models.EmailField(max_length=75,
                                            verbose_name='email address', blank=True)),
                ('is_staff', models.BooleanField(
                    default=False, help_text='Designates whether the user can log into this admin site.',
                    verbose_name='staff status'
                )),
                ('is_active', models.BooleanField(
                    default=True, verbose_name='active', help_text=(
                        'Designates whether this user should be treated as active. Unselect this instead of deleting '
                        'accounts.'
                    )
                )),
                ('date_joined', models.DateTimeField(
                    default=timezone.now, verbose_name='date joined')),
                ('groups', models.ManyToManyField(
                    to='auth.Group', verbose_name='groups', blank=True, related_name='user_set',
                    related_query_name='user', help_text=(
                        'The groups this user belongs to. A user will get all permissions granted to each of their '
                        'groups.'
                    )
                )),
                ('user_permissions', models.ManyToManyField(
                    to='auth.Permission', verbose_name='user permissions', blank=True,
                    help_text='Specific permissions for this user.', related_name='user_set',
                    related_query_name='user')
                 ),
            ],
            options={
                'swappable': 'AUTH_USER_MODEL',
                'verbose_name': 'user',
                'verbose_name_plural': 'users',
            },
            managers=[
                ('objects', django.contrib.auth.models.UserManager()),
            ],
        ),
        migrations.AlterModelManagers(
            name='permission',
            managers=[
                ('objects', django.contrib.auth.models.PermissionManager()),
            ],
        ),
        migrations.AddField(
            model_name='permission',
            name='name_de',
            field=models.CharField(
                max_length=255, null=True, verbose_name='name'),
        ),
        migrations.AddField(
            model_name='permission',
            name='name_en',
            field=models.CharField(
                max_length=255, null=True, verbose_name='name'),
        ),
        .... ETC for other languagues ....
    ]

No need for MIGRATION_MODULES in settings.py.
This is working great for me :)


translation options

please include settings to change translator service to other popular online translators. Yandex is very limited in language support. (very few languages)

@horusra do you have a list of languages that you need?

I mostly need Dutch, and Persian

French & English would be great with Google Translate?

@elmerehbi unfortunately only yandex.translate is supported now, but it has French and English. Google doesn't provide good and free way to have an access to Google translate.
Plugin tries to understand source language and based on source language and your system language guess what is the target language.


CLI configuration command needs to show details and remove the specific heading details only from control plane

The dapr configuration command only shows a few pieces of relevant info from the control plane. This should only show a list of configurations and no particular settings in columns, since what you want to see from apps is different from the control plane.
If you want to see more details then you should also be able to get the specific configure in yaml format with this command
dapr get configuration --name nodeappconfig -o yaml
dapr get configuration -n nodeappconfig -o yaml
Relates to #526

This is what is shown today

hide daprsystem, remove mtls, add metrics column.


Broken objects after trying to import/remove flight model demo objects

On our on-prem S/4 HANA 2023 system, I tried to import the 2023 branch of the flight model demo (https://github.com/SAP-samples/abap-platform-refscen-flight) from an exported zip file, as our system is blocked by the firewall from accessing github.com.
I was using the "1.121.0 - Standalone Version" ABAP Git program that came with the S/4 system.
The initial import did not manage to bring in all the objects, e.g. NRO (same as reported here https://github.com/SAP-samples/abap-platform-refscen-flight/issues/17).
I then tried to Uninstall the package, but this left more object that could not be deleted.
Then, in my desperation, I tried to import the version of the flight model objects that comes with the "ABAP RESTful Application Programming Model" book from SAP PRESS (which is the reason I am importing the flight model in the first place), thinking it will perhaps override the objects, but this seems just to have made matters worse.
I have spent some hours trying to manually delete all the objects created in the /DMO/ namespace, but it seems now I have a number of object entries that are in the state described in note 1156585. Deleting a seemingly empty package is prevented by the assignment of objects which seem to still have a TADIR entry but no actual other presence.
I'm hoping that, in future, abapGit would have some kind of recovery or emergency utility to sort out these inconsistencies or, better yet, prevent them altogether.
A this stage, however, I am a bit wary of using abapGit for large and complex deployments, as I don't think it is mature enough to handle it.

Deleting a seemingly empty package is prevented by the assignment of objects which seem to still have a TADIR entry but no actual other presence

thats SAP behavior defined by SAP, we cannot do anything about that
SAP provides the sflight using latest features, seemingly without testing, I recommend to test things work before putting it into the public
SAP recently released the ABAP cloud trial for 2022, until then the community has only been able to test 1909
as for the sflight, its not needed in the productive system, so I recommend putting it into a non-transported package
Pull requests welcome with fixes

from an abapGit perspective I think we would like to test everything as often as possible, but as open source we dont have any license to run any SAP product and provide services to the internet
also the budget for running it is $0, so the project does rely on users to help testing on different versions

also the budget for running it is $0, so the project does rely on users to help testing on different versions

That is interesting. I was under obviously laboring under the misapprehension that SAP is somehow funding it, at least by employing the maintainers of abapGit to work on it 🤔
Considering they shipped it with our S/4 HANA system, it certainly appears they endorse it. Least they could do is provide some funding!

Anyway, I think I will reach out to the sflight maintainers and explain the predicament. Perhaps they can also provide their releases as transport packages. Thanks!

That is interesting. I was under obviously laboring under the misapprehension that SAP is somehow funding it, at least by employing the maintainers of abapGit to work on it

I dont think SAP has provided a free cup of coffee to any of the maintainers ☕
transport packages: its possible, but doesnt provide transparency as to what is changed and whats part of the transport, and makes it impossible for users to contribute back, so thats also a loose-loose situation IMHO

https://github.com/SAP-samples/abap-platform-refscen-flight/issues/34 opened

I dont think SAP has provided a free cup of coffee to any of the maintainers ☕

Yet their endorsement of abapGit stretches even to officially sanctioned SAP tutorials (like this one: https://developers.sap.com/tutorials/abap-environment-abapgit-transfer.html).
IMO, they are playing a very dangerous game! 🔥🔥🔥

abapGit can be used in mostly any way by everyone, https://github.com/abapGit/abapGit/blob/main/LICENSE

Considering they shipped it with our S/4 HANA system, it certainly appears they endorse it. The least they could do is provide some funding!

They do, but endorse <> support. There was an attempt by SAP to officialise and support it but that failed miserably because they thought it would be a good idea to put it their version into an own namespace that nobody else could modify. I can't remember the name (can't immediately find it, but I vaguely recall something about Aarhus or Aalborg or some Danish city?).
Also, regarding:

this stage, however, I am a bit wary of using abapGit for large and complex deployments, as I don't think it is mature enough to handle it.

Lars already explained the resources the contributors have available (2-3 major versions behind on average), but I'd like to also point out the modular nature of the object de/serialisers.
abapGit consists of a core application and a bunch of distinct modules to handle each object type. Programs and classes have been around for a long time and are pretty stable. RAP objects are new and under active development by SAP. Until a month ago we only had 1909 with limited RAP available. So when SAP release 2023, it is reasonable to expect that objects under active development by SAP may have some differences in the way they are created/updated.
BTW, I tried uninstalling the flight model on 2022 and ran into similar issues in that the RAP stuff didn't uninstall cleanly. I don't know if this is an issue with abapGit or SAP. At first glance it looks like there may be a dependency to something unsupported and the deletion doesn't fail cleanly. I'll try to have a further look into this and raise a more specific issue if I can identify a clear scenario.

(I had another comment I was writing, but I'm glad to only post it now after @pokrakam's comment).
Sorry, no offense meant. I think I should clarify what I mean:
It has been my experience that SAP has a history of taking from the developer community and giving little or nothing back. An example would be that there is, on the one hand, a seeming move to encourage innovation in the developer community, but not providing any means to do so. For instance, for a long time, it was impossible to get a trial ABAP system for personal use and even now it has a very restrictive license when, instead, SAP could choose to make it easy for developers to have a system to play with.
Moreover, while SAP makes some things open source, like UI5, some of the VS Code plugins built around it are not.
The big one of course is the ABAP kernel. Understandably, on one hand, much of their competitive advantage is tied up in ABAP, as is a lot of their intellectual property, so we are unlikely to see that ever.
But SAP, in my opinion, does very little overall to help the developer community, from which they benefit a great deal. That there are people who, in the face of such conditions, are still prepared to build on top of their platforms in an open source manner, is quite mind-boggling to me.
I mean, thank you for all your efforts, and I really appreciate and applaud what you are doing, but it seems kind of insane.

this stage, however, I am a bit wary of using abapGit for large and complex deployments, as I don't think it is mature enough to handle it.

I'm doing this for almost 10 years by now with our 2 products.

~10 clients (read: environments), ECC and S4
One of the products is a complex Tax reporting tool (kind of business critical)
800+ abap object
DDIC+PROG+CLAS+FUGR+XSLT+MIME+BADI obj types (and more)
100k+ lines of code

No major issues so far. Though it was not 100% smooth in early days ;) And I still have some deplyment-centric wishes ... and contributing them ... and planning to continue doing so (though a bit tough time-wise lately).

Found it! It was called Project Odense (hah, I was right about a Danish city, not a complete goldfish memory yet).
It was a nice idea and the blog was full of optimism, but not sure what SAP expected us to do with a locked down tool though. They should have open-sourced it. Perhaps unsurprisingly it has recently been deprecated, which is a shame.

Hi @mydoghasfleas,
beside the issue Provide transport packages as releases #35 that you have created I would suggest that you also file one at https://github.com/SAP-samples/abap-platform-refscen-flight to track the issue the import causes there. Because I think it should be tracked and solved by SAP there.

After eventually managing to remove the objects from the system, I am repeating the process. This time, again, there are issues (the same ones I started with originally) but I will log a separate issue for that.