Support snapshot in rbdplugin

Snapshot is one of the added features in CSI 3.0 spec.
This PR implements functions about the snapshot feature in rbdplugin.
What has been added:

CreateSnapshot:

creates a snapshot with rbd snap create
the snapshot is protected with rbd snap protect for cloning

DeleteSnapshot:

the snapshot is unprotected with rbd snap unprotect
remove snapshot with rbd snap rm

ListSnapshots:

get all snapshots from the cache.
if requested source volume id exists, get all snapshots which have same source volume id
if requested snapshot id exists, get a specific snapshot

Store snapshot information into the persistent file

use the same method like storing volume information but store in the different directory

What has been changed:

CreateVolume:

if this request is for restoring snapshot called by snapshot controller, creates a volume from the snapshot with rbd clone

@sngchlko thanks for getting this done here
@xing-yang can you take a look? thanks

@sngchlko just to confirm, does rbd snapshot and restore work on your krbd test without any special rbd image features?

It will be great if you can test this driver with the snapshot controller PR: https://github.com/kubernetes-csi/external-snapshotter/pull/7

@rootfs I had to use layering feature to create the new volume from the snapshot. I think it will be great to check image features at the time of creating the snapshot.
@xing-yang External-snapshotter sends CreateSnapshotSecrets as nil at the time of creating the snapshot. Actually, I tested with my own implemented csi-snapshot-contoller based on snapshot-controller in kubernetes-incubator. I hope to test on external-snapshotter soon.

@sngchlko thanks!

@sngchlko We need to fix the "CreateSnapshotSecrets as nil" issue in the External-snapshotter PR.  Thanks for your feedback!  Please let us know if you see other issues in that PR.


Prevent unused parameter with none()

Prevent the unused-parameter warning when using none().
This commit uses a void expression to explicitly mark this variable as unused so the compiler does not raise any warning.
Reference:

Stack Overflow: How do I best silence a warning about unused variables?

Thanks for submitting this pull request. I think it might be better to just remove the t parameter, making
auto get(T&& t)  into auto get(T&& )
If you want to do this, I would be happy to accept the pull request.

Well, it took some time but I pushed the updated version.


Shadowing of partially transparent textures does not work correctly for alphaMode: MASK

When using gltf/glb models with partially transparent textures shadowing works when using a material with alphaMode: BLEND, but not when using alphaMode: MASK
The following example is from an x-board model of a tree when using MASK:

The same model with alphaMode BLEND:

The fix would be to always go through the non-opaque path in the code below. The downside is that it'll render the full shader (texturing, lighting, and all). But the pick shaders do something similar for the same reason.
https://github.com/CesiumGS/cesium/blob/48385b800aa87c689e75845191a24b522f3cc930/Source/Scene/ShadowMapShader.js#L95-L106
Longer term we would have different shader variants for pick, shadow, render so that the model shader would apply the apply the base color texture for purposes of using the alpha mask but nothing else.


Updated message display feed when image is missing and updated redirect when a link to a missing file is clicked

Fixes #23739
Updated redirect
When a user uploads a file it has a link as shown below.

If the user that uploaded the file then deletes the file, any user that clicks on the link will have the following displayed:

This has been changed to instead display the following for clarity:

Because we now display a new message when a file doesn't exist, \zerver\tests\test_upload.py needs to be updated to check for the new display message when trying to download a file that is missing.
Updated message display feed
In the message view if a file is missing the inline image thumbnail looks like the following:

When opening the preview the following is displayed:

This has been changed to instead display the following for the thumbnail and preview respectively:

Because now when trying to thumbnail a missing file it instead displays the missing_file.png test_thumbnail.py was updated because for nonexistent files a HttpsResponseForbidden is not returned.

Code review checklist

[x] Self-reviewed the changes for clarity and maintainability

Communicate decisions, questions, and potential concerns.

[x] Explains differences from previous plans (e.g., issue description).
[x] Highlights technical choices and bugs encountered.
[x] Calls out remaining decisions and concerns.
[x] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[x] Visual appearance of the changes.
[x] Responsiveness and internationalization.
[x] Strings and tooltips.
[x] End-to-end functionality of buttons, interactions and flows.
[x] Corner cases, error conditions, and easily imagined bugs.

Thanks! When you're ready, please re-insert and go throw the self-review checklist in the PR description, then post a comment to request a review.

It's also helpful to explicitly mark your PR as a draft when it's not ready for review.

@alya Thank you for the advice! I completed the checklist and my pull request should be ready to review. Also, I apologize for the delayed response.


added parser init for treesitter to nvim plugin

closes #8

I think we should also update the readme, and by the way does this affect the Nix configuration? I don't use nix and I wasn't the one who made it.
(Asking because you seem to also use nix)

I think we should also update the readme, and by the way does this affect the Nix configuration? I don't use nix and I wasn't the one who made it.
(Asking because you seem to also use nix)

Hi,
i updated the readme. The nix code looks fine (i cannot test it, because my nvim config is not nixified). It should run better now, because the initialization was missing there as well.

Ok, we got it :). On the downside we must seperate both configurations to be able to use two different kind of inits. The normal plugin uses the 'normal' way of registering the parser. Nix on the other hand builds the parser and query on its own and registers the build one directly.
Building the flake gives now the following output again.
󱄅 󰎑 󰉋 tree-sitter-hypr at 󰊤 on 󰘬 master !1 is 󰏗 v1.0.0 via  via 󱘗 took 2s
11:23:28 : open result/plugin/init.lua
vim.filetype.add({
  pattern = { [".*/hypr/.*%.conf"] = "hypr" },
})

vim.api.nvim_create_autocmd("FileType", {
  pattern = "hypr",
  callback = function(event) vim.bo[event.buf].commentstring = "# %s" end,
})

vim.treesitter.language.require_language("hypr", "/nix/store/c5sn8bqfh9spipqq6yrac4hl49ldm461-hypr-g
rammar-v1.0.2/parser")

Thanks a lot for your PR!


adding kubernetes cluster installation instructions and clarifications

Submitting this PR per @ryane agreement with me  to document that kubernetes cluster install should be separate cluster than mantl, as otherwise it will brake existing mantl.
I made it as a separate readme doc which is easy to notice when you start aiming for kubernetes installation. Probably  this "should" ?  end up one day in official documentation.
This contribution will also minimize  frequently asked questions on our gitter chat how to setup kubernetes as there is no existing detailed docs on it.
Let me know guys what you think....

If we figure out a good way for playbooks to detect whether or not Mantl has been installed (vis a vis #1084), we should include that in kubernetes.yml and fail when Mesos is already present.

Could this go in docs/ rather than in the root directory? And be linked from index.rst? I'd be happy to merge this then and make edits afterwards. Thanks for the contribution @zoza1982!

@siddharthist I did as proposed.  my file needs Sphinx markup to look pretty.
Let me know if new section in index.rst makes sense.

@zoza1982 It would be best if we could move it to docs/components/kubernetes.rst and convert to rst format, for consistency with our other documentation (if you don't mind). And I'm not sure the new section is necessary, but it's no big deal either way. We can always take it out. Thanks for working on documentation, it's super important!

@siddharthist no I don't mind of course. It would not make sense not doing it. I am just not native to rst format and I have to print some cheat sheet and start doing it by example. I will start working on it.

@siddharthist so lets clarify my intentions:

Rename the file to README.rst and do the Sphinx formatting.
Move it to /roles/kubernetes/README.rst
create file docs/components/kubernetes.rst  with content of  .. include:: ../../roles/kubernetes/README.rst

That how all others are done under components.
Is this sounds good?

@zoza1982 Sounds perfect!

Cool ...I am on it....

@siddharthist  , this is done. Please do quick review and let me know how it looks and if its ready for merge.

@zoza1982 I added a few comments, but this is looking really good so far! I'm excited to have a README for this component. If you're feeling up to it, you could also add a section documenting the available variables available in the kubernetes defaults. See the Consul readme for an example. We can always add that later, as well!

@siddharthist  thanks for feedback.Very appreciated as I am not a big on documentation. First contribution in public so far :-) .  I will get it updated.

@siddharthist  alright ....one more check...I believe we should be good.

@siddharthist I tried to mimic consul variables doc section as much as I could .....but there is not much variables  that you wanna customize in kubernetes defaults as there is in consul.I hope it looks decent for a now.

@siddharthist  are we ready to merge this? Let me know if anything else needs correction. I am fully  open to criticism :-) for the benefit of improvement. Feel free to pitch in more...

@zoza1982 LGTM :+1: It would be awesome if you could rebase this and squash all the commits into one, but nbd either way.

@siddharthist  ....As first time doing rebase...
I think I did what I needed to do...I rebased my kubernetes-setup notes into my master.
And its all now here....  https://github.com/zoza1982/microservices-infrastructure.git and deleted my branch .
I am not sure what I am supposed to do now...do I make new pull request from my master and reference this?

@zoza1982 Here's a good tutorial. What I was talking about in particular is covered under "Squash commits together". I'll just make a pull request into your branch, though, so don't worry about it. Sorry, didn't realize that it would be complicated!


Add support to not destroy xcomponent when parent page navigates away

Unfortunately we are seeing some lag in our Sharepoint environment where we have an xcomponent  rendering. When navigating to a new page we receive a slow response from a server,  can we not destroy the xcomponent immediately as it disappears off view before anything else on the page?

I created a PR for this. You may want to change the prop name.

What do you think about keeping this non-configurable, but only applying it to popups?
The reason this was added in the first place was so popup components would close on page change -- but iframe components automatically close when the page is unloaded, so seems like doing a destroy here isn't really necessary to anyone.
That or we could just make this happen on unload rather than beforeunload so the component doesn't disappear too soon?

That makes sense to me to be non configurable and to not have the logic to destroy the component if it is an iframe.

+1 for only apply close component on page unload when coponent is a popup.
Browsers should take care of closing the iframe.

On a related note, I think making this happen on '''unload''' is better than '''beforeUnload''' . I'm not sure what's the behavior if someone have this in the page that opened the popup to show a navigating away confirmation and user cancels navigation.
'''
window.onbeforeunload = function() {
return true;
};
'"

sorry writing on mobile :(

Yeah it's a fair point. I think it was mainly just me not trusting unload enough to do everything I needed it to before the page unloads.

OK, made this change and published. Please let me know if it works ok for you. For now I've gone with the unload fix rather than branching the logic any further for popups vs iframes.

Is there currently a way to prevent a popup from being destroyed when parent is reloaded? We are trying to open a separate widget in a popup window that needs to persist while the user is navigating through different pages under our domain but as soon as the main (parent) page is reloaded or user clicks in a new link the popup is destroyed.


GSK-2092 Fix LLM test doc to properly render in frontend

Description
Reformatted the docstrings of the LLM tests so that the parsing on the frontend can read the properly and set hints instead of it being shown as a description in the test.

We are standardizing to numpydoc and improving parsing here: #1580 #1581


Wasmtime sets multiple Host header on wasi-http when guest specifies it.

It appears that wasmtime's implementation of outbound calls using wasi-http unconditionally set the Host header here. It then appears to iterate over the headers provided by the guest here leading to there being multiple host headers when the guest automatically sets a host header which is the case for apps built using StarlingMonkey.
This leads to certain servers rejecting the request because of duplicate headers. Should the guest not be setting a header or should wasmtime only set it conditionally?

Wasmtime in theory should be stripping the host header via functionality like this. The issue of setting the host and port should be fixed through https://github.com/bytecodealliance/wasmtime/pull/8563 (which required a follow-up fix at https://github.com/bytecodealliance/wasmtime/pull/8671, but both should be in the 21.0.0 release).
Auditing a bit it looks like any guest-specified header goes through filtering before being inserted into req.headers, so now I'm also a bit confused how a second one could show up there.
Do you have an example guest which runs afoul of the double-host header?

(and/or were y'all testing with 21.0.0? or an older version?)

We were testing with version 18.0.4.

Aha ok makes sense! The Host header was made "forbidden" in https://github.com/bytecodealliance/wasmtime/pull/7905 which was first landed in 19.0.x so I think this might be fixed with a wasmtime upgrade?

I can confirm that this issue has been fixed by upgrading to wasmtime 21.


Update Get-SmbShare.md

Updated Example 2 and 3

Issue#https://github.com/MicrosoftDocs/windows-powershell-docs/issues/1923

Docs Build status updates of commit 6ae4125:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/windows/smbshare/Get-SmbShare.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 09a70e4:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/windows/smbshare/Get-SmbShare.md
:white_check_mark:Succeeded

docset/winserver2012-ps/smbshare/Get-SmbShare.md
:white_check_mark:Succeeded

docset/winserver2012r2-ps/smbshare/Get-SmbShare.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

@dariomws @get-itips @tseward
Peer review needed for this PR. Thanks.

Docs Build status updates of commit 8827bca:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/windows/smbshare/Get-SmbShare.md
:white_check_mark:Succeeded

docset/winserver2012-ps/smbshare/Get-SmbShare.md
:white_check_mark:Succeeded

docset/winserver2012r2-ps/smbshare/Get-SmbShare.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 8f61fbb:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/windows/smbshare/Get-SmbShare.md
:white_check_mark:Succeeded

docset/winserver2012-ps/smbshare/Get-SmbShare.md
:white_check_mark:Succeeded

docset/winserver2012r2-ps/smbshare/Get-SmbShare.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

This also resolves https://github.com/MicrosoftDocs/windows-powershell-docs/issues/1996


Update verbage on autoexpanding archives.

@markjjo We need to update text on Unlimited archives not really be unlimited.  We have gotten feedback that the AuxArchive are limited to 1 TB of storage.  Can we update this document to state:> Office 365 auto-expanding archiving feature provides up to 1 TB of additional storage in archive mailboxes.
We need to get this updated in other articles that have this messaging as well.

@lindabrGithub merged


[RESOLVED]: Sidebar not scrolling upto bottom in mobile devices

Fixes Issue
Closes #2069

Changes proposed
Made modification in the code.

Note to reviewers
@Anmol-Baranwal Can you review the PR.

Can you post a screenshot of your proposed changes @Killer2OP? It'll make it easier for us to review your PR.

Can you post a screenshot of your proposed changes @Killer2OP? It'll make it easier for us to review your PR.

He is trying to fix the issue, like the sidebar is not scrollable to the bottom in mobile size devices, so the below categories are not visible.

Can you approve this @Anmol-Baranwal?

Can you approve this @Anmol-Baranwal?

There is a huge gap in the main section when we scroll.

If that is okay, let me know. I will approve the PR.

@Killer2OP can we fix this gap??

@Killer2OP can we fix this gap??

@rupali-codes So, the thing is that when i viewed in the mobile device the gap is not visible but when we check the code by going in inspect then the gap is visible because the screen is wide enough.


codeunit 1334 "Cancel PstdSalesCrM (Yes/No)" - new integration event in procedure CancelCrMemo

To use a different page in our product knkPublishing we would like to have an additional event:
OnBeforeShowSalesInvDocument
    local procedure CancelCrMemo(var SalesCrMemoHeader: Record "Sales Cr.Memo Header"): Boolean
    var
        SalesInvHeader: Record "Sales Invoice Header";
        CancelledDocument: Record "Cancelled Document";
        CancelPostedSalesCrMemo: Codeunit "Cancel Posted Sales Cr. Memo";
        // local var added
        IsHandled: Boolean;
    begin
        CancelPostedSalesCrMemo.TestCorrectCrMemoIsAllowed(SalesCrMemoHeader);
        if Confirm(CancelPostedCrMemoQst) then
            if CancelPostedSalesCrMemo.CancelPostedCrMemo(SalesCrMemoHeader) then
                if Confirm(OpenPostedInvQst) then begin
                    CancelledDocument.FindSalesCancelledCrMemo(SalesCrMemoHeader."No.");
                    SalesInvHeader.Get(CancelledDocument."Cancelled By Doc. No.");
                    // event call added
                    OnBeforeShowSalesInvDocument(SalesInvHeader, IsHandled);
                    if not IsHandled then
                        PAGE.Run(PAGE::"Posted Sales Invoice", SalesInvHeader);
                    exit(true);
                end;

        exit(false);
    end;

    // new event
    [IntegrationEvent(false, false)]
    local procedure OnBeforeShowSalesInvDocument(var SalesInvoiceHeader: Record "Sales Invoice Header"; var IsHandled: Boolean)
    begin
    end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 46663.


Image upload : sanitize filename ?

Informations
Browser + version: Chrome & Safari, latest version
OS: MacOS Sierra
Resolution: Retina screen (13")
When you upload a file with some special characters, the picture doesn't not show up in Safari, but it works in Chrome.
Is there a way to sanitize filename before sending file to the server ?
How to reproduce the bug?
Upload a picture named "Capture d’écran 2017-04-10 à 15.42.37.png", and it will work on Chrome, but not on Safari.

You should do that server side.

Hm, but what if I'm directly uploading to S3 or another object storage service ? I don't have access to the server side part, and can't implement a sanitize script.
Can't we setup a hook or something in the plugin to rename the file ?

You should make a proxy. Never trust client.


fix: thread leak

Signed-off-by: Sean Tedrow<EMAIL_ADDRESS>Related issue(s):
Fixes #1084
Notes for reviewer:

Checklist

[ ] Documented (Code comments, README, etc.)
[X] Tested (unit, integration, etc.)

cc @steven-sheehy


Add missing ConfigDrive support

The images that we build for SWITCHengines users are currently configured to use only the Ec2 data source.  This is configured (at least on Debian/Ubuntu) in /etc/cloud/cloud.cfg.d/91-dib-cloud-init-datasources.cfg, which includes something like
datasource_list: [  Ec2, None ]

Some users may want to use ConfigDrive as a data source instead.  Possible reasons include paranoia (the unsecured HTTP-over-IPv4 access to the metadata server might be intercepted), or the fact that IPv6-only instances cannot access the metadata server at all.  Therefore it would seem better to change the default to e.g.
datasource_list: [  ConfigDrive, Ec2, None ]

For those users who don't user config drives (that is the vast majority of users now), this shouldn't change anything.  The hope is that cloud-init will quickly and discreetly (e.g. without generating scary error messages) determine that no ConfigDrive source is present, and fall back to the usual Ec2 data source.  That would need to be verified.

There are other possible reasons to use config drives: For example, a user may not trust our network-based (HTTP) metadata service... with some reason, because in the past, we sometimes had trouble with DHCP or metadata servers.  Config drives make them immune against those issues.

Setting the variable DIB_CLOUD_INIT_DATASOURCES="Ec2, ConfigDrive" or DIB_CLOUD_INIT_DATASOURCES="Ec2, ConfigDrive" in the correct env_source files will magically generate the /etc/cloud/cloud.cfg.d/91-dib-cloud-init-datasources.cfg file


[SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

What changes were proposed in this pull request?
Change AvroSuite."Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES" to use episodesAvro, which is loaded as a resource using the classloader, instead of trying to read episodes.avro directly from a relative file path.
Why are the changes needed?
This is the proper way to read resource files, and currently this test will fail when called from my IntelliJ IDE, though it will succeed when called from Maven/sbt, presumably due to different working directory handling.
Does this PR introduce any user-facing change?
No, unit test only.
How was this patch tested?
Previous failure from IntelliJ:
Source 'src/test/resources/episodes.avro' does not exist
java.io.FileNotFoundException: Source 'src/test/resources/episodes.avro' does not exist
	at org.apache.commons.io.FileUtils.checkFileRequirements(FileUtils.java:1405)
	at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1072)
	at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1040)
	at org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34(AvroSuite.scala:397)
	at org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34$adapted(AvroSuite.scala:388)

Now it succeeds.

Test build #134466 has started for PR 31332 at commit 4401517.

Test build #134466 has started for PR 31332 at commit 4401517.

Test build #134466 has finished for PR 31332 at commit 4401517.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134466 has finished for PR 31332 at commit 4401517.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39052/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39052/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134466/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134466/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39052/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39052/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39052/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39052/

@mridulm would you be able to help review this small change?

@mridulm would you be able to help review this small change?

Merged to master, branch-3.1, branch-3.0 and branch-2.4.

Merged to master, branch-3.1, branch-3.0 and branch-2.4.

Thanks @HyukjinKwon !

Thanks @HyukjinKwon !

Good catch on the PR title as well, thank you for fixing 🙏🏼

Good catch on the PR title as well, thank you for fixing 🙏🏼


Hip ra x technical review

Please review this code

Note: this PR could be closed/cancelled since it was superseded by https://github.com/NREL/GEOPHIRES-X/pull/96


[Keyvault] az keyvault delete --hsm-name: Add warning when deleting managed HSM

Description

Customers are not aware that deleted managed HSM still incur billing before getting purged, either manually or at the end of the retention period. Add a warning when deleting managed HSM.
Testing Guide

Create a managed HSM
az keyvault create -g {resource-group-name} --hsm-name {hsm-name} --administrators {oid-of-the-administrator}
Delete a managed HSM, should see a warning.
az keyvault delete --hsm-name {hsm-name}

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

Keyvault


importing users

how to import users to UNA along with their document, photos and and contatd details ?

For such type of import custom coding is required


Questions about the unstable training

Thanks for your great work.
I have a question about "we empirically find using too large kernel size (e.g., 11×11) can lead to unstable training" in Section 5.
What do you mean by unstable training? Does that mean the peformances on ImageNet-1K vary a lot when training mutiple times?

Yes, for the inverted block (4x hidden dim), we find using kernel_size=11 often leads to larger variation, and degraded performance compared to kernel_size=7

Thanks for your explanation.


Site preloader - Vertical tiles

Description
Create the site preloader component as shown in the video attached below. Please note that you will need to build the loader (vertical tiles), not the actual content that is shown in the video. Make sure that the loader is fully responsive
Animation Preview

Screenshot/Video:

https://github.com/user-attachments/assets/add2c6d1-ae0f-4c0f-9137-bea5794420f5

[!NOTE]
Please refer to the Animata contributing guidelines available here for rules on how to contribute to the project. Let us know in the comments if you’re working on this issue, and feel free to ask any questions!

Requirements

Create a new animated component that matches the provided design.
Add example usage of the component in the documentation or storybook.

 Guidelines & Best Practices

Write reusable and clean code following the project’s style guide.
The component should be fully customizable. The user should be able to modify props such as duration, easing, color, and size where applicable.
Follow the Contributing Guidelines before submitting your pull request.

[!IMPORTANT]
To ensure more contributors have the opportunity to participate, we kindly request that:

Each contributor submits only one pull request related to this issue.
If you're interested in working on this issue, please comment below. We will assign the issue on a first-come, first-served basis.

Additional Resources
Referenced from: www.steviaplease.me

I'd like to volunteer. Please assign me this feat. @sudhashrestha

I wanted to confirm the desired file structure for this feature. Based on the initial description, I'm assuming it should look something like this:
animata/
    background/
    bento/
    ...
    preloader/
        vertical-tiles.tsx

Could you please confirm if this is correct? Specifically, I need to understand where the vertical-tiles.tsx preloader should be placed.
Thanks in advance for the clarification! 😊

Hi @SatyamVyas04 You can use our script yarn animata:new to create a new component. This will automatically handle which folder the component should be put into. If there's no type called "preloader" you can create one. For more info: https://animata.design/docs/contributing/components


Expand the dataset to include a wider variety of urls and labels

Context
A wider variety of URLs and the record type labels associated with them are needed to facilitate Hugging Face model training.
Requirements

More URLs need to be manually labeled, especially those associated with under-utilized labels.

Under-utilized labels

Accident Reports
Booking Reports
Budgets & Finances
Car GPS
Citations
Dispatch Recordings
Field Contacts
Inmate Records
List of Data Sources
Sex Offender Registry
Stops
Use of Force Reports
Vehicle Pursuits
Wanted Persons

@josh-chamberlain @EvilDrPurple What's the procedure for obtaining additional URLs as-is? Are there specific sources or websites where these types of records are more likely to be found? Do we have a way of finding them automatically, or is it done manually?

I believe I answered here: https://github.com/Police-Data-Accessibility-Project/data-source-identification/issues/16#issuecomment-1964651708

Current thought on this is to try creating batches with keywords in #88


Faker is already import but may be a comment as indication or a mention in the doc with a link to faker documentation’s will be useful.

Improved doc


copy Namespace atomically in WithConstLabels

Copy with * is racy
Signed-off-by: Alexander Morozov<EMAIL_ADDRESS>

LGTM

LGTM


Maintain aspect ratio on resizing?

Are there any plans to add this functionality anytime soon? Possibly as a prop boolean value or another common experience is when users hold the shift key on resize to maintain the ratio.

Yes I'm planning to add this functionality, sometimes in the future. However I really cannot make estimates at this point.

Any news on this feature?  This would be REALLY helpful :)

Think this feature is mandatory. Would love to see it being added, so we can use this great module.

+1

Just giving my support on this feature

#97 Maybe some of you guys wanna try my pull request?

Aspect ratio functionality is available with the new major version on the dev branch. Right now it's in beta, but hopefully this will get released soon. It just need a little bit more of testing.

Will this get merged in soon? Waiting for this :P

@mauricius  when aspect ratio get released? look forward to new features!

Use v2 from dev branch.
npm install https://github.com/mauricius/vue-draggable-resizable#develop --save
Then in your main.js:
import VueDraggableResizable from 'vue-draggable-resizable/src/index'
import 'vue-draggable-resizable/src/components/vue-draggable-resizable.css'
Vue.component('vue-draggable-resizable', VueDraggableResizable)

LockAspectRation


New Formula: allegro 5.2.0

What's the justification here? It seems to be a newer version of the one in homebrew/core?

It may be as you say. Ver.5.2 release news say

Allegro 5.2 is source but not binary compatible with 5.0.
Allegro 5.2 is developed a little differently than 5.0 and 5.1 were.

If you close this, I will move to homebrew/core.

Can you elaborate on both those points? Is this a development release or a stable release that's not backwards compatible with the 5.0.11 branch?

Ver.5.2 release news say

After many years of development in the unstable branch, the next stable version of Allegro is ready for release! This version brings most of the features from the unstable branch, but now with a guarantee of backwards source and ABI compatibility. Allegro 5.2 is source but not binary compatible with 5.0.
Allegro 5.2 is developed a little differently than 5.0 and 5.1 were. This version combines both unstable and stable APIs in one branch. The unstable branch is no more! Unstable APIs may be changed between versions, or even removed entirely.

They say this is stable and removed unstable code of ver.5.1 which has stopped to develop and unreleased in the end. 5.2 may be compatible with 5.0 but not 5.1. It can't be judged about incompatible with 5.1.
Current allegro.rb has devel part for 5.1. It may be self-responsibility to use devel. But there was the reason that 5.1 supported theora and libogg.

Nothing in Homebrew depends on allergro so this should just replace the existing Homebrew allegro formula.

I move this to homebrew-core. https://github.com/Homebrew/homebrew-core/pull/375


MaPLE zero shot ability

Thank you very much for your tremendous contribution. As a beginner, I am not very familiar with some aspects and I greatly need your help. I hope you have time to reply to me. Thank you!
For the clip model, I can directly use its zero shot ability to implement classification tasks on my own dataset. I would like to ask if it is also possible to directly use the trained MaPLE and then use the zero shot ability on my own dataset? What operations are required?
1.How do I need to train MaPLE, what changes do I need to make, and on what dataset do I need to train?
2. Which files do I need to use?

Hi @zbb1111,
Thank you for showing interest in MaPLe !
Regarding your queries, please note the following:

I would like to ask if it is also possible to directly use the trained MaPLE and then use the zero shot ability on my own dataset?
Yes you can use MaPLe to directly evaluate it on your dataset. You will only need to set up your own dataset first so that the repository can load your dataset. Please checkout previous closed issues in how you can define your own dataset in this repository.
After that, you can simply use cross-dataset evaluation instructions (in RUN.md) to evaluate trained MaPLe on your own dataset.

How do I need to train MaPLE, what changes do I need to make, and on what dataset do I need to train?
Please refer to the instructions provided at RUN.md on training MaPLe.

Which files do I need to use?
Again, all instruction have been provided in RUN.md.

Let us know if you still encounter any issue.
Thank you and kind regards!

Hi @zbb1111,
I am closing the issue now.
Feel free to reopen it if there are still any queries.


Bump debian golang image

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve

/lgtm


Sentence-transformer library not found by Nuitka executable

Before submitting an Issue, please review the
Issue Guidelines.

Please check whether the bug was already reported or fixed.

Please check out if the develop version of Nuitka works better for you.
Download source, packages from here where you will also
find instructions how to do it via pip or git.

If you want to post a problem/bug, to help us understand and resolve your issue please check that
you have provided at least the information below, and discard up to here:

Nuitka version, full Python version, flavor, OS, etc. as output by this exact command.

python -m nuitka --version

If you are not providing the full output. The issue cannot be solved and will be considered
invalid. The command outputs more than you think, and we are adding more all the time. We do not
want to left guessing or trying things out to reproduce your issue.

How did you install Nuitka and Python
Did you use pip, anaconda, deb, rpm, from source, git clone, then install into a virtualenv or
not, this is very important usually and one thing, the above command does not tell us (yet).

The specific PyPI names and versions
It should be taken from this output if there specific packages involved, e.g. numpy, you are
expected to shorten this to the relevant ones.

python -m pip list -v

Many times when you get an error from Nuitka, your setup may be special
Then even a print("hello world") program will not work, please try that and report that error
instead. We do not need a report that says "Tensorflow does not work" when nothing actually works
for you. You ought to also read the User Manual and check that your setup is actually supported
already.

Also supply a Short, Self Contained, Correct, Example
That demonstrates the issue i.e a small piece of code which reproduces the issue and can be run
with out any other (or as few as possible) external dependencies. Issues without this may will
likely be rejected without much consideration. Often this can be as simple as importing a package,
if this is a packaging issue, try that first.
Pointers to repositories with usage of pipenv are very welcome, archives with examples are too,
e.g. if a certain package structure is needed. This must be only source code, binaries are not
used ever by us. But beware, that e.g. we cannot click around and stuff. Still do an effort to
make the usage obvious. Having a compile script in the repo is perfect.
But this cannot be much code for non-commercial users, since that causes too much effort. We
cannot just compile your code, run it and have it download a control and command center.

Provide in your issue the Nuitka options used
Ideally use the # nuitka-project: options feature in the code, so options and example code go
along. Alternatively state the command line.
Nuitka Options in the code

Avoid unnecessary options
Do not use --deployment as it disables all potential bug catching, we are not here to replace
what it does, remove that option the moment you face an issue and report only then.
Do not use --onefile if the issue also happens with --standalone. Minimize the use of options
as far as you can, please. Do not disable outputs with --quiet and do not disable warnings.

Note if this is a regression
If it used to work in an earlier version of Nuitka, please note what you know about that. Since
git bisect is a thing for which we do a lot of error to make it usable, this will help
dramatically to isolate the issue.

Consider getting commercial support
Nuitka commercial offers subscriptions and priority
support. This will accelerate your problem solution and helps to sustain Nuitka development. Also
you then have a chance to provide source code of your project to us, which might simplify things,
or pay for time to solve your issues in your environment.

Some things are not welcome, please consider it.

Do not post screenshots
These are not welcome unless absolutely necessary, e.g. because of Qt display problem, instead
capture the output of programs, so things are searchable and copy&paste will work. I just plainly
don't want to manually copy strings and hope they match.

Do not close the issue yourself, we will close things on stable releases
We close issues only when they are released as a stable version, e.g. in a hotfix or a new
release, before that it will be "Done" in planning and go through factory and develop tags to
indicate they are solved there.
Of course, if you find out your issue is invalid, please do close it, and we then attach the
invalid tag.

Do not report against factory version
Unless you were asked to test it there, it is frequently very broken, and there is only noise to
be had. Telling me about it on Discord would be a better idea.

Do not let this template remain part of the issue, it's noise.

Hi,
I am trying to do RAG using langchain. I used the following libraries:
from langchain.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import TextLoader
from langchain.prompts import ChatPromptTemplate
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain.chains import create_retrieval_chain
The environment is same as above. But when I run the executable, it shows the following error:
main.dist/main.bin --gen_code --file_name ./data/query.txt
Error: Could not import sentence_transformers python package. Please install it with pip install sentence-transformers.
However, sentence-transformer is already present in the environment. Is there an way to overcome this error?
Following are my pip installs in my env:
certifi==2024.8.30
charset-normalizer==3.4.0
filelock==3.16.1
fsspec==2024.10.0
huggingface-hub==0.26.2
idna==3.10
Jinja2==3.1.4
joblib==1.4.2
MarkupSafe==3.0.2
mpmath==1.3.0
networkx==3.4.2
Nuitka==2.4.11
numpy==2.1.2
nvidia-cublas-cu12==<IP_ADDRESS>
nvidia-cuda-cupti-cu12==12.4.127
nvidia-cuda-nvrtc-cu12==12.4.127
nvidia-cuda-runtime-cu12==12.4.127
nvidia-cudnn-cu12==<IP_ADDRESS>
nvidia-cufft-cu12==<IP_ADDRESS>
nvidia-curand-cu12==<IP_ADDRESS>
nvidia-cusolver-cu12==<IP_ADDRESS>
nvidia-cusparse-cu12==<IP_ADDRESS>
nvidia-nccl-cu12==2.21.5
nvidia-nvjitlink-cu12==12.4.127
nvidia-nvtx-cu12==12.4.127
ordered-set==4.1.0
packaging==24.1
pillow==11.0.0
PyYAML==6.0.2
regex==2024.9.11
requests==2.32.3
safetensors==0.4.5
scikit-learn==1.5.2
scipy==1.14.1
sentence-transformers==3.2.1
setuptools==75.2.0
sympy==1.13.1
threadpoolctl==3.5.0
tokenizers==0.20.1
torch==2.5.0
tqdm==4.66.6
transformers==4.46.0
triton==3.1.0
typing_extensions==4.12.2
urllib3==2.2.3
zstandard==0.23.0
~Pingu

@kayhayen please habe a look into the issue.


dropTarget is null error when dragging & dropping class

Steps to Reproduce:

Increase level dropdown to higher level
Exit advancement window, canceling the process
Click and drag the class just changed, dropping back onto the same character sheet

I'm not able to reproduce this in lastest v10-dev on core build 279.

Seems to have been fixed by a change in core. Good to see :).


[Docs] Update layout docs to add coloured images to clarify the grid

Description
Also added alt text. Hopefully this is valuable for scraping @Anna-Xiong
Built docs are here
@huong-li-nguyen I'm sorry about the colour choices but it allows differentiation in the grid layouts.
This was way too time consuming. I hope it's OK. 🤞
Screenshot
Notice

[x] I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":

I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.
I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorized to submit this contribution on behalf of the original creator(s) or their licensees.
I certify that the use of this contribution as authorized by the Apache 2.0 license does not violate the intellectual property rights of anyone else.
I have not referenced individuals, products or companies in any commits, directly or indirectly.
I have not added data or restricted code in any commits, directly or indirectly.

Thanks @huong-li-nguyen @petar-qb that makes sense. I'll make another ticket to change the colors. It took most of the learning afternoon on Friday to do all this so it's a bit of a task, but replacing the graphics should be swifter than making them from scratch as the markdown changes are done already.


Decrease distribution size

At the moment, wheels are about 11 MB in size. Uncompressing/installing the data takes about 36 MB. This appears to be quite large and might be reduced:

The serialized JSON data currently does not use the most compact representation where there is no whitespace after the comma and colon.
The word lists have lots of duplicate entries (see #21 as well).
About half of the (uncompressed) resources in the wheel file is test data (18.5 MB), which is a lot and is not required by most of the users.

@goneall I stumbled upon this repository when skimming through the parsing/matching docs (as you might have seen in the specification repository as well) and looking for existing implementations. When evaluating the general code quality, I opened some of the immediate issues I got aware of.
For now, the overall impression is not the best, as there is not really any development besides the GSoC stuff. Resources are part of an external personal repository without any indication on how to generate them (https://github.com/m1kit/yalm-resources, #20), it uses a library without any self-tests (https://github.com/m1kit/terregex) and so on.
What is the purpose/aim of the SPDX team with this repository? Should it become the reference implementation for the matching algorithm in Python as the https://github.com/spdx/tools-python repository does for the SPDX documents itself?
That being said, I will see if I manage to get some PR done to start removing the temporary stuff from this repository which does not belong here.

Thanks @stefan6419846 for looking into this.
In terms of purpose, I only know of one implementation of the full SPDX license matching algorithm which is written in Java.
Having a Python implementation as a library would be very beneficial to the community.  We're currently using a Python Java Bridge in the SPDX online tools which has proven to be rather problematic - I'd love to replace that with a native Python implementation.
One suggestion before we invest too much time in this library, we should test that it properly matches the text files in the SPDX license list as a test just to make sure there isn't any significant design issues.  This is documented in issue #14


[Bug]: getDataValidation() doesnt work on export with XLXS, but works in XLS

Is the bug applicable and reproducable to the latest version of the package and hasn't it been reported before?

[X] Yes, it's still reproducable

What version of Laravel Excel are you using?
3.1
What version of Laravel are you using?
8
What version of PHP are you using?
7.3
Describe your issue
getDataValidation() doesnt work on export with XLXS, but works in XLS
$spreadsheet = $event->sheet;
$validation = $spreadsheet->getCell('K2')->getDataValidation();
$validation->setType(DataValidation::TYPE_LIST );
$validation->setErrorStyle(DataValidation::STYLE_INFORMATION );
$validation->setAllowBlank(false);
$validation->setShowInputMessage(true);
$validation->setShowErrorMessage(true);
$validation->setShowDropDown(true);
$validation->setErrorTitle('Input error');
$validation->setError('Value is not in list.');
$validation->setPromptTitle('Pick from list');
$validation->setPrompt('Please pick a value from the drop-down list.');
$validation->setFormula1(''Categories'!$A$2:$A$9');
$spreadsheet->setCellValue("N2", '=VLOOKUP(K2,Categories!$A$2:$B$9,2,0)');
It worked when i export it with
How can the issue be reproduced?
when using this line of code
$validation = $spreadsheet->getCell('K2')->getDataValidation();
What should be the expected behaviour?
Excel completed file level validation and repair. Some parts of this workbook may have been repaired or discarded.

Please report to PhpSpreadsheet


hack-manager.js sleep loop?

When I use hack-manager.js it fills up the queues in the Active Scripts tab.  But then, after a while, it seems to "finish" and only the original hack-manager.js is running with no child scripts on other servers.
Would appreciate if someone could at least explain what might be going on here.  Is this likely because of low server money?  High security?  What gives.
TIA

Could you please send the logs of the script when it enters this inactive state?

Of course, now that I post about it, it doesn't go into the inactive state.
From what I remember, beccause I /did/ check the logs, it would do an available ram check then sleep and just cycle through all the servers doing that: ram check; sleep, etc.  I will definitely post the logs when/if it goes back into that state.

This is what the log looks like during the inactive phase.  I don't know if there's a way to turn up the verbosity.
sleep: Sleeping for 1 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
sleep: Sleeping for 1 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
sleep: Sleeping for 1 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
sleep: Sleeping for 1 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
sleep: Sleeping for 1 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
sleep: Sleeping for 1 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
sleep: Sleeping for 1 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
sleep: Sleeping for 1 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
sleep: Sleeping for 1 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
sleep: Sleeping for 1 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 16.40GB
sleep: Sleeping for 1 milliseconds

I am running into the same issue following the string below.
exec: Could not find script 'targeted-grow.js' on 'nectar-net'
sleep: Sleeping for 10 milliseconds
getServerMaxRam: returned 8.00GB
getServerUsedRam: returned 0.00B
exec: Could not find script 'targeted-grow.js' on 'CSEC'
sleep: Sleeping for 10 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 0.00B
exec: Could not find script 'targeted-grow.js' on 'neo-net'
sleep: Sleeping for 10 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 0.00B
exec: Could not find script 'targeted-grow.js' on 'iron-gym'
sleep: Sleeping for 10 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 0.00B
exec: Could not find script 'targeted-grow.js' on 'zer0'
sleep: Sleeping for 10 milliseconds
sleep: Sleeping for 82944.8275862071 milliseconds

I am running into the same issue following the string below.
exec: Could not find script 'targeted-grow.js' on 'nectar-net'
sleep: Sleeping for 10 milliseconds
getServerMaxRam: returned 8.00GB
getServerUsedRam: returned 0.00B
exec: Could not find script 'targeted-grow.js' on 'CSEC'
sleep: Sleeping for 10 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 0.00B
exec: Could not find script 'targeted-grow.js' on 'neo-net'
sleep: Sleeping for 10 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 0.00B
exec: Could not find script 'targeted-grow.js' on 'iron-gym'
sleep: Sleeping for 10 milliseconds
getServerMaxRam: returned 32.00GB
getServerUsedRam: returned 0.00B
exec: Could not find script 'targeted-grow.js' on 'zer0'
sleep: Sleeping for 10 milliseconds
sleep: Sleeping for 82944.8275862071 milliseconds

Change all the ns.scp "await ns.scp('targeted-weaken.js', "home" , server)"
to "await ns.scp('targeted-weaken.js', server , 'home')"

The last two replies are for a separate problem. The scp parameter order is switched, and
grow_threads = Math.floor(grow_threads); weaken_threads = Math.floor(weaken_threads); hack_threads = Math.floor(hack_threads);
Needs to be added at line 430, just before the final 3 ns.exec() to ensure threads are integers and not floats.
But having done that, I'm still getting an infinite sleep loop like the one d4t4king posted. Repeatedly showing
sleep: Sleeping for 1 milliseconds getServerMaxRam: returned 2.05TB getServerUsedRam: returned 1.45TB getServerMaxRam: returned 2.05TB getServerUsedRam: returned 1.45TB getServerMaxRam: returned 2.05TB getServerUsedRam: returned 1.45TB sleep: Sleeping for 1 milliseconds getServerMaxRam: returned 2.05TB getServerUsedRam: returned 1.45TB getServerMaxRam: returned 2.05TB getServerUsedRam: returned 1.45TB getServerMaxRam: returned 2.05TB getServerUsedRam: returned 1.45TB sleep: Sleeping for 1 milliseconds
and never refills the queues. It sat for at least 10 minutes. It works if it's killed and re-run.

I couldn't find where these calls were coming from, but I inserted ns.print()s and found theyre at
194
getServerMaxRam: returned 2.05TB
getServerUsedRam: returned 897.60GB
211
getServerMaxRam: returned 2.05TB
getServerUsedRam: returned 897.60GB
sleep: Sleeping for 1 milliseconds
getServerMaxRam: returned 2.05TB
getServerUsedRam: returned 897.60GB
172
194
getServerMaxRam: returned 2.05TB
getServerUsedRam: returned 897.60GB
211
getServerMaxRam: returned 2.05TB
getServerUsedRam: returned 897.60GB
sleep: Sleeping for 1 milliseconds```

Solved it. Need to apple Math.floor() to hack_threads, weaken_threads, and grow_threads at line 173, before the first if in the whileloop. If you dont, they can be an arbitrarily small number like 0.08375643, which is too small for a single thread, and itll catch you in the loop. and since youve used up as much space as possible in the first iteration, you always get a number <1.

Just looked at the problem, I still had the issue of the program geting stuck in a sleeping loop. From what I have found out the problem is that the loop doesn't take into consideration the amount of RAM you have reserved when calling the script (I tried reserving 50 GB of ram for running other minor scripts, and thus run into this problem). By changing the while loop around line 173 to take into consideration the actual amount of available RAM on the home server should fix the problem.

Solved it. Need to apple Math.floor() to hack_threads, weaken_threads, and grow_threads at line 173, before the first if in the whileloop. If you dont, they can be an arbitrarily small number like 0.08375643, which is too small for a single thread, and itll catch you in the loop. and since youve used up as much space as possible in the first iteration, you always get a number <1.


Request for Dockerfile or Prebuilt Docker Container

Hi,
I’m using the DeepSeek-V3-Base model and have set up everything manually so far. However, it would be much more convenient if there were an official Dockerfile or prebuilt Docker container available for easier integration.
Does an official Docker setup for this model exist? If not, would it be possible to provide one?
A prebuilt container bundling the model and its dependencies would greatly simplify the deployment process for users.
Thank you in advance for your help!
Looking forward to your response.
Best regards,
Emilio Frittrang
Are there any specific optimizations or configurations required to efficiently run the DeepSeek-V3-Base model in a Docker environment?
Base image
FROM python:3.10-slim
Install git and clean up temporary files
RUN apt-get update && apt-get install -y git && apt-get clean && rm -rf /var/lib/apt/lists/*
Set working directory
WORKDIR /app
Clone the repository and install dependencies
RUN git clone https://github.com/deepseek-ai/DeepSeek-V3.git . &&
pip install --no-cache-dir -r inference/requirements.txt
Default command to run the model
CMD ["python", "inference/generate.py"]

Actually the infer demo is slow but the compatibility is good because we built the demo entirely using pytorch and triton code. No special configuration is required.

For actual deployment, you can use the Docker files provided by sglang, lmdeploy, or vllm, among others. The community support for us is also quite robust.


Fixing up shebang detection to match new tokenizer behaviour

Since https://github.com/github/linguist/pull/1604 the build has been broken :crying_cat_face:
The change to the tokenizer behaviour here and the new sample file with a shebang that looked like this:
#! /usr/bin/env A=003 B=149 C=150 D=xzd E=base64 F=tar G=gz H=head I=tail sh

meant that the shebang extraction in [shebang.rb] (https://github.com/github/linguist/blob/master/lib/linguist/shebang.rb#L20-L51) was returning A=003 as the interpreter (when it should be sh).
This pull request brings the shebang detection in tokenizer.rb and shebang.rb into parity to fix the current build failure. Given how similar these methods are they should basically be the same. I'm saving that for another day.

/ cc @bkeepers for :eyes:

TIL StringScanner
:+1: if you add an explicit test.


Dot Matrix Fonts and Tesseract's Connected Component Analysis

Tesseract Version:
tesseract 4.0.0-beta.1
leptonica-1.75.3
libgif 5.1.4 : libjpeg 8d (libjpeg-turbo 1.5.2) : libpng 1.6.34 : libtiff 4.0.9 : zlib 1.2.11 : libwebp 0.6.1 : libopenjp2 2.3.0
Found AVX512BW
Found AVX512F
Found AVX2
Found AVX
Found SSE
Platform: uname -a output: Linux CUDA1 4.15.0-29-generic #31~16.04.1-Ubuntu SMP Wed Jul 18 08:54:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"
Current Behavior:  I am trying to fine-tune Tesseract for dot-matrix fonts such as that in the attached picture.

When the dots are closely spaced together and touch, Tesseract can more or less handle the dot-matrix font with some fine-tuning and image processing.  However, when the dots do not touch, as in the picture, Tesseract struggles.
I read in An Overview of the Tesseract OCR Engine that the first step in Tesseract's processing pipeline is a connected component analysis (second paragraph of Section 2).  Since the letters in a dot-matrix font do not form connected components, I am wondering if Tesseract's connected component analysis may be one reason that Tesseract struggles on the image attached.
Is there a command to see how Tesseract performs connected component analysis on this image?
Expected Behavior:  Perform better on dot-matrix fonts where the dots are spaced far apart
Suggested Fix:  Understand if connected component analysis fails.

I read in An Overview of the Tesseract OCR Engine that the first step in Tesseract's processing pipeline is a connected component analysis (second paragraph of Section 2).

That applies to base/legacy tesseract (3.0x). Tesseract4 is neural net/LSTM based though it still supports the legacy models.
I tried some finetuning using dot matrix fonts. While the image you gave is not recognized correctly, if split in separate lines the results are better, but only with --psm 8 (single word).
/home/ubuntu/TEST/ex_20_1 dotslayer
LOT#S18FO70

/home/ubuntu/TEST/ex_20_2 dotslayer
EXP:09/2020

The traineddata file is available at https://github.com/Shreeshrii/tessdata_shreetest/blob/master/dotslayer.traineddata.
Note: I had also replied in forum with the same msg and files, maybe it didn't go thru because of attachments.

@ameera3 has shared her completed project and traineddata files - see https://github.com/ameera3/OCR_Expiration_Date/tree/master/Custom_Dot_Matrix_Dataset/Dot_Matrix_Test_1/0


Fix rustdoc error in Triangle.

Commit 4ffa010f99a38ebd6d7ac763355403e98454fc09 introduced a method that is only available with feature = "dim3", but a method that is present when feature = "dim2" was linking to it. The doc commnets referencing it must be protected by cfg_attr checks.

@Vrixyz This is needed so that one of #224 or #227 can land.

Thank you for this PR! The issue here is that the Triangle::normal and ::scaled_normal method should only be available in 3D too.

@sebcrozet Well, that's a ton easier than what I did. :)


images are not uploaded in default image widget

Describe the bug

I am trying to upload an image from the admin panel. It shows the dialogue to upload the image but it just shows Loading.. text. I have also tried to push buggy code to Github and tried to upload the image from the published website but it doesn't work.
To Reproduce
Steps to reproduce the behavior. For example:

Go to admin panel and create a new blog(I am using gatsby-starter-hello-world and netlify-cms)
Click on create blog
Scroll down to Cover field and try to upload an image
See following error in the console-

netlify-cms-app.js:28 API_ERROR: Not Found
    at new t (http://localhost:8000/admin/netlify-cms-app.js:60:77455)
    at new o (http://localhost:8000/admin/netlify-cms-app.js:60:77821)
    at u.handleRequestError (http://localhost:8000/admin/netlify-cms-app.js:98:139449)
    at http://localhost:8000/admin/netlify-cms-app.js:40:152654
    at async Promise.all (index 0)
(anonymous) @ netlify-cms-app.js:28

Uncaught (in promise) TypeError: Cannot read property 'find' of undefined
    at netlify-cms-app.js:28
    at netlify-cms-app.js:60
    at netlify-cms-app.js:28
    at netlify-cms-app.js:91
    at Object.Bi (react-dom.production.min.js:175)
    at Ai (react-dom.production.min.js:14)
    at Di (react-dom.production.min.js:14)
    at wf (react-dom.production.min.js:14)
    at Ei (react-dom.production.min.js:176)
    at zd (react-dom.production.min.js:15)

Expected behavior

I should be able to select and upload an image from the image widget. After uploading the image, it should be stored in the folder specified as media_folder.
Screenshots

Applicable Versions:

Netlify CMS version: "netlify-cms-app": "^2.10.1",
Git provider: Github
OS: Ubuntu 18.04.3 LTS x86_64
Browser version: Google Chrome | 79.0.3945.88

Node.JS version:
I am using yarn.
yarn: 1.21.1
CMS configuration

# backend:
#   name: github
#   repo: aditya81070/gatsby-with-netlify-cms-learn
backend:
  name: git-gateway
# media_folder: static/assets
publish_mode: editorial_workflow
media_folder: static/images/uploads
# public_folder: assets
# media_library:
#   name: uploadcare
#   config:
#     publicKey: 46afd475684287af5b68
#     imagesOnly: true
#     imageShrink: 480x480
collections:
  - name: blog
    label: Blog
    folder: blog
    create: true
    fields:
      - { name: path, label: Path }
      - { name: date, label: Date, widget: datetime }
      - { name: title, label: Title }
      - { name: body, label: Body, widget: markdown }
      - { name: cover, label: Cover Image, widget: image }

Additional context

I tried to find a solution to this problem in previous issues. Some issues advised enabling git-gateway but I already enabled it.
Additionally, I tried to use Uploadcare to store my images/files and it worked perfectly. But I want to store my images in my repository.

API_ERROR: not found usually means that the cms can’t find the media folder.

@tomrutgers do I have to create media folder explicitly? I used cms to store my blogs and folder created automatically. The error I am getting is to only store images.

@tomrutgers you were right. Creating a folder and pushing it to GitHub solved issue. Thanks mate :)

@tomrutgers  but the media modal should still load properly even if the cms can't find the media folder.

@barthc definitely, just clarifying the cause of the error here

@tomrutgers you just saved me. I've been banging my head on the wall for several days now.  Thank @aditya81070 for posting this issue here.

@barthc Media modal was loading properly. There was no issue for that. @tomrutgers I still have one thought that if cms can create a custom folder for collections, why it can not do the same things while uploading media?


Field level streaming

This is a WIP for adding partial field-level streaming similar to what exists in the instructor-js library.
Currently Partial() fails when encountering fields with generic types like List[User]. This means you can do
class User(BaseModel):
    name: str
    age: str

return client.chat.completions.create(..., response_model=Iterable[Partial(User)], stream=True)

But not this at the moment
class Users(BaseModel):
    users: List[User]
    comments: str

return client.chat.completions.create(..., response_model=Partial(Users), stream=True)

Adding for direct reference in the comment chain

I could use some help figuring out how to deal with generics when constructing the Partial model or if there's an easier way to accomplish this. I've also temporarily hijacked multitask.tasks_from_chunks to show how the partial json parsing will look like.

[x] Support generics in Partial / find a better way to take existing model and make all its fields Optional
[x] Refactor partial json parsing
[x] Dedupe partial models to only yield when there are differences

can you add the gif you had before?

Adding for direct reference in the comment chain

Streaming in action:

can you try to fix the test while i change the example

How does the doc look @jxnl? Also still can't make sense of how the tests pass locally but fail here

looks good @shanktt do you think you're able to benchmark the raw stream vs the this partial json thing?
I'm curious what the additional overhead is or if the tokens per second >> json parse.

@shanktt  Thanks for the awesome work!
I am trying to implement in into my project of fastapi endpoint + nextjs 14 frontend
Is there an official doc for this change for reference?

@thisiskeithkwan These should work until the pr gets merged (still in-progress so could change):
partial_streaming.py
field_streaming.md

@jxnl Let me know if the benchmark looks good to you.

Got the benchmark and graph streaming fixed up. Do you still want to mock the streamed response and just benchmark the partialjson function like you suggested on discord?

Got the benchmark and graph streaming fixed up. Do you still want to mock the streamed response and just benchmark the partialjson function like you suggested on discord?

prosb easier to figure out the actual latency costs! Then we can merge once we add a note thats like "this adds about 5ms per token" (i dont think its that high

This looks very cool, any examples on how to hook up a JS to read the stream?

This looks very cool, any examples on how to hook up a JS to read the stream?

Yo, I need this too

@alew3 @thisiskeithkwan any reason you won't want to use the js library in general?

Because harder to pair with other reranker API in js ?


Memory leak in string interop

TorchSharp uses PinnedArray for string interop, and it does not release the handles explicitly and wishes PinnedArray to release the IntPtr handles when disposing. However, PinnedArray auto-release needs IDisposable to work with, which IntPtr does not implement. I suggest using a dedicated PinnedPtrArray, which is able to release all the handles properly.

@zgxnet -- thank you for noticing this problem. I will take a look at it and see what we can do. You're also welcome to submit a PR with a fix.

It seems that the problem is not fixed.

You are right -- it was linked to a PR that I had first imagined would take care of it, but with numerous process crashes, it turned into a much smaller change. The problem remains.


HDFS-17250. EditLogTailer#triggerActiveLogRoll should handle thread Interrupted

Description of PR
https://issues.apache.org/jira/browse/HDFS-17250
issue:
When the NameNode attempts to trigger a log roll and the cachedActiveProxy is a "the machine has been shut down of the namenode" it is unable to establish a network connection. This results in a timeout during the socket connection phase, which has a set timeout of 90 seconds. Since the asynchronous call for "Triggering log roll" has a waiting time of 60 seconds, it triggers a timeout and initiates a "cancel" operation, causing the executing thread to receive an "Interrupted" signal and throwing a "java.io.InterruptedIOException" exception.
Currently, the logic not to handle interrupted signal, and the "getActiveNodeProxy" method hasn't reached the maximum retry limit, the overall execution process doesn't exit and it continues to attempt to
call the "rollEditLog" on the next NameNode in the list. However when a socket connection is established, it throws a "java.nio.channels.ClosedByInterruptException" exception due to the thread being in an "Interrupted" state.
this cycle repeats until it reaches the maximum retry limit (nnCount * maxRetries) will exits.
However in the next cycle of "Triggering log roll," it continues to traverse the NameNode list and encounters the same issue and the cachedActiveProxy is still a "shut down NameNode."
This eventually results in the NameNode being unable to successfully complete the "Triggering log roll" operation.
To optimize this, we need to handle the thread being interrupted and exit the execution.
Detailed logs such as:
the Observer node "ob1" will execute "Triggering log roll" is as follows:
nns list is [ob2(shut down machine),nn1(active),nn2(standy)]

The Observer node "ob1" periodically executes "triggerActiveLogRoll" and asynchronously calls "getNameNodeProxy" to request "ob2" for the "rollEditLog" operation, since the "ob2" machine is shut down, it cannot establish a network connection, this results in a timeout during the socket connection phase (here set timeout is 90 seconds).

2023-11-03 10:27:41,734 INFO ha.EditLogTailer (EditLogTailer.java:triggerActiveLogRoll(465)) [Edit log tailer] - Triggering log roll on remote NameNode

2023-11-03 10:28:41,734 WARN ha.EditLogTailer (EditLogTailer.java:triggerActiveLogRoll(478)) [Edit log tailer] - Unable to finish rolling edits in 60000 ms

As the asynchronous call for "Triggering log roll" has a waiting time of 60 seconds, it triggers a timeout and initiates a "cancel" operation, causing the executing thread to receive an "Interrupted" signal and will throw "java.io.InterruptedIOException".
2023-11-03 10:28:41,753 WARN ipc.Client (Client.java:handleConnectionFailure(930)) [pool-33-thread-1] - Interrupted while trying for connection
2023-11-03 10:28:41,972 WARN ha.EditLogTailer (EditLogTailer.java:call(618)) [pool-33-thread-1] - Exception from remote name node RemoteNameNodeInfo [nnId=ob2, ipcAddress=xxx, httpAddress=http://xxx], try next.
java.io.InterruptedIOException: DestHost:destPort xxx , LocalHost:localPort xxx. Failed on local exception: java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connection-pending remote=xxx:8040]. Total timeout mills is 90000, 30038 millis timeout left.
at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:931)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:906)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1583)
at org.apache.hadoop.ipc.Client.call(Client.java:1511)
at org.apache.hadoop.ipc.Client.call(Client.java:1402)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:261)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:141)
at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:152)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:454)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:451)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:615)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connection-pending remote=xxx:8040]. Total timeout mills is 90000, 30038 millis timeout left.
at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:604)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:662)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:783)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:346)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1653)

Since current logic not to handle interrupted signal and hasn't reached the maximum retry limit, the overall execution process doesn't exit and it continues to attempt to call the "rollEditLog" on the next NameNode in the list. However when a socket connection is established, it throws a "java.nio.channels.ClosedByInterruptException" exception due to the thread being in an "Interrupted" state. this cycle repeats until it reaches 9 retry limit (3*3) and then exits.

2023-11-03 10:28:41,991 WARN ipc.Client (Client.java:handleConnectionFailure(930)) [pool-33-thread-1] - Interrupted while trying for connection
2023-11-03 10:28:41,991 WARN ha.EditLogTailer (EditLogTailer.java:call(618)) [pool-33-thread-1] - Exception from remote name node RemoteNameNodeInfo [nnId=nn1, ipcAddress=xxx:8040, httpAddress=http://xxx:50070], try next.
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:656)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:604)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:662)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:783)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:346)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1653)
at org.apache.hadoop.ipc.Client.call(Client.java:1449)
at org.apache.hadoop.ipc.Client.call(Client.java:1402)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:261)
at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:141)
at com.sun.proxy.$Proxy21.rollEditLog(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:152)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:454)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$2.doWork(EditLogTailer.java:451)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$MultipleNameNodeProxy.call(EditLogTailer.java:615)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

2023-11-03 10:28:42,095 WARN ipc.Client (Client.java:handleConnectionFailure(930)) [pool-33-thread-1] - Interrupted while trying for connection
2023-11-03 10:28:42,095 WARN ha.EditLogTailer (EditLogTailer.java:call(618)) [pool-33-thread-1] - Exception from remote name node RemoteNameNodeInfo [nnId=nn2, ipcAddress=xxx:8040, httpAddress=xxx:50070], try next.
java.nio.channels.ClosedByInterruptException

2023-11-03 10:28:42,104 WARN ipc.Client (Client.java:handleConnectionFailure(930)) [pool-33-thread-1] - Interrupted while trying for connection
2023-11-03 10:28:42,104 WARN ha.EditLogTailer (EditLogTailer.java:call(618)) [pool-33-thread-1] - Exception from remote name node RemoteNameNodeInfo [nnId=ob2, ipcAddress=xxx:8040, httpAddress=xxx:50070], try next.
java.nio.channels.ClosedByInterruptException

2023-11-03 10:28:42,118 WARN ipc.Client (Client.java:handleConnectionFailure(930)) [pool-33-thread-1] - Interrupted while trying for connection
2023-11-03 10:28:42,118 WARN ha.EditLogTailer (EditLogTailer.java:call(618)) [pool-33-thread-1] - Exception from remote name node RemoteNameNodeInfo [nnId=nn1, ipcAddress=xxx:8040, httpAddress=xxx:50070], try next.

2023-11-03 10:28:42,119 WARN ipc.Client (Client.java:handleConnectionFailure(930)) [pool-33-thread-1] - Interrupted while trying for connection
2023-11-03 10:28:42,119 WARN ha.EditLogTailer (EditLogTailer.java:call(618)) [pool-33-thread-1] - Exception from remote name node RemoteNameNodeInfo [nnId=nn2, ipcAddress=xxx:8040, httpAddress=xxx:50070], try next.
java.nio.channels.ClosedByInterruptException

2023-11-03 10:28:42,148 WARN ipc.Client (Client.java:handleConnectionFailure(930)) [pool-33-thread-1] - Interrupted while trying for connection
2023-11-03 10:28:42,148 WARN ha.EditLogTailer (EditLogTailer.java:call(618)) [pool-33-thread-1] - Exception from remote name node RemoteNameNodeInfo [nnId=ob2, ipcAddress=xxx:8040, httpAddress=xxx:50070], try next.
java.nio.channels.ClosedByInterruptException

2023-11-03 10:28:42,148 WARN ipc.Client (Client.java:handleConnectionFailure(930)) [pool-33-thread-1] - Interrupted while trying for connection
2023-11-03 10:28:42,148 WARN ha.EditLogTailer (EditLogTailer.java:call(618)) [pool-33-thread-1] - Exception from remote name node RemoteNameNodeInfo [nnId=nn1, ipcAddress=xxx:8040, httpAddress=xxx:50070], try next.
java.nio.channels.ClosedByInterruptException

2023-11-03 10:28:42,149 WARN ipc.Client (Client.java:handleConnectionFailure(930)) [pool-33-thread-1] - Interrupted while trying for connection
2023-11-03 10:28:42,149 WARN ha.EditLogTailer (EditLogTailer.java:call(618)) [pool-33-thread-1] - Exception from remote name node RemoteNameNodeInfo [nnId=nn2, ipcAddress=xxx:8040, httpAddress=xxx:50070], try next.
java.nio.channels.ClosedByInterruptException

In the next cycle of "Triggering log roll," it continues to traverse the NameNode list and proceeds to "ob2," causing the same issue to recur. This eventually results in the NameNode being unable to successfully complete the "Triggering log roll" operation.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 54s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
49m 22s

trunk passed

+1 :green_heart:
compile
1m 24s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 14s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 11s

trunk passed

+1 :green_heart:
mvnsite
1m 24s

trunk passed

+1 :green_heart:
javadoc
1m  9s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 36s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 17s

trunk passed

+1 :green_heart:
shadedclient
40m 44s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 12s

the patch passed

+1 :green_heart:
compile
1m 15s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 15s

the patch passed

+1 :green_heart:
compile
1m  7s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m  7s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  1s

the patch passed

+1 :green_heart:
mvnsite
1m 13s

the patch passed

+1 :green_heart:
javadoc
0m 57s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 30s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 21s

the patch passed

+1 :green_heart:
shadedclient
40m 30s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
240m 56s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 42s

The patch does not generate ASF License warnings.

397m 27s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

hadoop.hdfs.TestDFSUtil

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6266/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6266

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 91bee368e523 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 4774224af6701bcd041da7cc1b47efcb7ea8aff9

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6266/1/testReport/

Max. process+thread count
2363 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6266/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
18m 11s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
49m 12s

trunk passed

+1 :green_heart:
compile
1m 22s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 15s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 11s

trunk passed

+1 :green_heart:
mvnsite
1m 22s

trunk passed

+1 :green_heart:
javadoc
1m 10s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 36s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 18s

trunk passed

+1 :green_heart:
shadedclient
40m 28s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 10s

the patch passed

+1 :green_heart:
compile
1m 15s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 15s

the patch passed

+1 :green_heart:
compile
1m  8s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m  8s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  2s

the patch passed

+1 :green_heart:
mvnsite
1m 14s

the patch passed

+1 :green_heart:
javadoc
0m 56s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 32s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 21s

the patch passed

+1 :green_heart:
shadedclient
40m 33s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
239m 56s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 43s

The patch does not generate ASF License warnings.

412m 54s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

hadoop.hdfs.TestDFSUtil

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6266/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6266

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux f5319d6216b0 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 27e67f4d05631ca8974ac77127198fe5fbe61072

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6266/2/testReport/

Max. process+thread count
2354 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6266/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hi @Hexiaoqiao @ZanderXu  @ayushtkn @zhangshuyan0 Would you mind to take a review this pr when you have free time? thank you very much~

Hi @tomscut @xinglin Would you mind to also take a review this pr when you have free time? thank you very much~

Thanks @haiyang1987 for your good work here. LGTM. +1 from my side. Let's wait if other folks would like to be involved here. Thanks.

Thanks @Hexiaoqiao for your review it.

qq: which version of hadoop did you run? Does triggerActiveLogRoll() actually try to connect other NN as well? Is your issue happening for trunk? By reading the source code, that does not seem to be the case.
log line of "Triggering log rolling to the remote NameNode, ": is this truncated in your paste? seems to be incomplete compared to trunk.
  /**
   * Trigger the active node to roll its logs.
   */
  @VisibleForTesting
  void triggerActiveLogRoll() {
    Future<Void> future = null;
    try {
     ...
   }

  Callable<Void> getNameNodeProxy() {
    return new MultipleNameNodeProxy<Void>() {
      @Override
      protected Void doWork() throws IOException {
        LOG.info("Triggering log rolling to the remote NameNode, " +
            "active NameNode = {}", currentNN.getIpcAddress());
        cachedActiveProxy.rollEditLog();
        return null;
      }
    };

qq: which version of hadoop did you run? Does triggerActiveLogRoll() actually try to connect to other NNs as well or only activeNN? Is your issue happening for trunk? By reading the source code, that does not seem to be the case: It seems triggerActiveLogRoll should only contact activeNN.
log line of "Triggering log rolling to the remote NameNode, ": is this truncated in your paste? seems to be incomplete compared to trunk.
  /**
   * Trigger the active node to roll its logs.
   */
  @VisibleForTesting
  void triggerActiveLogRoll() {
    Future<Void> future = null;
    try {
     ...
   }

  Callable<Void> getNameNodeProxy() {
    return new MultipleNameNodeProxy<Void>() {
      @Override
      protected Void doWork() throws IOException {
        LOG.info("Triggering log rolling to the remote NameNode, " +
            "active NameNode = {}", currentNN.getIpcAddress());
        cachedActiveProxy.rollEditLog();
        return null;
      }
    };

Thanks @xinglin for your comment.
The current version is hadoop 3.3, triggerActiveLogRoll() will actually try to connect to other NNs until active nn is requested, and recorded as cachedActiveProxy
for this log
LOG.info("Triggering log rolling to the remote NameNode, " +  "active NameNode = {}", currentNN.getIpcAddress());

It is from by PR: HDFS-1630, it only record the remote NameNode address in logs. this PR is not introduced in our current version.
trunk branch still has the problem described in the current issue.

thanks @haiyang1987.  The name ofgetActiveNodeProxy() caused the confusion. It should probably have been named as getAliveNodeProxy. ActiveNN has a specific meaning in the context of HA-NN. getActiveNodeProxy indeed iterates over all NNs.
Neither MultipleNameNodeProxy.call() or MultipleNameNodeProxy.getActiveNodeProxy() catch InterruptedExecution in their try_catch block.
have two dummy questions
Question 1:
when we submit a task to an executorService and later on we cancel it due to timeout, and we don't capture interrupted exception in our task, what state will the thread from executorService be left? Do we have to capture interrupted exception in our task in this case?
ob1 trying to connect to ob2. we timed out after 60 sec. Are you saying the thread in the executorService would still be in Interrupted state, even though we have throw up the Interrupted Execution, as showed in the log (also because it is not captured within MultipleNameNodeProxy)? This seems to be contradicting to a statement I found from below blog, which says when an interruptedExecution is threw, the interruption status of that thread will be cleared.
"Before a blocking code throws an InterruptedException, it marks the interruption status as false."
https://praveer09.github.io/technology/2015/12/06/understanding-thread-interruption-in-java/
Another question:  assuming the thread in the executorService is still in interrupted state, then how is it gotten cleared by your PR? Thread.currentThread().isInterrupted() will not clear the interrupt flag for a thread, as the Oracle Java doc says.

When a thread checks for an interrupt by invoking the static method Thread.interrupted, interrupt status is cleared. The non-static isInterrupted method, which is used by one thread to query the interrupt status of another, does not change the interrupt status flag.
https://docs.oracle.com/javase/tutorial/essential/concurrency/interrupt.html#:~:text=When a thread checks for,change the interrupt status flag.

InterruptedIOException is a subClass of IOException. In the try_catch block in getActiveNodeProxy, we do catch for IOException. So, interruptedIOException will be captured there.
https://docs.oracle.com/javase/8/docs/api/java/io/InterruptedIOException.html
What I think should be happening is as following.
main_Thread calling triggerActiveLogRoll, wait for 60 secs, timeout, cancel this task, and return.
MultipleNameNodeProxy.call() thread:
-> getActiveNodeProxy()
-> nnLookup.next = ob2 (down node)
-> RPC.waitForProxy(ob2)
-> after 60 secs, interrupted.
-> output "Failed to reach ob2", increment nnLoopCount.
-> nnLookup.next = n1 (live node).
then, it should succeed to connect to n1.
This does not seem to be the case from the logs you shared. thoughts?
A possible fix might be to explicitly capture InterruptedIOException in getActiveNodeProxy, and just finish for this thread (assuming all InterruptedIOExceptions are invoked from triggerActiveLogRoll). For the following triggerActiveLogRoll calls, we should be good, since we will move the nnLookup to next one.

@xinglin Thanks for your review.

when we submit a task to an executorService and later on we cancel it due to timeout, and we don't capture interrupted exception in our task, what state will the thread from executorService be left? Do we have to capture interrupted exception in our task in this case?

This is very nice question,  you can refer to the source code of ThreadPoolExecutor#Worker#runWorker(), the worker in the ThreadPoolExecutor  will clear the Interrupt flag.

Thanks  @xinglin @ZanderXu for your detailed review.

when we submit a task to an executorService and later on we cancel it due to timeout, and we don't capture interrupted exception in our task, what state will the thread from executorService be left? Do we have to capture interrupted exception in our task in this case?

This is very nice question, you can refer to the source code of ThreadPoolExecutor#Worker#runWorker(), the worker in the ThreadPoolExecutor will clear the Interrupt flag.

Yeah, totally agree with this.

InterruptedIOException is a subClass of IOException. In the try_catch block in getActiveNodeProxy, we do catch for IOException. So, interruptedIOException will be captured there.
https://docs.oracle.com/javase/8/docs/api/java/io/InterruptedIOException.html
What I think should be happening is as following.
main_Thread calling `triggerActiveLogRoll`, wait for 60 secs, timeout, cancel this task, and return.

MultipleNameNodeProxy.call() thread:
   -> getActiveNodeProxy()
        -> nnLookup.next = ob2 (down node)
        -> RPC.waitForProxy(ob2)
        -> after 60 secs, interrupted.
        -> output "Failed to reach ob2", increment nnLoopCount. Ideally, we should just stop here, since we already time out.
        -> nnLookup.next = n1 (live node).
             then, it should succeed to connect to n1.

This does not seem to be the case from the logs you shared. thoughts?
A possible fix might be to explicitly capture InterruptedIOException in getActiveNodeProxy, and just finish for this thread (assuming all InterruptedIOExceptions are invoked from triggerActiveLogRoll). For the following triggerActiveLogRoll calls, we should be good, since we will move the nnLookup to next one.

Yeah, For MultipleNameNodeProxy#call() explicitly capture InterruptedIOException and then exit execution is also a solution

Feel free to commit this PR. did not intend to block here. I will spend more time on my own trying to understand the code/and the change .

Hi @Hexiaoqiao @ZanderXu could you mind to push this modification forward when you have free time ? Thank you very much.

Committed to trunk. Thanks @haiyang1987 @ZanderXu and @xinglin .

Thanks @Hexiaoqiao @ZanderXu @xinglin for your review and merge.


Translate strings to Japanese

Translated (translate me) strings to Japanese and improved former translation slightly.
#462

LGTM

Thank you for the translation! Added it to the String resource translation section of the Notable Community Contributions wiki page.

Thank you, too!


{i*,x86_64}-pc-windows-gnu could be supported

Supporting cross compilation from unix to *-pc-windows-gnu is easy with the mingw toolchain, which happens to include .libs for the system libraries.

Certainly. IIRC, cross compiling to x86_64 mingw works out of the box with Ubuntu's packaged toolchain but i686 mingw because that toolchain has been configured with a different threading model or something. We'l have to get or build a different toolchain for that target.
It may even be possible to hook binfmt with wine to get cargo run working. (Or perhaps that already works with having to do anything? I don't remember)


Fix for expansion within lists of strings

Codecov Report

Merging #58 (5cd75df) into main (81dab1a) will decrease coverage by 0.12%.
The diff coverage is 66.66%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff             @@
##             main      #58      +/-   ##
==========================================
- Coverage   89.64%   89.52%   -0.12%
==========================================
  Files           4        4
  Lines         676      678       +2
==========================================
+ Hits          606      607       +1
- Misses         70       71       +1

Impacted Files
Coverage Δ

adtl/__init__.py
85.00% <66.66%> (-0.17%)
:arrow_down: