Should we always call destory before removing a mapped file ?

The issue tracker is ONLY used for bug report(feature request need to follow RIP process). Keep in mind, please check whether there is an existing same report before your raise a new one.
Alternately (especially if your communication is not a bug report), you can send mail to our mailing lists. We welcome any friendly suggestions, bug fixes, collaboration and other improvements.
Please ensure that your bug report is clear and that it is complete. Otherwise, we may be unable to understand it or to reproduce it, either of which would prevent us from fixing the bug. We strongly recommend the report(bug report or feature request) could include some hints as the following:
BUG REPORT

Please describe the issue you observed:

In MappedFileQueue:273 we have
        while (iterator.hasPrevious()) {
            mappedFileLast = iterator.previous();
            if (offset >= mappedFileLast.getFileFromOffset()) {
                int where = (int) (offset % mappedFileLast.getFileSize());
                mappedFileLast.setFlushedPosition(where);
                mappedFileLast.setWrotePosition(where);
                mappedFileLast.setCommittedPosition(where);
                break;
            } else {
                 // Can we add destroy here ?
                 // mappedFileLast.destroy();
                iterator.remove();
            }
        }

What did you expect to see?

What did you see instead?

Please tell us about your environment:

Other information (e.g. detailed explanation, logs, related issues, suggestions how to fix, etc):

FEATURE REQUEST

Please describe the feature you are requesting.

Provide any additional detail on your proposed use case for this feature.

Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

sub-task1-issue-number: sub-task1 description here,
sub-task2-issue-number: sub-task2 description here,
...

RocketMQ will delete the files periodically through configuration or disk is full. But if we call destory in this method, the file will be deleted and the messages will be lost. So I think this place we should not call the destory.

Sorry I didn't make it claer. Actually this is found in resetOffset():
    public boolean resetOffset(long offset) {
        MappedFile mappedFileLast = getLastMappedFile();

        if (mappedFileLast != null) {
            long lastOffset = mappedFileLast.getFileFromOffset() +
                mappedFileLast.getWrotePosition();
            long diff = lastOffset - offset;

            final int maxDiff = this.mappedFileSize * 2;
            if (diff > maxDiff)
                return false;
        }

        ListIterator<MappedFile> iterator = this.mappedFiles.listIterator();

        while (iterator.hasPrevious()) {
            mappedFileLast = iterator.previous();
            if (offset >= mappedFileLast.getFileFromOffset()) {
                int where = (int) (offset % mappedFileLast.getFileSize());
                mappedFileLast.setFlushedPosition(where);
                mappedFileLast.setWrotePosition(where);
                mappedFileLast.setCommittedPosition(where);
                break;
            } else {
                iterator.remove();
            }
        }
        return true;
    }

and I saws  deleteLastFile()  actually deleted the file.
    public void deleteLastMappedFile() {
        MappedFile lastMappedFile = getLastMappedFile();
        if (lastMappedFile != null) {
            lastMappedFile.destroy(1000);
            this.mappedFiles.remove(lastMappedFile);
            log.info("on recover, destroy a logic mapped file " + lastMappedFile.getFileName());
        }
    }

So it appears to me that when we remove a mappedFile from this.mappedFiles，we actually want to delete the underlying file as well. I thought if we use iterator.remove() without deleting the file, we lose the reference to that file as well. (because this.mappedFiles are supposed to hold the )
Otherwise it might be that we don't need to call iterator.remove(), because I saw Java 8 doc

(For CopyOnWriteList iterators)Mutation operations on iterators (remove, set, and add) are not supported. These methods throw UnsupportedOperationException. (which is a runtime exception)

Which seems weird to me. If that's true then the iterator.remove() is not working at all.

The problem here is iterator.remove() doesn't really do anything (Unsupported Operation) Could be a potential bug. Will catch up with this one recently


Explicitly state that Zacas depends upon the A extension

This wasn't explicitly stated anywhere, but I assume it's intended.

Thanks for taking a look.
Done - let me know if that wasn't where you meant.

Thanks.


Added 3sum

Added 3 problems

@vJechsmayr Hello, the codes are working perfectly fine. If you want to run them on leetcode you must paste the code without disturbing the class solution.

@VaibhaveS  The class Solution is there as you can see 😃

Hey, yeah it's due to indentation error while copy pasting...


Conflict with modules named dataset

System Info:

Arch: Linux
Distro: Ubuntu 22.04

Package Version(s): 1.1 and 1.2
Issue: Import error possibly due to friction between pyarrow and image reward package? (not sure)
Description:
I've tried installing t2v-metrics through pip and building from source, but I still get the same error:
/opt/conda/envs/t2v/lib/python3.10/site-packages/transformers/utils/generic.py:441: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  _torch_pytree._register_pytree_node(
/opt/conda/envs/t2v/lib/python3.10/site-packages/transformers/utils/generic.py:309: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  _torch_pytree._register_pytree_node(
/opt/conda/envs/t2v/lib/python3.10/site-packages/transformers/utils/generic.py:309: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  _torch_pytree._register_pytree_node(
/opt/conda/envs/t2v/lib/python3.10/site-packages/transformers/utils/generic.py:309: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  _torch_pytree._register_pytree_node(
Traceback (most recent call last):
  File "/home/ubuntu/gan-unlearning/evaluate.py", line 8, in <module>
    import t2v_metrics
  File "/opt/conda/envs/t2v/lib/python3.10/site-packages/t2v_metrics/__init__.py", line 9, in <module>
    from .itmscore import ITMScore, list_all_itmscore_models
  File "/opt/conda/envs/t2v/lib/python3.10/site-packages/t2v_metrics/itmscore.py", line 7, in <module>
    from .models.itmscore_models import list_all_itmscore_models, get_itmscore_model
  File "/opt/conda/envs/t2v/lib/python3.10/site-packages/t2v_metrics/models/itmscore_models/__init__.py", line 2, in <module>
    from .image_reward_model import IMAGE_REWARD_MODELS, ImageRewardScoreModel
  File "/opt/conda/envs/t2v/lib/python3.10/site-packages/t2v_metrics/models/itmscore_models/image_reward_model.py", line 6, in <module>
    import ImageReward as reward
  File "/opt/conda/envs/t2v/lib/python3.10/site-packages/ImageReward/__init__.py", line 3, in <module>
    from .ReFL import *
  File "/opt/conda/envs/t2v/lib/python3.10/site-packages/ImageReward/ReFL.py", line 27, in <module>
    from datasets import load_dataset
ImportError: cannot import name 'load_dataset' from 'datasets' (/home/ubuntu/gan-unlearning/datasets/__init__.py)

Trigger: When running import t2v_metrics
When building from source, I don't get the Future warnings but the import error persists. I've tried looking through the source code and image reward issues but couldn't figure anything out. Perhaps I'm missing something really obvious (torch version, etc.), any help is very much appreciated.

Oh, I seem to have found the cause. This error only occurs if I have a module called datasets in my current working directory. In my project, I have several files to handle loading of datasets under this module name. I'm hoping some kind of fix can be made since using the module name datasets is quite common in research repos. In any case, thanks for sharing your work, this is really cool!

Ah thanks for your kind words and glad you found the cause!


How to search node modules in one step back parent folder

Hi,
Currently we can search only node modules in workspaceFolder but how we can search in parent folder..where i want to change in extension.js?
My project folder structure :
http://localhost/projects/mynodemodules/myproject
mynodemodules->folder [  http://localhost/projects/mynodemodules/  ]

node_modules
myproject->folder [ http://localhost/projects/mynodemodules/myproject ]
-index.html
-text.js
-style.css

I am facing a similar issue.
I have the following structure ⸺
- root/
     - project/
         - index.js
         - package.json
         - node_modules
     - README.md

I have opened root/ folder in VSCode & I entered Search node_modules in search. It threw an error Search node_modules: No node_modules folder in this workspace.
Any ideas to solve this?

@deadcoder0904 : We can resolve easy,
getWorkspaceFolder().then(folder => folder && getProjectFolder(folder)).then(folder => {
if (folder) {
var f_path = path.join(folder.path, '../');
searchPath(folder.name, f_path, modulesPath);
}
});

@SaranSank PR might be more helpful :)

@deadcoder0904 : You are from india? You well in nodejs?

Yeah

Which location in india? I have one doubt in nodjs

You should ask such questions on StackOverflow. Never spam an issue.
But to answer your question, that is not the correct way unless your host is publicly accessible. Do it just as you do locally & then push to server & it should work.

I did in locally.But not working for server url

Try posting on reddit.com/r/nodejs

Can you send me ur email id or any other mode to contact. That is very use to contact you. Here we can not discuss more

@deadcoder0904 : Are you there?


Refactor: Update default nav template

Re: #23
Overview
This makes a variety of changes to the default patterns, pages, and navigation.

A few more patterns have been added ("Elements") to better demonstrate that you can break patterns into multiple sections (aside from "Components")
The color swatches have been placed in a new standalone page
Metadata has been added to affect the sort order of some collections
The navigation template has been updated with headings for each grouping, and now uses the sorting feature (https://github.com/cloudfour/drizzle-builder/pull/95) of the {{collections}} helper to control the order or collections.

/CC @mrgerardorodriguez @saralohr @nicolemors

LGTM :+1:

Yay! 👍


Provide option to reboot after Boxstarter script is complete

Good evening
I discussed this subject in discord and was told to create a Feature Request if we really need it. Thanks a lot for this support and this possibility!
Is Your Feature Request Related To A Problem? Please describe.
The Feature Request is related to this quality workflow:

I use BoxStarter to initialize new Notebooks for non-profit organizations and afterward, I send those notebooks far away.
Therefore I start BoxStarter, it works for about 50 minutes and installs Windows Updates, too
Afterward, as part of the quality assurance, I must reboot the computer and validate that the most important things are ok.

For example, I ensure:

that Windows actually boots and it doesn't take 40 minutes to finish installing updates. It's great if users can just start working 😄
that all Windows updates have actually been installed which BoxStarter cannot always guarantee (and that's fine because Windows updates are still not reliable).
That the most critical applications, such as OpenVPN and Office, work, are configured correctly, and have the right licenses.

Of course, I can restart the computer manually after the BoxStarter installation. But because there are very useful use-cases in which the restart is part of the installation process, it would be great if BoxStarter could do it too.
Describe The Solution. Why is it needed?
Therefore,
it would be great if BoxStarter would offer a reboot command which does the following:

Cleanup all BoxStarter configurations so that, for example,
there is no longer an autologin
and BoxStarter does not continue with the installation script after login

Put simply:
A reboot command that disables BoxStarter and then restarts the computer normally.
By the way, I am responsible for several hundred employees at work and I would like to introduce Chocolatey & BoxStarter there as well. But there, too, it is part of the quality control that we restart the computer after the SW installation is done and check the most important things.
Additional Context.
N/A
Related Issues
N/A
Thanks a lot, kind regards,
Thomas

Without getting into all the detail, may I suggest you add
Install-WindowsUpdate -AcceptEula

at the very end of your Boxstarter install script(s) ?
I suspect this would solve the problem at hand.

There has been no updates to this in some time, so I'm going to go ahead and close this, but we can reopen it if necessary.


Firebase-UI and Espresso testing

Hello, I'm using this library for quite some time now and it works fine. Thanks for the effort!
At this moment I'm running into an issue setting up my test code: I would like to sign in a (test) user using an Espresso test case. Starting the sign in flow works fine, but as soon as I need to click the "NONE OF THE ABOVE" button in the email selection dialog, I run into the issue that this code is not part of the library (I'm not able to click on it within the test cases). 2 possible options that might help:

make the email selection behavior part of the library (in order to make Espresso testing possible)
make the behavior optional (e.g. with a disableMailSelectionDialog method on the createSignInIntentBuilder)?

Not sure if this is technically possible and maybe there are better more elegant solutions?

@peterhav thanks for the report.  You're right, there is more we could do to enable Espresso testing.
I believe the dialog you're talking about is the email selection dialog from Google Sign In.  As you said, it's not in the application code because that's actually a system dialog served by Google Play Services.  In the past I have personally used UiAutomator tests to test that flow but I'll see if I can think of a way to enable Espresso testing there.

@peterhav I'm not sure if this is still relevant, but the new FirebaseUI versions have a setIsSmartLockEnabled(boolean) which would let you disable the Google Account Picker dialog. Then you can sign in via email using a fake account for your Espresso tests.
@samtstern I'm combing through old issues here, would you consider this one resolved?

I guess this is as resolved as it is going to be.  SmartLock and Google Sign In both display their UI from Google Play services so there's no way to use Espresso testing for them.  I tend to avoid them in tests when possible and use UiAutomator tests (slow, but can access anything on the screen) when I can't.


Change profile picture

I can't find any option to change my profile picture in the app. If it is not there, I would like to implement it.

Assign to me

Assign to me


XmlPackage.java:34-35: Find all usages of metadata files...

The puzzle 159-8e7ac748 from #159 has to be resolved:
https://github.com/artipie/rpm-adapter/blob/762e9d6cd167f35feb5637f7b2cde50e812a4ec7/src/main/java/com/artipie/rpm/meta/XmlPackage.java#L34-L35
The puzzle was created by @olenagerasimova on 15-May-20.
Estimate: 30 minutes,  role: DEV.
If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

@g4s8/z please, pay attention to this issue

The puzzle 159-8e7ac748 has disappeared from the source code, that's why I closed this issue.

Job gh:artipie/rpm-adapter#194 is not assigned, can't get performer


[server] Move public folder to configuration

Folder moved to public_files in units/_configuration/urb-base-server/public_files


New religions, minor bugfixes for religious bonuses, added shillings

Changes / Comments
Roll20 Requests
Include the name of the sheet(s) you made changes to in the title.
Comments are very helpful for reviewing the code changes. Please answer the relevant questions below in your comment.

Is this a bug fix?
Does this add functional enhancements (new features or extending existing features) ?
Does this add or change functional aesthetics (such as layout or color scheme) ?
Are you intentionally changing more that one sheet? If so, which ones ?
If changing or removing attributes, what steps have you taken, if any, to preserve player data ?
If this is a new sheet, did you follow Building Character Sheets standards ?

If you do not know English. Please leave a comment in your native language.

Why is the sheet really wide now? Is this to accommodate the manor work sheet?  If so that could be split onto 2 lines.


Signal when ignition is reached via event, metric and condition and delay MHC creation

What this PR does / why we need it:
There're scenarios where is diffcult too understand why an instance does not become a Kubernetes Node. Once concrete problem is the instances because of reasons are not able to reach the ignition endpoint, e.g. bad networling blocking security groups.
This PR tries to makes obvious and expliciti when an instance for a NodePool was able to reach the ignition endpoint. When that happens the ign server creates an event, sends a metric and sets the conditions in the NodePool.
Additionally this PR delays MHC creation until the previous conditions is set and true, as otherwise keep recreating Machines in loop won't fix the problem but would rather result in very awkward UX.
ref https://issues.redhat.com/browse/HOSTEDCP-678
ref https://issues.redhat.com/browse/HOSTEDCP-659
Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #
Checklist

[ ] Subject and description added to both, commit and PR.
[ ] Relevant issues have been referenced.
[ ] This change includes docs.
[ ] This change includes unit tests.

/hold to drop the cp namespace->nodePool access.

/hold cancel


Update version to pass build metadata with "+" separator

https://semver.org/#spec-item-10
We hit a problem with a prior version of this operator (https://github.com/openshift/dedicated-admin-operator/commit/0159209) where the first 7 chars of git hash were numeric and had a leading zero (0).  This is not allowed per https://semver.org/#spec-item-9
See https://jira.coreos.com/browse/SREP-1811

/lgtm
/hold
So this change looks fine, but will it break CSV generation going backwards if the image tag format is changing? Placing a hold to answer this question. If it won't break anything, then feel free to /hold cancel

@cblecker csv generation uses the package file in the catalog source to determine the current version, it doesn't regenerate it.  You can verify with make build-catalog-image and check the file generated in bundles-staging/

Happy to keep the hold until you remove it, another set of eyes to confirm what I'm seeing is good if it might break upgrades.

/lgtm

/hold cancel
Will review what happens in stage wrt breaking upgrade. Based on what I see in the generation of the catalogsource it'll be fine.


Missing link to TOC

link to TOC seem to be in random places, because certain places here and there don't have them.

The link to TOC is placed such that it fits perfectly into the pdf


Drop doesn't move item

Please remove sections that are not required.

I'm submitting a ...
[ X ] bug report
[ ] feature request
[ ] other

What is the current behavior?

Dropping an item doesn't move it

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem via https://plnkr.co or similar.

Title

Subsection 1

    My Text

 DROP ZONE

What is the expected behavior?

When I drag something into the drop zone it should move

Please tell us about your environment:

Angular version: 6.0.3
Chrome 64

I don't think this is a bug, it is expected behaviour. You need to specify the (onDrop) event to move items between containers. See https://github.com/ObaidUrRehman/ng-drag-drop#transfer-data-via-drag-drop

Hi, i have a similar error; Could you help me?
My case its the following:
I take (Drag) a element of my list1 and when i (Drop) this element on list2, It isnt move & send a Error on console:
TypeError: Cannot read property 'map' of undefined
at UserDashboardComponent.push../src/app/
I have reviewed the code and it is in this segment where it marks the error:
  onDropSelected(e: DropEvent){
this.productList.push(e.dragData);
this.removeItem(e.dragData, this.products);
}
onDropProducts(e: DropEvent){
this.products.push(e.dragData);
this.removeItem(e.dragData, this.productList);
}
removeItem(item: any, list: Array) {
console.log(list);
  let index = list.map(e => {
return e.name
}).indexOf(item.name);
list.splice(index, 1);
}
Thanks, bros


Remove testing from command line in readme

Looks like this functionality was removed when the Rakefile was deleted in e0815da

:beers:


Conversion Stage immediately exits & Non-BONELAB game question

I'm attempting to run asset_migrator.exe in Windows (10) Terminal in a D: drive, passing in my ripped project's Assets folder as the first argument and the destination Assets folder as the second. My final command looks like this:
 .\asset_migrator.exe "D:\SteamLibrary\steamapps\common\<game>\<game>_Data\Ripped\<game>\ExportedProject\Assets" "D:\Dev Stuff\Projects\VS Community\Unity\My project\Assets"
The collection stage only lasts a couple seconds and the program exits silently immediately after the conversion stage.

Also possibly related, is there any BONELAB/IL2CPP-specific code in this? I'm asking because I'm ripping a game with a Mono backend.

There's no specific code for BONELAB. I've actually used this tool in personal projects to port over old maps.
Although the problem you're experiencing is poor documentation and terrible instructions on my part, sorry!
After you specify the input asset directory and output asset directory you must specify the assets you want to convert over. Such as this ./asset_migrator.exe "C:/SrcProject/Assets" "C:/DstProject/Assets" "MyScene.unity". Then after running this command you should see MyScene.unity and dependent assets inside of the ConversionOutput folder

There's no specific code for BONELAB. I've actually used this tool in personal projects to port over old maps.
Although the problem you're experiencing is poor documentation and terrible instructions on my part, sorry!
After you specify the input asset directory and output asset directory you must specify the assets you want to convert over. Such as this ./asset_migrator.exe "C:/SrcProject/Assets" "C:/DstProject/Assets" "MyScene.unity". Then after running this command you should see MyScene.unity and dependent assets inside of the ConversionOutput folder. You can then copy these into the destination project and everything should work as-is

Ah, thank you for the quick response. This got me further, I've got another problem now that I'll make another issue for.


Create Translator UI subset of MONDO terms

I'll add to this ticket as folks at Translator explore the ontology, if that is ok?  Translator would like to have a subset of MONDO that excludes terms that don't match their use cases.
Exclusion rules:
any disease or disorder that is this term or a subclass_of or part_of this term: https://www.ebi.ac.uk/ols4/ontologies/mondo/classes/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0000947?lang=en
any disease or disorder that is this term or a subclass_of or part_of this term:
https://www.ebi.ac.uk/ols4/ontologies/mondo/classes/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0000595?lang=en
any disease or disorder that is this term or a subclass_of or part_of this term:
https://www.ebi.ac.uk/ols4/ontologies/mondo/classes/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0001500?lang=en

@sierra-moxon can you comment on the advantage of us creating a specific translator subset vs us giving you two lines of code that achieve the same thing as a preprocessing step for Translator?

@sierra-moxon do you think we should actually obsolete these terms that you'd like to exclude in your subset? @sabrinatoro noted we probably need to review the psychiatric disorder branch in Mondo. (There is a related discussion for this: https://github.com/monarch-initiative/mondo/discussions/2678)

Hi @nicolevasilevsky - @matentzn  and I met this morning and he showed me the code to create this -- it works great! :) Thank you both for the help here.  I think we can make this standalone code work for Translator, the benefits here are that Translator folks can work on their own timeline to add/remove terms from the subset.  The drawbacks are that Translator is using a non-official subset from Mondo (but its probably ok).  I'll let you know if there are any concerns with this.
monarch-initiative/mondo-translator

that's so cool! Will you be able to update the subset based on the Mondo releases?
Should we close this ticket?

Per the obsoletion questions, I think https://github.com/monarch-initiative/mondo/issues/6366, https://github.com/monarch-initiative/mondo/issues/6364 cover a couple of instances that I found that I think should be obsoleted (and these are included in this exclusion list).  I would agree that the branch itself should be refactored a bit; there are terms not in Mondo that are in the DSM-5 PDF and others that have been changed.

I'm going to close this one as completed! thanks so much again @matentzn!


Create repo before cloning with Repository.clone_from

Fix https://github.com/rwightman/pytorch-image-models/issues/1546 by first creating the repo if it doesn't exist yet.
cc @nateraw @rwightman

I installed huggingface_hub main and timm main, and was able to successfully push a model as before, btw. here it is.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@Wauplin tx, there currently aren't unit tests for this or many utility fns, the unit tests focus on ensuring models don't break right now... but open to PRs re any tests for hub functionality...

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


[FEATURE]Support CSI fsGroupPolicy

https://kubernetes.io/blog/2020/12/14/kubernetes-release-1.20-fsgroupchangepolicy-fsgrouppolicy/
Ref:
#1221
https://github.com/longhorn/longhorn/issues/1842#issuecomment-749522327
Note: Manual test plan is required because:

We haven't had the v1.20 test environment ready yet.
Need to make sure we can speed up the fsgroup change

Applicable Kubernetes version: v1.20+

From the Kubernetes' code perspective, it looks like everything should work fine.
There are 3 cases we need to consider:

Kubernetes v1.19 with CSIVolumeFSGroupPolicy enabled
Kubernetes v1.20 with CSIVolumeFSGroupPolicy disabled
Kubernetes v1.20 with CSIVolumeFSGroupPolicy enabled

Case#1 and Case#3 should work because when CSIVolumeFSGroupPolicy is enabled, Kubernetes automatically fills in the default value for fsGroupPolicy: ReadWriteOnceWithFSType. Therefore, we don't have this error
Case#2 should also work because when CSIVolumeFSGroupPolicy is disabled, Kubernetes automatically clears FSGroupPolicy field but it checks the feature CSIVolumeFSGroupPolicy gate here and returns immediately. Therefore,  we shouldn't have the error neither.

@Khushboo already verified that Case#1 and Case#3 working as expected.
We are verifying Case#2

Validated the longhorn basic functionality with v1.20, It looks good.
As mentioned in the https://github.com/longhorn/longhorn/issues/2131#issuecomment-750746830 Longhorn is validated with the below cases.

Kubernetes v1.19 with CSIVolumeFSGroupPolicy enabled
Kubernetes v1.20 with CSIVolumeFSGroupPolicy disabled
Kubernetes v1.20 with CSIVolumeFSGroupPolicy enabled

Running the integration test on the case#2, will update the result.

Validated the longhorn basic functionality with v1.20, It looks good.
As mentioned in the https://github.com/longhorn/longhorn/issues/2131#issuecomment-750746830 Longhorn is validated with the below cases.

Kubernetes v1.19 with CSIVolumeFSGroupPolicy enabled
Kubernetes v1.20 with CSIVolumeFSGroupPolicy disabled
Kubernetes v1.20 with CSIVolumeFSGroupPolicy enabled

Running the integration test on the case#2, will update the result.

@PhanLe1010 the fuzzer you linked  to is code used only for testing, as recognized by the fact of the usage of random here :)
https://github.com/kubernetes/kubernetes/blob/af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38/pkg/apis/storage/fuzzer/fuzzer.go#L43
So in the case where the feature gate exists and the driver object exists as well as not having defined a FSgroup setting, we will trigger this error:
https://github.com/kubernetes/kubernetes/blob/af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38/pkg/volume/csi/csi_plugin.go#L912

@PhanLe1010 the fuzzer you linked  to is code used only for testing, as recognized by the fact of the usage of random here :)
https://github.com/kubernetes/kubernetes/blob/af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38/pkg/apis/storage/fuzzer/fuzzer.go#L43
So in the case where the feature gate exists and the driver object exists as well as not having defined a FSgroup setting, we will trigger this error:
https://github.com/kubernetes/kubernetes/blob/af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38/pkg/volume/csi/csi_plugin.go#L912

@joshimoo

the fuzzer you linked to is code used only for testing, as recognized by the fact of the usage of random here :)
https://github.com/kubernetes/kubernetes/blob/af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38/pkg/apis/storage/fuzzer/fuzzer.go#L43
So in the case where the feature gate is enabled and the driver object exists as well as not having defined a .Spec.FSGroupPolicy setting, we will trigger this error:
https://github.com/kubernetes/kubernetes/blob/af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38/pkg/volume/csi/csi_plugin.go#L912

Thank you for pointing out the mistake in the coded link. Yeah, it is testing code. The default values are filled here instead https://github.com/kubernetes/kubernetes/blob/af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38/pkg/apis/storage/v1/defaults.go#L56. So it is not possible that  the feature gate is enabled and the driver object exists as well as not having defined a .Spec.FSGroupPolicy setting. Moreover, if K8s don't fill the .Spec.FSGroupPolicy, it will break many CSI providers. I think that K8s must already consider that.

Do we define .Spec.FSGroupPolicy when creating the driver object?

No, we don't define. We actually cannot define it. The reason is that we are using client-go v1.16, and it doesn't have the field .Spec.FSGroupPolicy in CSIDriver struct. So, we depend on the K8s to fill in the default value for .Spec.FSGroupPolicy.

@joshimoo

the fuzzer you linked to is code used only for testing, as recognized by the fact of the usage of random here :)
https://github.com/kubernetes/kubernetes/blob/af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38/pkg/apis/storage/fuzzer/fuzzer.go#L43
So in the case where the feature gate is enabled and the driver object exists as well as not having defined a .Spec.FSGroupPolicy setting, we will trigger this error:
https://github.com/kubernetes/kubernetes/blob/af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38/pkg/volume/csi/csi_plugin.go#L912

Thank you for pointing out the mistake in the coded link. Yeah, it is testing code. The default values are filled here instead https://github.com/kubernetes/kubernetes/blob/af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38/pkg/apis/storage/v1/defaults.go#L56. So it is not possible that  the feature gate is enabled and the driver object exists as well as not having defined a .Spec.FSGroupPolicy setting. Moreover, if K8s don't fill the .Spec.FSGroupPolicy, it will break many CSI providers. I think that K8s must already consider that.

Do we define .Spec.FSGroupPolicy when creating the driver object?

No, we don't define. We actually cannot define it. The reason is that we are using client-go v1.16, and it doesn't have the field .Spec.FSGroupPolicy in CSIDriver struct. So, we depend on the K8s to fill in the default value for .Spec.FSGroupPolicy.

Analysis
1. Running pod as non-root user and the problem
It is good practice to run pod as a non-root user to prevent bugs or malicious code from taking over the system. An example of a pod that runs as non-root user could be:
apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
  volumes:
  - name: sec-ctx-vol
    emptyDir: {}
  containers:
  - name: sec-ctx-demo
    image: busybox
    command: [ "sh", "-c", "sleep 1h" ]
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
    securityContext:
      allowPrivilegeEscalation: false

The above YAML file tells Kubernetes to run every process of every container inside the pod with user ID 1000 and primary group ID 3000. So far so good, processes are not running as root. However, there is a problem here. If the volume sec-ctx-vol contains files/directories that the user ID 1000 and the group ID 3000 don't have permission to access, the processes in the pod cannot access those files/directories. In the worst case, if the root of the volume sec-ctx-vol don't have read/write permission for user ID 1000 and group ID 3000, all processes cannot access the volume.
2.  fsGroup and the problem
To solve the above problem, users can specify the field fsGroup  as below:
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000

Upon seeing the field fsGroup: 2000, Kubernetes will take the following actions:

Make sure that all processes of the containers inside the pod are part of the supplementary group ID 2000
Any new files created in the volume sec-ctx-vol will be Group ID 2000
If fsType of the PersistentVolumes is defined and the PersistentVolumes's accessModes is RWO, then Kubernetes will attempt to modify the volume ownership and permissions (to be Group ID 2000 in this case) every time the volume is mounted to the pod.

So, adding the field fsGroup: 2000 solves the problem we see in section 1. However, fsGroup introduces a new problem as described in K8s doc:

The side-effect of setting fsGroup is that, each time a volume is mounted, Kubernetes must recursively chown() and chmod() all the files and directories inside the volume - with a few exceptions noted below. This happens even if group ownership of the volume already matches the requested fsGroup, and can be pretty expensive for larger volumes with lots of small files, which causes pod startup to take a long time.

The bad news is that there is no workaround for this problem in K8s version v1.19 and before. The good news is that K8s v1.20 provides 2 options to solve this problem as discussed in the following sections.
3. pod.spec.securityContext.fsGroupChangePolicy
In section 1, K8s never modifies the volume ownership and permissions before giving it to the pod. In section 2, K8s always recursively modifies the volume ownership and permissions before giving it to the pod. Each of them has its own problem as mentioned.
Since version v1.20,  Kubernetes introduces a new beta feature: the field fsGroupChangePolicy. When  fsGroupChangePolicy is set to OnRootMismatch, if the root of the volume already has the correct permissions, the recursive permission and ownership change will be skipped. It means that if users don't change the pod.spec.securityContext.fsGroup  between pod's restarts, K8s will only have to check the permissions and ownership of the root and the mounting process will be much faster compared to always recursively change the volumes' ownership and permissions.
Note that K8s v1.20 still allows users to keep the previous behavior. By setting fsGroupChangePolicy to always, K8s always changes permission and ownership of the volume when volume is mounted.
Note: if users don't set fsGroupChangePolicy, it will be the same as setting fsGroupChangePolicy to always
4. csiDriver.spec.fsGroupPolicy
For certain volume types, such as NFS or Gluster, the cluster doesn’t perform recursive permission changes even if the pod has a fsGroup. So, K8s v1.20 even optimizes the volume's permissions checking process further by allowing the CSI storage providers to explicitly indicate whether or not they support modifying a volume's ownership or permissions when the volume is being mounted.
CSI providers can specify the following values for csiDriver.spec.fsGroupPolicy:

None: Indicates that volumes will be mounted with no modifications, as the CSI volume driver does not support these operations.
File: Indicates that the CSI volume driver supports volume ownership and permission change via fsGroup, and Kubernetes may use fsGroup to change permissions and ownership of the volume to match user requested fsGroup in the pod's SecurityPolicy regardless of fstype or access mode.
ReadWriteOnceWithFSType: Indicates that volumes will be examined to determine if volume ownership and permissions should be modified to match the pod's security policy. Changes will only occur if the fsType is defined and the persistent volume's accessModes contains ReadWriteOnly.

Note:

If undefined, csiDriver.spec.fsGroupPolicy will default to ReadWriteOnceWithFSType, keeping the previous behavior.
when csiDriver.spec.fsGroupPolicy is set to None or File, it will overwrite the setting for pod.spec.securityContext.fsGroupChangePolicy. Users of the CSI provider cannot control the volume's permissions checking process in those cases.

Conclusion

As tested in https://github.com/longhorn/longhorn/issues/2131#issuecomment-754182609, Longhorn works with Kubernetes v1.20 by default. Also, Kubernetes automatically set  csiDriver.spec.fsGroupPolicy=ReadWriteOnceWithFSType for Longhorn CSI driver since Longhorn doesn't specify it.
Longhorn should not change the csiDriver.spec.fsGroupPolicy to a value different than the default value ReadWriteOnceWithFSType because Longhorn only provides block device storage. Base on the filesystem and volume access mode, the users of Longhorn block devices should be able to control the volume's permissions checking process by setting pod.spec.securityContext.fsGroupChangePolicy. If Longhorn sets csiDriver.spec.fsGroupPolicy to None or File, it will overwrite the setting pod.spec.securityContext.fsGroupChangePolicy.

In a nutshell, Longhorn doesn't need to do anything to support csiDriver.spec.fsGroupPolicy and pod.spec.securityContext.fsGroupChangePolicy in Kubernetes v1.20

Testing
The above analysis is supported by the linked documents and the below tests.
1. Running pod as non-root but don't set pod.spec.securityContext.fsGroup

Remove the read, write, and execute for other users on Longhorn volume's root. I.e., change permission for Longhorn volume's root to drwxr-x---. In order to do that, we will first running the flowing pod as root:apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sec-ctx-vol-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  storageClassName: longhorn
  resources:
    requests:
      storage: 30Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
  namespace: default
spec:
  restartPolicy: Always
  containers:
  - name: sec-ctx-demo
    command: [ "sh", "-c", "sleep 1h" ]
    image: ubuntu
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
  volumes:
  - name: sec-ctx-vol
    persistentVolumeClaim:
      claimName: sec-ctx-vol-pvc

exec into the pod, and create some fileroot@security-context-demo:/data# id
uid=0(root) gid=0(root) groups=0(root)
root@security-context-demo:~# cd /data/
root@security-context-demo:/data# ls -l
total 4
drwxr-xr-x 3 root root 4096 Feb 15 00:05 demo
root@security-context-demo:/data# ls demo/ -l
total 16
drwx------ 2 root root 16384 Feb 15 00:05 lost+found
root@security-context-demo:/data# chmod 750 demo/
root@security-context-demo:/data# ls -l
total 4
drwxr-x--- 3 root root 4096 Feb 15 00:05 demo

Now delete the above pod, re-deploy the pod as non-root user by adding runAsUser, and runAsGroup . Note that we don't add fsGroup :apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
  namespace: default
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
  restartPolicy: Always
  containers:
  - name: sec-ctx-demo
    command: [ "sh", "-c", "sleep 1h" ]
    image: ubuntu
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
  volumes:
  - name: sec-ctx-vol
    persistentVolumeClaim:
      claimName: sec-ctx-vol-pvc

Exec into the pod. Observe that we cannot access directory /data/demo as user ID 1000 and primary group ID 3000I have no name!@security-context-demo:/$ id
uid=1000 gid=3000 groups=3000
I have no name!@security-context-demo:/$ ls /data/ -l
total 4
drwxr-x--- 3 root root 4096 Feb 15 00:05 demo
I have no name!@security-context-demo:/$ ls /data/demo -l
ls: cannot open directory '/data/demo': Permission denied
I have no name!@security-context-demo:/$ touch /data/demo/file.txt
touch: cannot touch '/data/demo/file.txt': Permission denied

2. Running pod as non-root and set pod.spec.securityContext.fsGroup

Delete the above pod. Re-deploy the pod and add fsGroup: 2000  as bellow:apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
  namespace: default
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  restartPolicy: Always
  containers:
  - name: sec-ctx-demo
    command: [ "sh", "-c", "sleep 1h" ]
    image: ubuntu
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
  volumes:
  - name: sec-ctx-vol
    persistentVolumeClaim:
      claimName: sec-ctx-vol-pvc

Exec into the pod, we can see that Kubernetes did 3 actions as mentioned in section # 2 in the above analysis.I have no name!@security-context-demo:/$ id
uid=1000 gid=3000 groups=3000,2000
I have no name!@security-context-demo:/$ ls /data/ -l
total 4
drwxrws--- 3 root 2000 4096 Feb 15 00:05 demo
I have no name!@security-context-demo:/$ cd /data/demo/
I have no name!@security-context-demo:/data/demo$ ls -l
total 16
drwxrws--- 2 root 2000 16384 Feb 15 00:05 lost+found
I have no name!@security-context-demo:/data/demo$ echo "hello world" > file.txt
I have no name!@security-context-demo:/data/demo$ ls -l
total 20
-rw-r--r-- 1 1000 2000    12 Feb 15 00:46 file.txt
drwxrws--- 2 root 2000 16384 Feb 15 00:05 lost+found

One key point to take away is that Kubernetes did recursively change ownership and permissions for every file/directory in the Longhorn volume.
Now, to see the problem with recursively change ownership and permissions. Let's create a lot of files inside /data/demo/files. We can use the following script to generate the files:#!/bin/bash
file_prefix="longhorn-file-"

POSITIONAL=()
while [[ $# -gt 0 ]]; do
        key="$1"
        case $key in
                -c|--count)
                count="$2"
                shift # past argument
                shift # past value
                ;;
                -h|--help)
                help="true"
                shift
                ;;
                *)
                echo "Error! invalid flag: ${key}"
                help="true"
                break
                ;;
        esac
done

usage () {
        echo "USAGE: $0 --count 10000"
        echo "  [-c|--count] number of file to be created inside the current folder"
        echo "  [-h|--help] Usage message"
}

if [[ $help ]]; then
        usage
        exit 0
fi

set -e -x

if [[ $count ]]; then
    i="0"
        while [ $i -lt $count ]
    do
    dd if=/dev/urandom of="$file_prefix$i" count=1 bs=512
    i=$[$i+1]
    done
fi

Copy the script into /data/demo/files/create_file.sh.kubectl cp create-file.sh default/security-context-demo:data/demo/files

Create 500000 files of 512 bytes inside data/demo/files:I have no name!@security-context-demo:/$ cd /data/demo/files/
I have no name!@security-context-demo:/data/demo/files$ ls -l
total 4
-rwxr-xr-x 1 1000 2000 663 Feb 15 01:02 create-file.sh
create-file.sh
I have no name!@security-context-demo:/data/demo/files$ ./create-file.sh --count 500000

Let's see how long it takes Kubernetes to remount Longhorn volume. Let's delete the pod and re-deploy it as in step 1 of this test case. Observe that it takes 3m30s for Kunertes to finish mounting the Longhorn volume.
Observe that it takes 15 mins for Kunertes to finish mounting the Longhorn volume if we create 2 millions files inside /data/demo/files/

3.  Set pod.spec.securityContext.fsGroupChangePolicy= OnRootMismatch

Delete the pod. Re-deploy the pod with pod.spec.securityContext.fsGroupChangePolicy= OnRootMismatchapiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
  namespace: default
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
    fsGroupChangePolicy: "OnRootMismatch"
  restartPolicy: Always
  containers:
  - name: sec-ctx-demo
    command: [ "sh", "-c", "sleep 1h" ]
    image: ubuntu
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
  volumes:
  - name: sec-ctx-vol
    persistentVolumeClaim:
      claimName: sec-ctx-vol-pvc

Observe that it takes only 13s for Kubernetes to finish mounting the Longhorn volume and the pod is in running state. It is because we didn't change the pod.spec.securityContext.fsGroup. So, it is matched with the volume root group ownership and Kubernetes doesn't recursively perform the chown()and chmod()
Let's change the fsGroup to 5000, delete the pod, and re-deploy the pod as:apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
  namespace: default
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 5000
    fsGroupChangePolicy: "OnRootMismatch"
  restartPolicy: Always
  containers:
  - name: sec-ctx-demo
    command: [ "sh", "-c", "sleep 1h" ]
    image: ubuntu
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
  volumes:
  - name: sec-ctx-vol
    persistentVolumeClaim:
      claimName: sec-ctx-vol-pvc

Because the fsGroup doesn't match the volume root group ownership. Kubernetes recursively perform the chown()and chmod(). Observe that it takes 3m30s for Kunertes to finish mounting the Longhorn volume.

4.  Test csiDriver.spec. fsGroupPolicy
So far we have seen the behavior of Kubernetes when the csiDriver.spec.fsGroupPolicy is set to ReadWriteOnceWithFSType which is the default value. Let's see what happens when we  csiDriver.spec.fsGroupPolicy is set to None which indicates that volumes will be mounted with no modifications.

Delete the pod from the previous step. Delete the PVC sec-ctx-vol-pvc which will automatically delete the corresponding PV and Longhorn volume.

Delete Longhorn CSI driver CR
kubectl delete csidriver driver.longhorn.io

Re-create Longhorn CSI driver CR with spec.fsGroupPolicy set to None:
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  annotations:
    driver.longhorn.io/kubernetes-version: v1.20.2+k3s1
    driver.longhorn.io/version: v1.1.0
  creationTimestamp: "2021-02-11T20:01:07Z"
  name: driver.longhorn.io
  resourceVersion: "50138"
  uid: 92358d9f-f82c-4157-975e-ec0120955591
spec:
  attachRequired: true
  # fsGroupPolicy: None
  podInfoOnMount: true
  volumeLifecycleModes:
  - Persistent

Deploy the following pod:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sec-ctx-vol-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  storageClassName: longhorn
  resources:
    requests:
      storage: 30Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
  namespace: default
spec:
  restartPolicy: Always
  containers:
  - name: sec-ctx-demo
    command: [ "sh", "-c", "sleep 1h" ]
    image: ubuntu
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
  volumes:
  - name: sec-ctx-vol
    persistentVolumeClaim:
      claimName: sec-ctx-vol-pvc

exec into the pod. Remove the read, write, and execute for other users on Longhorn volume's root. I.e., change permission for Longhorn volume's root to drwxr-x---.
root@security-context-demo:/data# id
uid=0(root) gid=0(root) groups=0(root)
root@security-context-demo:~# cd /data/
root@security-context-demo:/data# chmod 750 demo/
root@security-context-demo:/data# ls -l
total 4
drwxr-x--- 3 root root 4096 Feb 15 00:05 demo

Now delete the above pod, re-deploy the pod as:
apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
  namespace: default
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
    fsGroupChangePolicy: "Always"
  restartPolicy: Always
  containers:
  - name: sec-ctx-demo
    command: [ "sh", "-c", "sleep 1h" ]
    image: ubuntu
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
  volumes:
  - name: sec-ctx-vol
    persistentVolumeClaim:
      claimName: sec-ctx-vol-pvc

Exec into the pod. Observe that we cannot access directory /data/demo as user ID 1000, primary group ID 3000, and supplemental group ID 2000
I have no name!@security-context-demo:/$ id
uid=1000 gid=3000 groups=3000,2000
I have no name!@security-context-demo:/$ ls /data/ -l
total 4
drwxr-x--- 3 root root 4096 Feb 15 00:05 demo
I have no name!@security-context-demo:/$ ls /data/demo -l
ls: cannot open directory '/data/demo': Permission denied
I have no name!@security-context-demo:/$ touch /data/demo/file.txt
touch: cannot touch '/data/demo/file.txt': Permission denied

We can see that setting csiDriver.spec.fsGroupPolicy to None will overwrite the setting pod.spec.securityContext.fsGroupChangePolicy. From Longhorn perspective, we don't want to overwrite this as explained in section # 4 in the above analysis.

Thanks, @PhanLe1010 , it's a well-written explanation.
I also feel that we should put this either into our documentation or KB.
After that, I will leave it for QA to verify the result, then we can close it.

FYI: During the team meeting, we decide that we should write a KB article for the issue.

Hey Folks,
I'm running into this problem and it seems as though something needs to be set in the csidriver. Even with the securityContext defined in the pod I'm still getting the kubernetes.io/csi: expected valid fsGroupPolicy, received nil value or empty string.
I've got this defined in spec.securityContext
  securityContext:
    fsGroup: 1000
    runAsGroup: 1000
    runAsUser: 1000

@carpenike Which Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version are you using? Also, which Longhorn version?

@PhanLe1010 --
Kubeadm deployed on top of Ubuntu. Longhorn 1.1.0.
ryan@h2-0:~$ sudo kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:25:59Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Heh, realized that only had the kubeadm version...
Kube version is 1.20.2-00:
ryan@h2-0:~$ dpkg --list |grep kube
hi  kubeadm                              1.20.2-00                           amd64        Kubernetes Cluster Bootstrapping Tool
hi  kubectl                              1.20.2-00                           amd64        Kubernetes Command Line Tool
hi  kubelet                              1.20.2-00                           amd64        Kubernetes Node Agent

@carpenike Thanks
Can you provide us the CSI Driver yaml
kubectl get csidriver driver.longhorn.io -o yaml

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  annotations:
    driver.longhorn.io/kubernetes-version: v1.19.8
    driver.longhorn.io/version: v1.1.0
  creationTimestamp: "2021-02-23T05:43:48Z"
  managedFields:
  - apiVersion: storage.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:driver.longhorn.io/kubernetes-version: {}
          f:driver.longhorn.io/version: {}
      f:spec:
        f:attachRequired: {}
        f:podInfoOnMount: {}
        f:volumeLifecycleModes: {}
    manager: longhorn-manager
    operation: Update
    time: "2021-02-23T05:43:48Z"
  - apiVersion: storage.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:fsGroupPolicy: {}
    manager: kubectl-edit
    operation: Update
    time: "2021-02-24T17:38:28Z"
  name: driver.longhorn.io
  resourceVersion: "1510508"
  selfLink: /apis/storage.k8s.io/v1/csidrivers/driver.longhorn.io
  uid: de782a0c-3da5-49dd-8178-b1e4ba5a1133
spec:
  attachRequired: true
  podInfoOnMount: true
  volumeLifecycleModes:
  - Persistent

From the yaml file, it is weird that Longhorn is saying that your Kubernetes version is v1.19.8
 driver.longhorn.io/kubernetes-version: v1.19.8

Yeah, it was an in-place upgrade.

ryan on ﴱ ryan-k8s in ~ ❯ kubectl get no -o wide                                                                                [18:08:26]
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION     CONTAINER-RUNTIME
h2-0     Ready    master   41h   v1.20.4   <IP_ADDRESS>   <none>        Ubuntu 20.10   5.8.0-44-generic   containerd://1.3.9
r610-0   Ready    <none>   41h   v1.20.4   <IP_ADDRESS>   <none>        Ubuntu 20.10   5.8.0-43-generic   containerd://1.3.9
r610-1   Ready    <none>   41h   v1.20.4   <IP_ADDRESS>   <none>        Ubuntu 20.10   5.8.0-43-generic   containerd://1.3.9
r610-2   Ready    <none>   41h   v1.20.4   <IP_ADDRESS>   <none>        Ubuntu 20.10   5.8.0-43-generic   containerd://1.3.9
r720-0   Ready    <none>   41h   v1.20.4   <IP_ADDRESS>   <none>        Ubuntu 20.10   5.8.0-43-generic   containerd://1.3.9

By default Kubernetes v1.20 should automatically field in the field fsGroupPolicy for Longhorn CSI Driver CR. So the error kubernetes.io/csi: expected valid fsGroupPolicy, received nil value or empty string shouldn't happen.
I am guessing order of the upgrading process might contribute to the issue. Can you give more detail about what you mean by in-place upgrade ?

Yup! They're ubuntu nodes so it's just an apt update process. Kubelet restarts on each node and reports in to the control-plane with the updated version.

Should I just remove the driver and re-create it?

@carpenike Yes, the workaround is just simple as that. You can also just delete the pod longhorn-driver-deployer-xxxx and it will be restarted and recreate the CSIDrive CR.
However, is it possible that you can hold on a second? We would like to learn about how to reproduce the issue. Is would be great if you can help us?

Sure no problem. :)

What is your upgrade order?
Is it

worker nodes (drain, upgrade kubelet and kubectl, uncordon) -> control plane nodes
or
ontrol plane nodes ->  worker nodes (drain, upgrade kubelet and kubectl, uncordon)

Generally the first one.

Ok, that explains why Longhorn is saying that your Kubernetes version is v1.19.8 in the CSI Drive CR. Since at the time the longhorn-driver-deployer pod starts it see that K8s server has the old v1.19 version. Latter on, when the K8s server is upgraded to v1.20. The longhorn-driver-deployer pod is not restarted so it doesn't see the new v1.20 version and it doesn't update the CSI Driver CR.

Can you try the workaround:

delete the pod longhorn-driver-deployer-xxxx and it will be restarted and recreate the CSIDrive CR.

Done. It seems like longhorn is caching the k8s version?
ryan on ﴱ ryan-k8s in ~ ❯ kubectl logs -n longhorn-system longhorn-driver-deployer-666c84fbb7-5b4qp                                                                                [18:50:46]
W0224 23:49:14.536576       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2021-02-24T23:49:14Z" level=debug msg="Deploying CSI driver"
time="2021-02-24T23:49:14Z" level=debug msg="proc cmdline detection pod discover-proc-kubelet-cmdline in phase: Pending"
time="2021-02-24T23:49:15Z" level=debug msg="proc cmdline detection pod discover-proc-kubelet-cmdline in phase: Pending"
time="2021-02-24T23:49:16Z" level=debug msg="proc cmdline detection pod discover-proc-kubelet-cmdline in phase: Pending"
time="2021-02-24T23:49:17Z" level=debug msg="proc cmdline detection pod discover-proc-kubelet-cmdline in phase: Pending"
time="2021-02-24T23:49:18Z" level=debug msg="proc cmdline detection pod discover-proc-kubelet-cmdline in phase: Running"
time="2021-02-24T23:49:19Z" level=debug msg="proc cmdline detection pod discover-proc-kubelet-cmdline in phase: Running"
time="2021-02-24T23:49:20Z" level=info msg="Proc found: kubelet"
time="2021-02-24T23:49:20Z" level=info msg="Try to find arg [--root-dir] in cmdline: [/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --node-ip r720-0.holthome.net ]"
time="2021-02-24T23:49:20Z" level=warning msg="Cmdline of proc kubelet found: \"/usr/bin/kubelet\x00--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf\x00--kubeconfig=/etc/kubernetes/kubelet.conf\x00--config=/var/lib/kubelet/config.yaml\x00--container-runtime=remote\x00--container-runtime-endpoint=/run/containerd/containerd.sock\x00--node-ip\x00r720-0.holthome.net\x00\". But arg \"--root-dir\" not found. Hence default value will be used: \"/var/lib/kubelet\""
time="2021-02-24T23:49:20Z" level=info msg="Detected root dir path: /var/lib/kubelet"
time="2021-02-24T23:49:20Z" level=info msg="Upgrading Longhorn related components for CSI v1.1.0"
time="2021-02-24T23:49:20Z" level=debug msg="Detected CSI Driver driver.longhorn.io CSI version v1.1.0 Kubernetes version v1.19.8 has already been deployed"
time="2021-02-24T23:49:20Z" level=debug msg="Detected service csi-attacher CSI version v1.1.0 Kubernetes version v1.19.8 has already been deployed"
time="2021-02-24T23:49:20Z" level=debug msg="Detected deployment csi-attacher CSI version v1.1.0 Kubernetes version v1.19.8 has already been deployed"
time="2021-02-24T23:49:20Z" level=debug msg="Detected service csi-provisioner CSI version v1.1.0 Kubernetes version v1.19.8 has already been deployed"
time="2021-02-24T23:49:20Z" level=debug msg="Detected deployment csi-provisioner CSI version v1.1.0 Kubernetes version v1.19.8 has already been deployed"
time="2021-02-24T23:49:20Z" level=debug msg="Detected service csi-resizer CSI version v1.1.0 Kubernetes version v1.19.8 has already been deployed"
time="2021-02-24T23:49:20Z" level=debug msg="Detected deployment csi-resizer CSI version v1.1.0 Kubernetes version v1.19.8 has already been deployed"
time="2021-02-24T23:49:20Z" level=debug msg="Detected service csi-snapshotter CSI version v1.1.0 Kubernetes version v1.19.8 has already been deployed"
time="2021-02-24T23:49:20Z" level=debug msg="Detected deployment csi-snapshotter CSI version v1.1.0 Kubernetes version v1.19.8 has already been deployed"
time="2021-02-24T23:49:20Z" level=debug msg="Detected daemon set longhorn-csi-plugin CSI version v1.1.0 Kubernetes version v1.19.8 has already been deployed"
time="2021-02-24T23:49:20Z" level=debug msg="CSI deployment done"

CSIDriver looks the same too:
ryan on ﴱ ryan-k8s in ~ ❯ kubectl get csidriver driver.longhorn.io -o yaml                                                                                [18:51:59]
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  annotations:
    driver.longhorn.io/kubernetes-version: v1.19.8
    driver.longhorn.io/version: v1.1.0
  creationTimestamp: "2021-02-23T05:43:48Z"
  managedFields:
  - apiVersion: storage.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:driver.longhorn.io/kubernetes-version: {}
          f:driver.longhorn.io/version: {}
      f:spec:
        f:attachRequired: {}
        f:podInfoOnMount: {}
        f:volumeLifecycleModes: {}
    manager: longhorn-manager
    operation: Update
    time: "2021-02-23T05:43:48Z"
  - apiVersion: storage.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:fsGroupPolicy: {}
    manager: kubectl-edit
    operation: Update
    time: "2021-02-24T17:38:28Z"
  name: driver.longhorn.io
  resourceVersion: "1510508"
  selfLink: /apis/storage.k8s.io/v1/csidrivers/driver.longhorn.io
  uid: de782a0c-3da5-49dd-8178-b1e4ba5a1133
spec:
  attachRequired: true
  podInfoOnMount: true
  volumeLifecycleModes:
  - Persistent

No, it fetches the info from the Kube API Server https://github.com/longhorn/longhorn-manager/blob/bb8a72669a61e36ab70d25651b9c8f67c5c7fb2b/csi/deployment_util.go#L158
Hmm

Can you try:
kubectl proxy --port=8080
curl http://localhost:8080/version

and
kubectl version

Figured it out -- the k8s kubelets had upgraded to 1.20.4 but the underlying API and services had not. Ran through the kubeadm upgrade process and now things are starting up as expected. Deleted the longhorn-driver and it's re-deploying everything.
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
This would likely not have been encountered with a fresh 1.20.x cluster.

Verified the scenarios from https://github.com/longhorn/longhorn/issues/2131#issuecomment-778897129, all the explanations hold good.
Validations - Pass

The pod with fsGroupChangePolicy: "Always" or no fsGroupChangePolicy or with different fsGroups and csiDriver.spec.fsGroupPolicy: ReadWriteOnceWithFSType takes around 6 min to mount the path containing 500000 files. This confirms the recursive change in the ownership and permissions.
The pod with fsGroupChangePolicy: "OnRootMismatch" and csiDriver.spec.fsGroupPolicy: ReadWriteOnceWithFSType takes around 18 sec to mount the path containing 500000 files.
If csiDriver.spec.fsGroupPolicy is set to None, the non-root users are unable to access the files in the mount path.
If csiDriver.spec.fsGroupPolicy is set to File, the pod always take around 6 min to mount the path irrespective of the fsGroupChangePolicy value.
On upgrade of the cluster from K8s v1.19.8 to K8s v1.20.4, the fsGroupPolicy: ReadWriteOnceWithFSType gets updated in CSIDriver driver.longhorn.io.


Adjust code to the API changes in V8 6.5.

V8 6.5 removed deprecated functions (see https://bit.ly/v8-api-changes).
Changes that apply to gjstest:

TryCatch constructor requires an isolate.
Function::NewInstance returns MaybeLocal<>.
ScriptCompiler::CompileUnbound is renamed to CompileUnboundScript
and returns a MaybeLocal<>.
Value::ToUint32 requires a context and returns a Maybe<>.

Thanks!


Update concepts-monitoring.md (Typo "flexible server"→"Flexible Server")

https://github.com/MicrosoftDocs/azure-databases-docs/blob/main/articles/mysql/flexible-server/concepts-monitoring.md
#PingMSFTDocs

@code-sidd - related #12


Ability to search for a text

Currently I have a pretty big diagram and I am looking for an ability to quickly search for a given text but I am unable to find a way.
The browser search(Cmd/Ctrl + f) isn't helping.
Is there a way to do so in Excalidraw?

Not currently. Similar feature is planned for http://plus.excalidraw.com/, but we may as well start with the support here.
I imagine we can hijack cmd/ctrl-f since there's not much keeping the native browser behavior.

Sure.
Not sure how easy or difficult is to search for a text on the canvas. But I am eagerly looking forward to this feature.

This feature would consist of 3 things:

Search for match across elements' texts
Difficulty: easy to medium depending on whether we want to lexemize the tokens so that e.g. searching for catch matches caught (if so, we'll likely want to use a search library for this).

Build a search dialog
Difficulty: medium

Support highlighting matches on canvas and scrolling them into view
Difficulty: medium to hard (finding and scrolling to target text element position is easy, but finding exact position and dimensions of matched word may not be easy).

@dwelle can I take this one?

Sure thing!
Let's implement it in stages, even if end up doing it in one PR.
For (1) I'd use something like https://github.com/bvaughn/js-search (he suggests using https://github.com/jedp/porter-stemmer for stemming). Both lightweight, but we can put them in conjunction with our implem code into a module that we dynamically import them once user starts to search.
Once that's done we can discuss next steps. Thoughts?

Sounds good! No questions right now - I'll reach out if something comes up

I'm not sure on the value of doing any fuzzy searching, sounds really hard to do in a localised way. What's the use case for that? I think exact text search (like the browser find) is 95% of the use case.

we can start with plain search and improve on later. But fuzzy/fulltext search shouldn't be hard either way, once we nail down the plumbing and UI stuff.

As far as I know, fuzzy search is useful in two cases:

to tolerate typos and capitalization, e.g. "excaldraw" would match "Excalidraw"
to query with non-exact phrases, e.g. "search text" would match "Ability to search for a text"

And these are helpful for people who are using Excalidraw not just for casual drawing, but to jot down thoughts and drawings that they're going to revisit later. (I'm one.)
I used a fuzzy search library called Fuse.js in a project before, it achieves the two cases I describe and it's easy to use.
Of course, exact text search is still viable when fuzzy search is present.

@dwelle I was thinking about text search and came across this. I hacked together a prototype quickly (#5435) that includes

Ctrl/Cmd+F to bring up a dialog
type to search and scroll matched text element into view
navigate between matched text elements with Enter or buttons
fuzzy search

I'd love to hear what you think of it.

@dwelle @ad1992  I've implemented text search which scroll to near by part of matched content

I am interested to contribute to integrate the search functionality as i had developed it already.
Can I get an opportunity here.

@EswaraSatya According to the Contributing documentation, you could to open a pull request for your implementation, and try to contact an Excalidraw developer to discuss how to merge it.

I am interested to contribute to integrate the search functionality as i had developed it already. Can I get an opportunity here.

@EswaraSatya an additional feature could be an option to find text with colors used on the whiteboard to be able f.i. to search for red text.

Hi,
It means user can able to search the text which is red in color on the
white board if I am not wrong
Thanks,
Eswara Satya
On Thu, Sep 14, 2023, 8:44 PM sojusnik @.***> wrote:

I am interested to contribute to integrate the search functionality as i
had developed it already. Can I get an opportunity here.
@EswaraSatya https://github.com/EswaraSatya an additional feature could
be an option to find text with colors used on the whiteboard to be able
f.i. to search for red text.
—
Reply to this email directly, view it on GitHub
https://github.com/excalidraw/excalidraw/issues/3826#issuecomment-1719651724,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AYFGKBSBS2B4KRIRFCHTGOTX2MNN3ANCNFSM5AFSNCQA
.
You are receiving this because you were mentioned.Message ID:
@.***>

It means user can able to search the text which is red in color on the white board if I am not wrong

This is the idea :)
This would allow to not only search for text, but also for text written in a certain color, that is used on the whiteboard.

Where can I start can you please help me out

Hi,
I am wondering that whether someone is on this one? Seems to be an important feature but missing.
I hacked it a bit, and below is the result

If this seems to be good enough, I can clean it up and create PRs. If so please just assign to me.

@liyu1981 That looks A W E S O M E !!!

I have created my changes as a pull request at #7300. I do not mind either mine or @dragonman225 's effort to be merged, but if this needs me, just @ me.

We've recently released the full-text search, championed by @ryan-di!
https://twitter.com/excalidraw/status/1834254530025779450


Revert "Workaround: CentralPackageVersions sdk as override"

This reverts commit 0df70cc344d264ea792b60c1511bd7a4f700c426 per my comment in #933:

A few ways forward:

Keep the workaround (using the source-built SDK resolver) that ensures this won't repro.
Remove the workaround because it doesn't seem to repro. However, if it does repro, we're in no better position to find out what happened. The problem becomes a source of flakiness.

I'm relatively OK with this, we can add the workaround from 1 back in if it repros again.

Ha, this PR hit the problem again already (https://dnceng.visualstudio.com/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_build/results?buildId=59681):
2018-12-17T20:54:20.4750312Z   /tb/tarball_59681/src/msbuild/Directory.Build.targets(14,3): error : [SourceBuiltSdkResolver] 0 overrides found for 'Microsoft.Build.CentralPackageVersions'
2018-12-17T20:54:20.4750753Z   /tb/tarball_59681/src/msbuild/Directory.Build.targets(14,3): error : Failed to resolve SDK 'Microsoft.Build.CentralPackageVersions'. Package restore was successful but a package with the ID of "Microsoft.Build.CentralPackageVersions" was not installed.

Closing this PR: workaround stays in.
I have an idea for a different approach to inject the diagnostic nuget resolver to try to repro. Also, I'll submit them as PRs instead of isolated runs to make sure it's in the exact same environment.


In engineconfigurationparser, create RedactedJson()

For logging, a redacted version of SENZING_TOOLS_ENGINE_CONFIGURATION_JSON is needed to print the contents into a log.  Specifically the database URLs need to have passwords redacted.
Suggested approach:
In
https://github.com/Senzing/go-common/blob/main/engineconfigurationjsonparser/engineconfigurationjsonparser.go
add
func (parser *EngineConfigurationJsonParserImpl) RedactedJson(ctx context.Context) (string, error) {
    :
}

Which returns a string of JSON with the database URLs redacted.

Pull requests:

https://github.com/Senzing/go-common/pull/28


Use non-mutable structs for types

You should try to avoid using a mutable struct for your TimeDate and TimeDateZone.

good tip -- will do

done


[Feat]: Auto Commenting when an Issue is Closed

This feature aims to solve the problem of poor communication and lack of acknowledgment when issues are closed. By automatically commenting on closed issues, it ensures that contributors are informed about the closure and are thanked for their efforts. This helps maintain a positive and engaging community atmosphere, encouraging continued contributions and fostering good relationships between maintainers and contributors.

@Swarnendu0123 please assign me this issue under gssoc

Fulfill the prerequisites #24


Add Prism as a co-Lens

TL;DR This is an exploration for using enums with Prisms, similarly on how structs are used with Lenses.
Next step is try having them working comfortably with/as Widgets.
Widget, Data, and Lenses
A Lens is, as documented, "a datatype that gives access to a part of a larger data structure".
This is a useful concept/tool that often facilitates the handling and manipulation of data, of which reusable Widgets benefit from. A Widget may be reusable because it's defined to work with, let's say, a bool, and there are a lot of places and different libraries that could want to also use that Widget.
One detail of Widgets is that they need to have their instance defined inside some function, inside some scope - and note that such functions usually return -> impl Widget<MyWholeData> or something along these lines. Assume MyWholeData to be a struct that contains some stuff, including a bool, which we want our Widget to work it.
But we can't just use/return a Widget<bool> - a widget that works with / manipulates a bool - in a function that is expecting to return a Widget that work with / manipulate a MyWholeData. And we also cannot access the data by ourselves (to then hand it over to the widget), because let's say we might try having something like:
my_widget_that_works_with_bool()
    .actually_here_is_the_data_that_you_work_with(my_whole_data.my_bool)

But that is not possible, because here in our scope we are only defining (the instance of) the Widget, and only that. We don't actually have / hold an instance of the data (the my_whole_data, which would be an instance of MyWholeData) and therefore we can't really "access" the bool and "hand it over" to the Widget<bool>.
What we really do is to instruct (that instance of) the Widget<bool> on how to handle a MyWholeData data when given such data. And what it will do, is to access the inner bool contained in the MyWholeData by itself! So that's where we use lenses: it's that instruction itself.
(and then, after the correct lensing is applied, that Widget<bool> can now be treated as a Widget<MyWholeData>, which was the intention, which is the return type of the function)
So by using lenses, we can actually make Widgets reusable for other users, which are gonna have TheirOwnWholeData. This "lensing appliance" is actually a combination of Lenses and Widgets, where each combination keeps being a Widget - it's just now able to "work with" TheirOwnWholeData. That combination, in fact, simply applies the lensing properties on the data before calling the inner, reusable, widget.
It's worth noting that lenses (instruction about "accesses on data") compose with each other - think of it as combining multiple layers - and thus are able to provide a tree of "data access", of which each branch is used by one or more widgets, and all branches start at the root data, which is the "UltimateOuterWholeAppData".
Limitation
Currently, it's not possible to have "Lenses" into an enum's variant, although it's not clear if this should, actually, be avoided or not.
The Lensing concept on enums are called Prisms in some Haskell libraries, therefore I think they could show themselves useful for druid. Disclaimer: I myself don't know Haskell, and I'm new into the Lens/Prism concepts.
So now, assuming/given that it's possible to have "Lenses for enums", which are the Prisms, the next issue would be figuring if there is a way to comfortably use lenses and prisms with widgets.
Currently, for the linked PR, I think it is not, and so I don't think the Prism currently present a benefit, widget-wise.
As said, there is a drafty, initial PR for introducing Prisms, but again, their usage/interaction with/as Widgets is currently "bad".
Previous Work and Discussions
There are previous issues and discussions on this area:

https://github.com/linebender/druid/issues/789
https://github.com/linebender/druid/pull/751
Finnerale/druid-enum-switcher
Finnerale/druid-enum-helpers
Finnerale/druid-enums

Where druid-enums reaches an interesting possibility, by being able to create a Widget that is able to handle enums.
I tried comparing this approach and druid-enums, see https://github.com/linebender/druid/issues/1135#issuecomment-716332150.

Would be interesting to know what you prefer about that approach, compared to a derived matcher widget as offered by druid-enums (I've definitely to many repos for these...).

yeah haha actually I did see that one as well (I'll be adding it now);
I didn't read that much about the code, but I did a cargo-expand on the example to see how the transformation occurred, and this enum transformation happened:
enum AppState {
    Login(LoginState),
    Main(MainState),
}

// generated
struct App {
    login: Option<Box<dyn ::druid::Widget<(LoginState)>>>,
    main: Option<Box<dyn ::druid::Widget<(MainState)>>>,
    default_: Option<Box<dyn ::druid::Widget<AppState>>>,
    discriminant_: Option<::std::mem::Discriminant<AppState>>,
}

So I understood that the enum had to be structurized (with a separated discriminant), but I had mixed feelings by seeing Box<dyn _>.. as it's not very common for me to use them - but I understand that probably there is no other way other than structuring them as such.
There is one other point that I found curious, which is executed at the App initialization:
self.login = Some(Box::new(widget)); // on login(..)
self.main = Some(Box::new(widget)); // on main(..)

And in this time again, I don't know if it would be possible avoiding this, but it's not common to see "in practice" all variants of the old enum instantiated at the same time. There is some disadvantage with memory size ofc, but it is something that I'd commonly not expect for an enum (the original enum).
Then I did some reading on lenses for enums, and got into those pages talking about prisms.. so I thought about giving it a shot!

I had mixed feelings by seeing Box<dyn _>.. as it's not very common for me to use them

Well, Box<dyn Widget<T>> simply is the way to go if you need to store a widget where you don't now the exact type. This is also what Flex does for example.
https://github.com/linebender/druid/blob/0c63b5948c0bb36215ba915f2cf0aed48f812cab/druid/src/widget/flex.rs#L142-L153

it's not common to see "in practice" all variants of the old enum instantiated at the same time.

Yeah, I've though quite a bit about this and it could indeed be done differently by using constructor functions to initialize only the widget for the current variant. I'm also tempted to change it to work like that, but I doubt anyone will build a complex enough application, that utilizes enums so much that it would be possible for this to be a performance issue. And just passing in the widget you want to have there is a bit simpler.
That aside, how would this Prism help you with that? You would just end up with exactly the same "issue" right?

I think the combination of Lens and Prism, at least 'data'-wise (as in algebraic-data-wise), is working ok now.
There is a test called with_traversal, where it shows how they can be used together.
// .. structs and enums omitted ..

// given this instance
let mut bob = Person {
    name: "Bob".into(),
    addr: Addr::Extensive(ExtensiveAddr {
        street: "bob's street".into(),
        place: Place::Apartment(Apartment {
            building_number: 3,
            floor_number: -1,
            apartment_number: 4,
        }),
    }),
};

// and given this traversal
// traversal for A -> B -> C -> D -> E -> F
let person_apt_number = {
    use druid::{LensExt, PrismExt};
    (Person::addr) // A -> B
        .then_prism(Addr::extensive) // B -> C
        .then_lens(ExtensiveAddr::place) // C -> D
        .then_prism(Place::apartment) // D -> E
        .then_lens(Apartment::apartment_number) // E -> F
};

// we can get bob's apt number with
assert_eq!(Some(4), person_apt_number.get(&bob));

// and we can change it with:
assert_eq!(Some(()), person_apt_number.with_mut(&mut bob, |n| *n = 7));

Even putting aside some re-naming of traits etc, I didn't really test how Widgets would interact with those Prism/Traversal. I mean, I did an implementation of Widget for them, but didn't really tested it.
Note: Currently (PR-wise) - I think - the Prism definition could be renamed to PartialPrism and/or AffineTraversal, as it is a subset of what full prisms are (which also have the replacement/upgrade functions, which are currently in a different trait).

@cmyr yes, I wrote a clearer view of the problem (first post updated).
But to be honest, I still need to study the druid-enums as only now this direction of work is trying to start involving widgets. It's an unknown area for me, and it's also unknown to me if they will actually end up mixing well.
So that's why I was unable to clearly answer @Finnerale for those weeks (and I still am unable to answer you..).
So I'll try reading and actually understanding druid-enums, and then try to feedback!

It is possible that we could move lenses/prisms elsewhere, certainly, but it does increase the maintenance burden on us; we need to update another crate, be more careful about breaking changes, etcetera. My position would be to keep these things in druid until they are properly mature, and then we can consider moving them outside if it seems like there is a clear use-case.

Just did a little update on the backstory, and did some ergonomics update for the drafty PR (now they all use then methods for linking); I also started studying druid-enums (and I'm impressed on how tidy it is). I'm still experimenting (and also trying to do comparisons on how some translations of it would/could work with prisms), and hope to reach somewhere in a few weeks! (I've been moving my home between cities for the past weeks)

My try at explaining the issue intention, while adding info about the PR and some comparisons to druid-enums:
About
The linked PR is a possible implementation that tries to 1) extend the lensing concept and 2) create new and related widget wrappers.
Extension of Lenses
Lenses are instructions of, when given some outer/bigger type of data T1, accessing inner/smaller type of data T2. eg. access, from some struct T1, a given field T2.
As the accessed inner/smaller data is always one single (and existent) element, Lenses could be called something "Linear". eg. given a struct, it's fields are always present.
The extension adds a "type" of Lenses for enums. To access a given data T2 from a given variant of some enum T1, we must consider the possibility of that enum being in another variant, and thus T2 will be missing. So instead of being "Linear" (always 1 T2), this "type" of lenses could be called "Affine" (0-or-1 T2).
On some other lensing libraries (from Haskell, Scala, etc), that "type" of Lenses are called Prisms. They are Lenses' siblings.
Details
The usage of prisms is similar of lenses: deriving the correct trait for enums and then being able to combine prisms with each other (and also with lenses), and then attaching those combinations into widgets.
Outside of widgets, lenses can be used just to deal with data accesses, and the same is true for prisms. You can see those on the test files related to Lens/Prism trait derivation.
Accompanying Widgets
Lenses can be attached into Widgets and transform a Widget<T2> into a Widget<T1>, ie. transform a widget of some inner data into a widget of some outer data. More specifically, there is a Widget that wraps both some inner Widget<T2> and a Lens<T1, T2>, and it itself becomes a Widget<T1> by applying the lensing and then forwarding the calls into the inner widget.
There is a similar wrapper for Prisms, where a given Widget wraps both a Widget<T2> and a Prism<T1, T2> and becomes a Widget<T1>, same as for lens.
Except that some event/lifecycle/layout still needs to be worked out, since the data may now "cease to exist" and "come back again", which is different from a "Linear" perspective.
note: I myself am not intimate with how those concepts work, so this in itself already blocks this PR I think..
Besides the direct wrapper for Widget<T2> + Prism<T1, T2>, there are second-layer wrappers, which were adapted from how Finnerale/druid-enums made his Widgets that ultimately, from a T1, may access-then-forward T2, or a T2', or a T2'', .., for it's inner widgets, as each inner widget may have their own kind of T2.
Comparison to Druid-Enums
I'll try laying a comparison of this implementation ("prism case") to "druid-enums", to the best of my knowledge.
Code Example
This shows a comparison on how the wrapping widget is used in the prism case:
#[derive(Clone, Data, Prism)]
enum AppState {
    Login(LoginState),
    Main(MainState),
}

fn ui() -> impl Widget<AppState> {
    druid::widget::prism_wrap::Prisms2::new(
        login_ui().prism(AppState::login),
        main_ui().prism(AppState::main),
    )
    .controller(LoginController)
}

struct LoginController;
impl<W: Widget<AppState>> Controller<AppState, W> for LoginController {
  // ..
}

And on the druid-enums case:
#[derive(Clone, Data, Matcher)]
enum AppState {
    Login(LoginState),
    Main(MainState),
}

fn ui() -> impl Widget<AppState> {
    AppStateMatcher::new()
        .login(login_ui())
        .main(main_ui())
        .controller(LoginController)
}

struct LoginController;
impl Controller<AppState, AppStateMatcher> for LoginController {
  // ..
}

Differences
TL;DR: I think druid-enums is sufficient and efficient at making it possible to use enums within druid as Data and making them usable within widgets, and I see no advantage of having the prisms concepts when comparing it to druid-enums.

druid-enums is already usable externally from druid (which is a positive I think).

I think the same should be achieved for this prism case, if possible.

druid-enums focuses on Widgets instead of extending the lenses concept.

It is more concise (which is a positive I think).
druid-enums derives a specific Widget for the enum.

It is easier to implement Controllers for it, which is a positive.

For the prism case, the controller implementation must deal with some generics, which is a disadvantage.

It cleverly depends on the Discriminant of enum comparison.

This requires an extra discriminant field (which is a minor disadvantage).
The wrapper widget knows if the variant has changed, which is a positive.
For the prism case, the widget doesn't know by itself if the variant has changed, and thus can't change appropriate properties related to that change (which is a disadvantage).

It should be possible to also add the discriminant field and thus know when the discriminant was changed, as it is for druid-enums.

The match happens only once per method, which is a positive.

For the prism case, the outer wrapper widget simply repeatedly forward the calls to each inner widget.

In the prism case, each inner widget tries to match, which is a negative.

Although this could, I think, be optimized by the compiler, I don't know if this could/would always/ever happen.

druid-enums, at least currently, focuses on dynamic inner widgets.

If this were a disadvantage, I think druid-enums could alternatively work with non-dynamic inner widgets, it would just be generic over various parameters instead.
For the prism case, it currently is non-dynamic. To be dynamic, I think a specific Widget derivation would be required. The T2, T2', T2'' types would be required to be known beforehand, and thus I think the prisms would lose their purpose.

druid-enums could have a limitation when dealing with many layers of enums

In a brief test I did, I encountered this limitation, but I don't know if I did something wrong or whatnot.
I consider this to be a disadvantage, but I could be actually a minor one.
For de prism case, this works ok.

The conclusion is the TL;DR at the top, which is, druid-enums is a better way to go I think. Having said that, I personally find it interesting to see these prisms, but I wouldn't recommend it over druid-enums, unless there is some reason that I don't know of.

Great write-up / comparison! I hope to find some time today to take another look at the prism branch.

I'm excited about prisms too. Could they be made to work with runtime-defined length (for data in a Vec rather than an enum)?

This might be a neat way to solve the problem of how best to view a dynamic-length list.

@derekdreery I'm afraid they can't work outside of enums. Their intention is to take a tuple of different types (T2, T2', T2'', T2''', ..), each for a variant, and converge them into a single type T1 (the enum type itself).
When accessing through a prism, you're affinely-traversing, ie. you may get a single value (let's say, a value of type T2'') if your enum value (of type T1) currently is on the correct variant (for T2''); otherwise, you don't get any value out of it.
When accessing through a lens, you're linearly-traversing, ie. you can always get a field out of a struct. Note that you always get a single value (no more, no less).
Maybe you're referring to a complete "traversal" access. I don't know what kind of "layer" it is (eg. lens/prism/etc), but it's the idea that you can get 0, 1 or many values out of such access.
In that case, the return type for the trait would likely need to be impl Iterator<Item=X>, but this kind of return type for a function is currently not available for trait methods, so I guess this can't be done.. unless you want to use dyn around, I guess.
For reference, such return value is Option<X> for prisms, and just X for lenses - and that is pretty much all of the difference between lenses and prisms.
But I'm not sure, and I haven't actually seen how they deal with that in haskell!

I would like to share my thoughts why i think prism might be useful.
I developed a few widgets (MultiCheckbox and MultiRadio) in the widget-nursery and after some time i noticed, what i need are prisms. For this reason i created a toy prism trait in the nursery to try out those ideas.
I also tried to develop an editor for formula in druid.
The data structure of the formula a bit simplified looks like this:
enum Term {
    Value(f64),
    Variable(String),
    Operator{left: Arc<Term>, Operator, right: Arc<Term>},
    Exp{base: Arc<Term>, exponent: Arc<Term>},
    Root(Arc<Term>),
}

The problem is: this doesn't work since druid-enums requires me to create the widgets for all variants up front (correct me if i am wrong). To do this the switcher widget would have to create the widget when it is used. Nevertheless, having all variants present at the same time still useful in some other cases. Therefore i implemented an alternative to druid-enums (Switcher and LazySwitcher). I think it has some advantages but also a few drawbacks.
Comparison of (Lazy-)Switcher and druid-enums
Pro druid-enums
The main point for druid-enum i see at this point is usability. It doesn't need any further concepts like prisms, you just use #[derive(Matcher)] and get methods named like your variants to provide the corresponding widgets. That is great!
another point is speed: druid-enums is probably always faster than an implementation using prisms, although the difference is not that much. Switcher and LazySwitcher also only do one check in update to test, if the variant is still the same (which is probably the case most of the time). Only if the variant changes, we have to check all other prisms to find the matching widget.
Pro Switcher
The main point here is flexibility: The most complex part of creating a switcher for enums is probably handling the enums and writing the derive-macro. When this done the rest is far easier to develop and to maintain when druid changes. For the same reason it is easier to develop other widgets with similar functionality like LazySwitcher for tree-like data, Switcher and LazySwitcher.
Example code druid-enums
#[derive(Clone, Data, Matcher)]
enum AppState {
    Login(LoginState),
    Main(MainState),
}

fn ui() -> impl Widget<AppState> {
    AppStateMatcher::new()
        .login(login_ui())
        .main(main_ui())
        .controller(LoginController)
}

Example code Switcher
#[derive(Clone, Data, Prism)]
enum AppState {
    Login(LoginState),
    Main(MainState),
}

fn ui() -> impl Widget<AppState> {
    Switcher::new()
        .with_variant(AppState::Login, login_ui())
        .with_variant(AppState::Main, main_ui())
        .controller(LoginController)
}

Example code LazySwitcher
#[derive(Clone, Data, Prism)]
enum AppState {
    Login(LoginState),
    Main(MainState),
}

fn ui() -> impl Widget<AppState> {
    LazySwitcher::new()
        .with_variant(AppState::Login, login_ui)
        .with_variant(AppState::Main, main_ui)
        .controller(LoginController)
}

My conclusion
druid-enums is really good at what it does, but it is limited in what it can do. One thing to keep in mind is: This comparison is maybe a bit biased. I am not sure how common/important these use-cases are.

I do think that enums are an important case; my current feeling, though, is that we should hold off on making any big changes or additions until we have a clearer sense of where we are going, architecture-wise; if we end up moving to something more like lasagne then we won't have a Data param, and this problem disappears. :)

Hi, I happened to revisit this and I'm opting to close it. Although I haven't used it, Xilem design appears to be easier and also very good.


ASAN fails to a memory error

Finally I've made ASAN work to a point where a developer attention may be needed, yet the error may not be descriptive as one may expect from the tool
SUMMARY: AddressSanitizer: SEGV ../nptl/pthread_mutex_lock.c:67 in __GI___pthread_mutex_lock
==6724==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f0e6c726fc4 bp 0x7ffe5632c940 sp 0x7ffe5632c7c8 T0)
==6724==The signal is caused by a READ memory access.
==6724==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
    #0 0x7f0e6c726fc4 in __GI___pthread_mutex_lock ../nptl/pthread_mutex_lock.c:67
    #1 0x7f0e6b2d59d1 in __gthread_mutex_lock /usr/share/miniconda/envs/omnisci-dev/x86_64-conda-linux-gnu/include/c++/12.3.0/x86_64-conda-linux-gnu/bits/gthr-default.h:749
    #2 0x7f0e6b2d59d1 in std::mutex::lock() /usr/share/miniconda/envs/omnisci-dev/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/std_mutex.h:100
    #3 0x7f0e6b2d59d1 in std::unique_lock<std::mutex>::lock() /usr/share/miniconda/envs/omnisci-dev/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/unique_lock.h:139
    #4 0x7f0e6b2d59d1 in std::unique_lock<std::mutex>::unique_lock(std::mutex&) /usr/share/miniconda/envs/omnisci-dev/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/unique_lock.h:69
    #5 0x7f0e6b2d59d1 in CalciteMgr::submitTaskToQueue(std::packaged_task<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (CalciteJNI*)>&&) /home/runner/work/hdk/hdk/omniscidb/Calcite/CalciteJNI.cpp:711
    #6 0x7f0e6b2d9d2b in CalciteMgr::process(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, SchemaProvider*, Config*, std::vector<FilterPushDownInfo, std::allocator<FilterPushDownInfo> > const&, bool, bool, bool) /home/runner/work/hdk/hdk/omniscidb/Calcite/CalciteJNI.cpp:618
    #7 0x7f0e74c1b21d in RelAlgCache::process(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<FilterPushDownInfo, std::allocator<FilterPushDownInfo> > const&, bool, bool, bool) /home/runner/work/hdk/hdk/omniscidb/Tests/ArrowSQLRunner/RelAlgCache.cpp:97
    #8 0x7f0e74bd5719 in operator() /home/runner/work/hdk/hdk/omniscidb/Tests/ArrowSQLRunner/ArrowSQLRunner.cpp:82
    #9 0x7f0e74bd5719 in execution<TestHelpers::ArrowSQLRunner::(anonymous namespace)::ArrowSQLRunnerImpl::getSqlQueryRelAlg(const std::string&)::<lambda()> > /home/runner/work/hdk/hdk/omniscidb/Shared/measure.h:32
    #10 0x7f0e74bd5719 in getSqlQueryRelAlg /home/runner/work/hdk/hdk/omniscidb/Tests/ArrowSQLRunner/ArrowSQLRunner.cpp:81
    #11 0x7f0e74bd5719 in makeRelAlgExecutor /home/runner/work/hdk/hdk/omniscidb/Tests/ArrowSQLRunner/ArrowSQLRunner.cpp:88
    #12 0x7f0e74be4534 in runSqlQuery /home/runner/work/hdk/hdk/omniscidb/Tests/ArrowSQLRunner/ArrowSQLRunner.cpp:100
    #13 0x7f0e74be54c0 in runSqlQuery /home/runner/work/hdk/hdk/omniscidb/Tests/ArrowSQLRunner/ArrowSQLRunner.cpp:127
    #14 0x7f0e74be54c0 in runSqlQuery /home/runner/work/hdk/hdk/omniscidb/Tests/ArrowSQLRunner/ArrowSQLRunner.cpp:133
    #15 0x7f0e74be54c0 in run_multiple_agg /home/runner/work/hdk/hdk/omniscidb/Tests/ArrowSQLRunner/ArrowSQLRunner.cpp:152
    #16 0x7f0e74be9734 in TestHelpers::ArrowSQLRunner::run_multiple_agg(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ExecutorDeviceType, bool) /home/runner/work/hdk/hdk/omniscidb/Tests/ArrowSQLRunner/ArrowSQLRunner.cpp:477
    #17 0x558e0b5509d7 in Errors_InvalidQueries_Test::TestBody() /home/runner/work/hdk/hdk/omniscidb/Tests/ArrowBasedExecuteTest.cpp:1620
    #18 0x7f0e74b88b5d in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/usr/share/miniconda/envs/omnisci-dev/lib/libgtest.so+0x59b5d)
    #19 0x7f0e74b88e30 in testing::Test::Run() (/usr/share/miniconda/envs/omnisci-dev/lib/libgtest.so+0x59e30)
    #20 0x7f0e74b891fe in testing::TestInfo::Run() (/usr/share/miniconda/envs/omnisci-dev/lib/libgtest.so+0x5a1fe)
    #21 0x7f0e74b89553 in testing::TestSuite::Run() (/usr/share/miniconda/envs/omnisci-dev/lib/libgtest.so+0x5a553)
    #22 0x7f0e74b91fda in testing::internal::UnitTestImpl::RunAllTests() (/usr/share/miniconda/envs/omnisci-dev/lib/libgtest.so+0x62fda)
    #23 0x7f0e74b89765 in testing::UnitTest::Run() (/usr/share/miniconda/envs/omnisci-dev/lib/libgtest.so+0x5a765)
    #24 0x558e0b46bfa0 in RUN_ALL_TESTS() /home/runner/work/hdk/hdk/omniscidb/ThirdParty/googletest/gtest/gtest.h:11696
    #25 0x558e0b46bfa0 in main /home/runner/work/hdk/hdk/omniscidb/Tests/ArrowBasedExecuteTest.cpp:17863
    #26 0x7f0e6c773082 in __libc_start_main ../csu/libc-start.c:308
    #27 0x558e0b46c48d in _start (/_work/build/omniscidb/Tests/ArrowBasedExecuteTest+0x19648d)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV ../nptl/pthread_mutex_lock.c:67 in __GI___pthread_mutex_lock

https://github.com/intel-ai/hdk/actions/runs/5518299811/jobs/10063283058#step:12:3993

Shouldn't we use --rel-alg-cache for the ASAN run? With the cache used, we wouldn't get such a call stack.

@ienkovich the launch line looks like  ./ArrowBasedExecuteTest --use-rel-alg-cache=cache.txt
see https://github.com/intel-ai/hdk/blob/65d2206047c18550daa5753f3af25570faab79fe/.github/workflows/test-docker.yml#L93

In this case, I don't see how we could get to CalciteMgr::proces. We shouldn't create CalciteMgr at all.

@ienkovich thanks! testing

all is working now, thanks


add CAP_SYS_NICE to ClickHouse container

Context: https://github.com/plausible/analytics/discussions/4629#discussioncomment-10774222

Won't be needed in the future: https://github.com/ClickHouse/ClickHouse/issues/70960


Allow SSH key use for Bitbucket source

Instead of using a Bitbucket user license with simple auth, repositories can use ssh access keys.

happy to take a PR