Update Dokan wrapper to 1.0.0

@arekbulski I updated regarding your comments.
I have found only 2 print, is there still remaining ?

Nope, all fixed.

All fixed.

Thank you for your feedback @lurch I fixed them

I'll try to find time to test this on my Windows7 laptop one evening this week..

:+1:
You will simply need to download and install last Dokan RC
https://github.com/dokan-dev/dokany/releases/download/v1.0.0-RC4/DokanSetup_redist.exe
If it does not found dokan1.dll, you can find it in system32 or syswow64 depending of python version used.

@Liryna I would like you to look at #241 and #242  and review these two. Either incorporate those into yours or advise to reject them.

Thats where a part of my changes comes :)

#242 remains. I suggest that you keep incorporating other dokan issues and PRs and advise lurch when to close them. Consolidation is gaining traction.

@lurch fixed :+1:

Done, I hope this will be the last change.

There is a saying among game developers: "Nothing is e-v-e-r done." ^^

@happy-monk Thank you for testing it !
Just to resume the situation, you have issue to create folder for now ? but the device mount without issue, you can read/write a file ?
I will take a look at the CreateFile.

Ditto, thanks for looking at this @happy-monk :+1:
(makes me glad I didn't prematurely merge it! ;) )

@Liryna Yes and yes.
@lurch :metal:

@Liryna There seems to be some major problems in your commit.
First problem: Your filesystem tends to work well on reading and writing, but not creating folders. Any attempt to create a folder instead creates a file with the same name at the target folder. This makes files' hierarchical manipulations impossible. This bug occurs on all filesystems. Screenshot as follows:

Second problem: The dokan wrapper literally removes files, albeit correctly in the filesystem, but it seems to raise exceptions on the way deleting the files. I do not quite understand the specific implementations, but the debugging messages are indeed irritating. This bug occurs on all filesystems. Debugging message as follows:
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 291, in wrapper
    return func(self, *args)
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 206, in wrapper
    res = func(*args, **kwds)
  File "C:\Programs\Python3\lib\site-packages\fs\errors.py", line 219, in wrapper
    return func(*args,**kwds)
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 492, in Cleanup
    self._pending_delete.remove(path)
KeyError: '/New folder'

Third problem: Minor, but sometimes effects overall experience, that drives may not be unmounted after quite a long time. I haven't found out the reason, but the code and the debugging traceback is here.
>>> from fs.osfs import OSFS
>>> from fs.expose import dokan
>>> f = OSFS('D:\\Desktop')
>>> mp = dokan.mount(f, 'G:\\')
>>> mp.unmount()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 1046, in unmount
    raise OSError("the filesystem could not be unmounted: %s" %(self.path,))
OSError: the filesystem could not be unmounted: G:\
>>> dokan.unmount('G:\\')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 1002, in unmount
    raise OSError("filesystem could not be unmounted: %s" % (path,))
OSError: filesystem could not be unmounted: G:\
>>>
^C
C:\Users\Administrator>python
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from fs.expose import dokan
>>> dokan.unmount('G:\\')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 1002, in unmount
    raise OSError("filesystem could not be unmounted: %s" % (path,))
OSError: filesystem could not be unmounted: G:\
>>>

Fourth problem: ZipFS could not be exposed to dokan. I am not sure whether this is a problem, but this could be related to implementation methods which are not yet implemented. Debugging methods as follows:
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from fs.zipfs import ZipFS
>>> from fs.expose import dokan
>>> f = ZipFS('D:\\minetest-0.4.14-win32.zip')
>>> f.listdir()
['minetest-0.4.14']
>>> mp = dokan.mount(f, 'H:\\')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 981, in mount
    mp = MountProcess(fs, path, kwds)
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 1039, in __init__
    cmd = cmd % (repr(pickle.dumps((fs, path, dokan_opts, nowait), -1)),)
TypeError: cannot serialize '_io.BufferedReader' object
>>>

Would be grateful if you could solve these problems.

@ht35268 I will fix the create folder issue but other reported problem that you point does not come from my commit and I don't have the knowledge to fix them.
Feel free to create a pull request with the fix.

Fourth problem: ZipFS could not be exposed to dokan.

I expect this might be a problem with ZipFS, rather than Dokan. (ISTR something similar being mentioned before a long time ago)
IIRC the 'fix' is to use the foreground=True option in the dokan.mount call (as this then means the filesystem being exposed (ZipFS in this case) doesn't need to be pickled).

Create directory and error self._pending_delete.remove(path) KeyError: '/New folder' are fixed 👍
and cannot reproduce the issue Third problem of @ht35268

@Liryna I can't reproduce the third problem either :P
Seems to me that it is a rare problem which has nothing to do with pyFilesystem or your dokan wrapper. Maybe it's just a bug in dokan but never mind.
@lurch It seems definite that the third problem has something to do with ZipFS and not dokan wrapper.

@Liryna You don't seem to use Python3 much. I created a pull request to your repo and it could fix Python3 compatibility problems.

Fifth problem: dokan wrapper is not synchronized with Python, changes in the dokan wrapper would not be shown in pyFilesystem and neither the other way.

Good to know that all issue you faced are now fixed.

You don't seem to use Python3 much. I created a pull request to your repo and it could fix Python3 compatibility problems.

@ht35268 my first python project here 👍 I tried with python 3 and got an error so I tought it was pyFilesystem that was not py3 compatible. Thank you for the PR it has been merged !
And for the Fifth problem, I have absolutely not idea how to do this, will need @lurch here.

Well, okay. I'll see if I can solve this issue myself. Currently working on it.
@Liryna Also there's a sixth problem :P
Chrome doesn't save files into dokan-wrapped-pyfilesystems, neither does any other program, i.e. notepads, wordpads, mspaints e.t.c. I'm currently solving this problem by using... VHDs and wrapped them inside MemoryFSs. This is indeed awkward, I must say.

You see you can't create any file here, it might be a direct consequenct of improper ZwCreateFile implementation?

And Problem six has nothing to do with dokan library.

Fifth problem: dokan wrapper is not synchronized with Python, changes in the dokan wrapper would not be shown in pyFilesystem and neither the other way.

Yeah, that's kinda related to the ZipFS problem, and is solved by specifying foreground=True.
With foreground=False it actually pickles the exposed FS (e.g. MemoryFS), effectively making a copy of it, and the dokan expose process then unpickles and serves this copied FS.

Anyway, there are still problems with this wrapper, e.g. Problem 6. Wonder if we use threading for the dokan expose process to serve the FS. I've been annoyed by the pickling process when it eats up lots of memory. Trying to solve problem 5 and not 6.

Not my area of expertise, but given that various layers of the OS filesystem may do different amounts of buffering, maybe it's not even possible for the Dokan and Python processes to share the same 'live view' of the same filesystem? *shrug*

Chrome doesn't save files into dokan-wrapped-pyfilesystems, neither does any other program, i.e. notepads, wordpads, mspaints e.t.c. I'm currently solving this problem by using... VHDs and wrapped them inside MemoryFSs. This is indeed awkward, I must say.

@ht35268 this is "probably" related to getfilesecurity that are not supported on pyfilesystem.
@lurch I am not sure I understand correctly, Dokan request each time pyfs if a file exists and it does, he proceeds the windows request. So for me pyfs is the only one who know the existing of files.

@lurch I am not sure I understand correctly, Dokan request each time pyfs if a file exists and it does, he proceeds the windows request. So for me pyfs is the only one who know the existing of files.

Yeah, but if you call mount with foreground=False then it fires off a separate background process. No, I dunno why, you'd need to ask the original dokan-expose author ;)

"probably" related to getfilesecurity that are not supported on pyfilesystem

Could you "fake" it, and assume that it's always allowed?

@lurch That is what ntfs-3g does in Linux on NTFS systems - make everything rwxrwxrwx...

@lurch Well, I was just trying to fix problem six, with a feeble attempt to change it to threading. Maybe I just realized why the original author did not use threading... Can anyone fix this?
This was in Line 1051 or so... I edited this code and commented the original lines. (fs/expose/dokan/init.py)
def __init__(self, fs, path, dokan_opts={}, nowait=False, **kwds):
    if libdokan is None:
        raise OSError("the dokan library is not available")
    _check_path_string(path)
    self.path = path
    def mount_worker_thr(data):
        MountProcess._do_mount(data)
        return
    threading.Thread(target=mount_worker_thr, args=[(fs, path, dokan_opts, nowait)]).start()
    # cmd = "try: import cPickle;\nexcept ImportError: import pickle as cPickle;\n"
    # cmd = cmd + "data = cPickle.loads(%s); "
    # cmd = cmd + "from fs.expose.dokan import MountProcess; "
    # cmd = cmd + "MountProcess._do_mount(data)"
    # cmd = cmd % (repr(cPickle.dumps((fs, path, dokan_opts, nowait), -1)),)
    # cmd = [sys.executable, "-c", cmd]
    cmd = [sys.executable, "-c", '']
    super(MountProcess, self).__init__(cmd, **kwds)

And dokan explicitly fails and refuses to mount MemoryFS.
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from fs.memoryfs import MemoryFS
>>> from fs.expose import dokan
>>> f = MemoryFS()
>>> mp = dokan.mount(f, 'I:\\')
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Programs\Python3\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "C:\Programs\Python3\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 1057, in mount_worker_thr
    MountProcess._do_mount(data)
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 1096, in _do_mount
    mount(fs, path, **opts)
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 996, in mount
    raise OSError("Dokan failed with error: %d" % (res,))
OSError: Dokan failed with error: -5

Traceback (most recent call last):
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 968, in check_ready
    os.stat(path)
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request: 'I:\\'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 1003, in mount
    check_ready(mp)
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 970, in check_ready
    check_alive(mp)
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 961, in check_alive
    raise OSError("dokan mount process exited prematurely")
OSError: dokan mount process exited prematurely
>>>

I tried with the following code and comment and uncomment the several lines... then I suspected that Dokan must be on the same thread as the pyFilesystem.
def mount_worker_thr(fs, path, dokan_opts, nowait):
    dokan_opts['foreground'] = True
    if nowait:
        dokan_opts["ready_callback"] = False
    mount(fs, path, foreground=True)
    import time
    time.sleep(15)
    return
# mount_worker_thr(fs, path, dokan_opts, nowait)
threading.Thread(target=mount_worker_thr, args=(fs, path, dokan_opts, nowait)).start()

When only using the mount_worker_thr, it works well until now, but when we use threading, it seems to collapse immediately, which throws the '-5' error.
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Programs\Python3\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "C:\Programs\Python3\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 1061, in mount_worker_thr
    mount(fs, path, foreground=True)
  File "C:\Programs\Python3\lib\site-packages\fs\expose\dokan\__init__.py", line 996, in mount
    raise OSError("Dokan failed with error: %d" % (res,))
OSError: Dokan failed with error: -5

So are there any limitations in native dokan library?

I can try to "fake" it probably later.
You can start dokan from another thread without problem.
It is just that DokanMain will block with this newly created thread.

@Liryna why? And I faked it with STATUS_SUCCESSes, yet it did not work. Is there anything wrong with my trick?
@handle_fs_errors
def GetFileSecurity(self, path, securityinformation, securitydescriptor, securitydescriptorlength, neededlength, info):
    return STATUS_SUCCESS
    # return STATUS_NOT_IMPLEMENTED

@handle_fs_errors
def SetFileSecurity(self, path, securityinformation, securitydescriptor, securitydescriptorlength, info):
    return STATUS_SUCCESS
    # return STATUS_NOT_IMPLEMENTED

@ht35268 Dokan throws -5 error when you try to mount path that is already mounted or wasn't unmounted properly. This has nothing to do with threads.

@happy-monk Got it. But does this mean I'll have to reboot the computer every time? I always have to kill python by hand... Maybe it will solve this.
And literally, I tried these operations thoroughly and it only throws -5 error in threading. I tried it without threading immediately after and no -5 errors were thrown.

@ht35268 You don't have to reboot, there is some timeout before the mount point unmounts automatically.
Also you can unmount manually using dokanctl or just call dokan.unmount in your Python code before mounting.
It's strange, that you have problems with threading. I have did just the same and it works as expected. Windows programming is truly unpredictable thing.

@ht35268 the trick is not that simple 😄 securitydescriptor need to be filled with the information request in securityinformation.
The same trick as in dokan fuse has to be done here.
See https://github.com/dokan-dev/dokany/blob/master/dokan_fuse/src/dokanfuse.cpp#L457-L484
We fake the request for all directories request. This code has to be convert in python 👍

Any progress on this?

win32security has to be used here for ConvertStringSecurityDescriptorToSecurityDescriptor and ConvertSecurityDescriptorToStringSecurityDescriptor
Is that alright as dependency ?

Is that alright as dependency ?

Yeah, other FS modules have their own custom requirements http://docs.pyfilesystem.org/en/latest/getting_started.html#prerequisites so adding another one for Dokan should be fine. Is it pip-installable?

On 2016-10-04 13:36, Andrew Scheller wrote:

Is that alright as dependency ?

Yeah, other FS modules have their own custom requirements
http://docs.pyfilesystem.org/en/latest/getting_started.html#prerequisites so
adding another one for Dokan should be fine. Is it pip-installable?

If pywin32 is not pip-installable, I think a ctypes-based implementation
would make for a nicer user experience, though.
I do not have the time to help with that atm. I think that would be
something for a later Pull Request. Or someone convinces pywin32 to
provide wheels that are distributed on PyPi.

I just check on my environement and seems like it is present by default and not linked to pywin32 .
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import win32security
>>> win32security.ConvertStringSecurityDescriptorToSecurityDescriptor
<built-in function ConvertStringSecurityDescriptorToSecurityDescriptor>
>>> import pywin32
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'pywin32'

I think maybe you (or some other package) must have installed that?
https://docs.python.org/3/search.html?q=win32security shows no results.

On 2016-10-04 14:05, Liryna wrote:

I just check on my environement and seems like it is present by default
and not linked to pywin32 .

I am pretty sure it belongs to pywin32 (or some other Python-package
that you installed) ;-) I cannot link to the PyWin32-documentation
because they provide their documentation in a CHM-file 😯 which seems
not  accessible without installing the beast. They may have
documentation on it.
See also the feature request on their
bugtracker. If
Twisted is really relying on it, I think it is a safe bet (in terms of
maintenance etc.) for Pyfilesystem as well, even though the PyPi package
is one release behind (I am not sure if they do, yet). See also the
discussion on the Twisted
bugtracker

It seems to be in 2.7.12 be default as well.

@Liryna There is no package or module that you can import like this: import pywin32. PyWin32 is the name of this package in PYPI, but in terms of Python modules it's not even a package, it's a collection of modules with names like win32api, and win32security, and pywintypes.
Personally, I'd prefer not to require 7 MB package as dependency, when all really needed is two WinAPI functions. Besides, I'm not sure how to use security descriptor returned by pywin32 with ctypes or whether it even possible.

Tahoe-LAFS depends indirectly on pypiwin32 (the name of the package on Pypi that provides wheels of PyWin32). So pyfilesystem has already a soft dependency on pywin32

Any news? I really want to use this...

I need help on this 😢
For now it seems that I have correctly converted the sddl string as I wanted but I am unable to set the argument pointer PySECURITY_DESCRIPTOR with the result.
https://github.com/PyFilesystem/pyfilesystem/pull/258/commits/6969c9be6fddc0c2e8502c0defdfaface3d1977e#diff-8a34a591f687d302fe173fc731c8f267R111
I don't get where PySECURITY_DESCRIPTOR is set http://docs.activestate.com/activepython/2.5/pywin32/PySECURITY_DESCRIPTOR.html and to make it work with POINTER()

You should be able to copy into another buffer like this:
securitydescriptor[:] = finalsd
I am not sure about whether this line would do the job, but this is how you copy one buffer into another.
Also make sure you test this on 2.7 once you got it working under 3.
I have not worked with the buffer interface much, so I may be wrong.

Don't use pywin32. It doesn't cooperate with ctypes and it wouldn't give you its buffer contents. Just leave it be and use ctypes to bind the functions you need.

@happy-monk I also think that using ctypes would probably be easier.
Still I think it might work because the pywin32 documentation claims to support the buffer protocol. Not sure if that includes writing as well

Note the PySECURITY_DESCRIPTOR object supports the buffer interface. Thus buffer(sd) can be used to obtain the raw bytes. tp_as_buffer

Oh, actually it does support buffer interface. So you can get the raw bytes, if you wrap it in buffer first.
It still seems highly unnecessary to me. Dokan support is functional now. Nothing prevents reading not writing of files. There is only minor inconveniences.

I agree with you @happy-monk . The implementation is correctly working now.
I removed the commit 0001-Start-GetFileSecurity.txt and let someone with more knowledge on this do it if it is needed for him.
@lurch The PR is ready 😉

Great! This is so cool. (I can confirm that it works as I have use daily it by modifiying my install with its changes.)

@lurch The PR is ready :wink:

So, from what I understand the new Dokan wrapper is still unable to create directories? I'm not sure I'd count that as 'ready' :-/

...or have I misunderstood and it's actually some other feature that dokan-wrapper doesn't currently support? Kind of hard to keep track with such a huge discussion thread.

it seems to just be file security that does not work.

Also when a FS is exposed it seems like I  can only make folders and you need admin permission to even do that so its still not working properly at all it seems.

No, you can create files and folders just fine. Explorer doesn't do that because he THINKS, that he doesn't have permissions. And that's because pyfs binding doesn't report any permissions through GetFileSecurity, but it doesn't really prevent anything.

Not even Chrome can save files. You can't save files with notepad, with mspaint, or pretty much every application that invokes the WinAPI SaveFile dialog module.

Even python invokes that API.

No. I mean, frankly, straightforward, without doubt and verbosity, explicitly meant the SaveFile module that opens a dialog for us to choose where to save the file. Not the functions you call like you call fopen or freopen etc.

Have you tested it with python?

Do you know anything in Python that calls Windows Graphical interface?
At least I've heard of some. But you see I'm an OIer and have no time peeking deep into these stuff. All I want to do is getting things work.
Problems lay down in SecurityAttributes.

You can't save files with notepad, with mspaint, or pretty much every application that invokes the WinAPI SaveFile dialog module.

With that restriction, the usefulness of dokan-expose seems a bit limited? :-(

@lurch It's just SecurityAttributes. Not me, but I bet someone is working on faking that 'permission' to windows in dokan wrapper.

I bet someone is working on faking that 'permission' to windows in dokan wrapper.

Hopefully :+1:

@lurch Yes... This PR had a timespan of about 3 months... About time, actually.
Which is better than not having a working API

What if we just tell it that it has full permissions?

@chibill That's what exactly we were talking about. Liryna probably stated that Liryna had nothing to do about it. And happy-monk was stating that to fix this problem he needs pywin32 which is not in the standard PyPI package index - a devastation.

https://github.com/PyFilesystem/pyfilesystem/pull/258#issuecomment-251396029 is sort of pywin32 in the index

You know... getting things to work on every Linux is sort of easy, which does not necessarily apply to getting things to work on every Windows PC, with the exception of vintage models. Windows tend to react differently at different situations and different environments, and sometimes contradicts its own documentations. As people working at ReactOS once said, never believe what Windows said in its documentations... test it yourself.
I'm not going to work out this problem - since I always disliked APIs and especially WINAPIs. Function and struct definitions are always tiring.

I do not have time to test this or work on this at the moment, but I would really like to see pyfilesystem to work smoothly under Windows.
If no-one else finds the time to do this within the next two months, I can contribute the missing pieces for GetFileSecurity in around two months from now (i.e. when I have more time). If I forget, feel free to ping me ;-)

@lurch After 2 month, It would be nice if you could clone the PR and make your own opinion.

>>> finalsd = ConvertStringSecurityDescriptorToSecurityDescriptor(ssd, SDDL_REVISION_1);
>>> memoryview(finalsd)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot make memory view because object does not have the buffer interface
>>> finalsd
<PySECURITY_DESCRIPTOR object at 0x02FB62B8>

I feel like someone lie to me 😞

memoryview should be used to get the buffer if I understood ?

I have no familiarity at all with buffers in Python, but I wonder if it's something to do with the old-style and new-style interfaces mentioned in https://docs.python.org/2/c-api/buffer.html ? The PySECURITY_DESCRIPTOR docs explicitly mention the buffer() function, which is only available in Python2 and not Python3?

I tested with python 2 and there is not buffer() attribute :'(

I tested with python 2 and there is not buffer() attribute :'(

It's a function, not an attribute... https://docs.python.org/2/library/functions.html#buffer

👍 for buffer(finalsd) @lurch
OK, @ht35268 and I was not talking about the same issue.
I was looking for being able to have an explorer propose different files when left click in the folder (now it only proposes to create a folder). For me this issue comes from GetFileSecurity that does not return a proper result for now.
This is the code I currently have but I am unable to copy buffer(finalsd) to the securitydescriptor 😢
c_void_p.from_buffer_copy(buffer(finalsd))  <---this is incorrect I guess because it only copy a c_void_p size and not all the buffer :'( and I don’t find a way to say the length of copy wanted.
    def GetFileSecurity(self, path, securityinformation, securitydescriptor, securitydescriptorlength, neededlength, info):
        path = self._dokanpath2pyfs(path)
        if self.fs.isdir(path):
            sd = ConvertStringSecurityDescriptorToSecurityDescriptor(
                "D:P(A;;GA;;;SY)(A;;GRGWGX;;;BA)(A;;GRGWGX;;;WD)(A;;GRGX;;;RC)",
                SDDL_REVISION_1);
            ssd = ConvertSecurityDescriptorToStringSecurityDescriptor(sd,
                SDDL_REVISION_1, securityinformation[0]);
            print("hello 3")
            finalsd = ConvertStringSecurityDescriptorToSecurityDescriptor(ssd, SDDL_REVISION_1);
            securitydescriptor = cast(securitydescriptor, c_void_p)
            securitydescriptor = c_void_p.from_buffer_copy(buffer(finalsd))
            securitydescriptorlength = finalsd.GetLength()
            return STATUS_SUCCESS
        return STATUS_NOT_IMPLEMENTED

I have fixed the notepad creation file issue @ht35268 with https://github.com/PyFilesystem/pyfilesystem/pull/258/commits/0d7badaf027a5da1b83db8cc6286c66fa7388b65

This is the code I currently have but I am unable to copy buffer(finalsd) to the securitydescriptor :cry:
c_void_p.from_buffer_copy(buffer(finalsd)) <---this is incorrect I guess because it only copy a c_void_p size and not all the buffer :'( and I don’t find a way to say the length of copy wanted.

As I said before, if both sides support the buffer/memeoryview interface you should be able to do:
a[:] = b

Again the disclaimer, that I have not done much with buffers in Python.

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 314, in 'calling callback function'
  File "fs\expose\dokan\__init__.py", line 217, in wrapper
    res = func(*args, **kwds)
  File "fs\errors.py", line 219, in wrapper
    return func(*args,**kwds)
  File "fs\expose\dokan\__init__.py", line 826, in GetFileSecurity
    securitydescriptor[:] = buffer(finalsd)
TypeError: 'c_void_p' object does not support item assignment

😢

I would help but I honestly don't understand ctypes

I've never written cytpes or buffer code, so forgive me if I'm missing something obvious, but are you sure that doing securitydescriptor = ... inside the Python function is the right approach? Doesn't python's syntax mean that the value of securitydescriptor will get lost outside of the GetFileSecurity function?
e.g.
>>> outside = 1
>>> def func(inside):
...     inside = 2
...
>>> func(outside)
>>> outside
1

I'd expect something like securitydescriptor.something = ... or securitydescriptor.something(...) ?
And as mentioned in an earlier comment, I've never written any Win32 API code, but it "looks strange" that you're doing
ConvertStringSecurityDescriptorToSecurityDescriptor(...)
ConvertSecurityDescriptorToStringSecurityDescriptor(...)
ConvertStringSecurityDescriptorToSecurityDescriptor(...)

?

securitydescriptor is a void pointer (struct SECURITY_DESCRIPTOR).
Therefor the value will best set outside GetFileSecurity.
It is already done in GetVolumeInformation with vnmBuf (volume name buffer that is also a PVOID)
https://github.com/PyFilesystem/pyfilesystem/pull/258/files#diff-bcd6666601556e9c56d197740debee23R756

And as mentioned in an earlier comment, I've never written any Win32 API code, but it "looks strange" that you're doing

This is made in purpose, normally you get the securitydescriptor from a real file but here we create a security descriptor:
ConvertStringSecurityDescriptorToSecurityDescriptor <- Create a SD from a string
ConvertSecurityDescriptorToStringSecurityDescriptor <- Get a new string from the SD WITH the information requested by securityinformation flag.
ConvertStringSecurityDescriptorToSecurityDescriptor <- Get the final SD
It is not beautiful but it should do the trick.

I think you cannot write to void-pointers and on a closer thought it does not make sense because you do not know how to convert stuff to "void".
Pointers to any other type than void allow writing to via subscripting, I did some experiments with c_byte to confirm this
>>> import ctypes
>>> some_byte = ctypes.c_byte(42)
>>> some_byte
c_byte(42)
>>> pointer_to_some_byte = ctypes.pointer(some_byte)
>>> pointer_to_some_byte[0] = 10;
>>> some_byte
c_byte(10)
>>> pointer_to_some_byte[1] # reading random memory, ouch!
0
>>> pointer_to_some_byte[2] = 123 # writing random memory, ouch!

Going by the MSDN description as well as the dokan wrapper, securitydescriptor points to a buffer allocated by someone else, of size securitydescriptorlength.
The caller expects us to copy to that buffer (if it is of sufficient length), and return the actual length in the integer pointed at by neededlength.

@melak47 That's what we are trying to do. Writing into that buffer using ctypes seems to be more difficult than expected for some reason.

@Rondom I tried to implement it like this:
@handle_fs_errors
def GetFileSecurity(self, path, securityinformation, securitydescriptor, securitydescriptorlength, neededlength, info):
    path = self._dokanpath2pyfs(path)
    if self.fs.isdir(path):
        sd = ConvertStringSecurityDescriptorToSecurityDescriptor(
            "D:P(A;;GA;;;SY)(A;;GRGWGX;;;BA)(A;;GRGWGX;;;WD)(A;;GRGX;;;RC)",
            SDDL_REVISION_1);
        ssd = ConvertSecurityDescriptorToStringSecurityDescriptor(sd,
            SDDL_REVISION_1, securityinformation[0]);
        finalsd = ConvertStringSecurityDescriptorToSecurityDescriptor(ssd, SDDL_REVISION_1);

        length = finalsd.GetLength()
        if length <= securitydescriptorlength:
            #              no cast needed, memmove already converts to unsigned char*
            #                      |       no copy needed, we are copying to an existing buffer
            #                      |                         |
            ctypes.memmove(securitydescriptor, c_void_p.from_buffer(buffer(finalsd)), length)
            neededlength[0] = length
            return STATUS_SUCCESS
        else:
            return STATUS_BUFFER_OVERFLOW
    return STATUS_NOT_IMPLEMENTED

Which I think should work, but the buffer always appears too small. I will debug some more.

Corrected some of my errors, and switched buffer for memoryview since I'm on python3 right now, but I get an access violation;
length = finalsd.GetLength()
neededlength[0] = length # caller needs to know length always - especially if the buffer is too small
if length <= securitydescriptorlength:
    print("getting memoryview/buffer...")
    src = ctypes.c_void_p.from_buffer_copy(memoryview(finalsd)) # copy needed after all
    print("attempting memmove...")
    ctypes.memmove(securitydescriptor, src, length)
    print("successfully returned security descriptor")
    return STATUS_SUCCESS
else:
    print("buffer too small for security descriptor")
    return STATUS_BUFFER_OVERFLOW

got security descriptor
buffer too small for security descriptor
got security descriptor
getting memoryview/buffer...
attempting memmove...
Traceback (most recent call last):
  File "d:\dev\pyfilesystem\build\lib\fs\expose\dokan\__init__.py", line 833, in GetFileSecurity
    ctypes.memmove(securitydescriptor, src, length)
OSError: exception: access violation reading 0x0000000090040001

@melak47 what is the length returned by finalsd.GetLength() ? (I will try to compare with the C struct)

@Liryna 112: attempting memmove from c_void_p(2416181249) to c_void_p(1459474084932) length 112

src always appears to be 0x0000000090040001, which is somewhat suspicious...

Oh that's strange !
It should be a rand memory value 🤔 (don't know how from_buffer_copy work but it should logically)

I was using from_buffer_copy incorrectly.
The following runs without error, but doesn't change the entries available in the context menu. New Folder also still has the admin shield icon.
length = finalsd.GetLength()
neededlength[0] = length
if length <= securitydescriptorlength:
    tempbuf = (ctypes.c_ubyte * length).from_buffer_copy(memoryview(finalsd))
    ctypes.memmove(securitydescriptor, tempbuf, length)
    return STATUS_SUCCESS
else:
    return STATUS_BUFFER_OVERFLOW

Also while experimenting with this PR (Testing a bit) I found that when you try and unmount windows throws an access violation.
>>> Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "C:\Python27\lib\site-packages\fs\expose\dokan\__init__.py", line 1110, in _do_mount
    mount(fs, path, **opts)
  File "C:\Python27\lib\site-packages\fs\expose\dokan\__init__.py", line 1013, in mount
    res = libdokan.DokanMain(ctypes.byref(opts), ctypes.byref(opstruct))
WindowsError: exception: access violation reading 0x0005006C```

This only happens if you unmount using dokan.unmount("Q:\\")

if you use the mountpoint instance to unmount its fine. Not sure if this is a bug or not.

Double checked, and the buffer appears to be written correctly, so I'm out of ideas why this isn't helping resolve the issue.
ssd = D:P(A;;GA;;;SY)(A;;GXGWGR;;;BA)(A;;GXGWGR;;;WD)(A;;GXGR;;;RC)
got security descriptor
buffer overflow
ssd = D:P(A;;GA;;;SY)(A;;GXGWGR;;;BA)(A;;GXGWGR;;;WD)(A;;GXGR;;;RC)
got security descriptor
before memmove
srcbytes  =<PHONE_NUMBER>00000000000000000000001400000002005C0004000000000014000000001001010000000000051200000000001800000000E00102000000000005200000002002000000001400000000E001010000000000010000000000001400000000A001010000000000050C000000
tempbytes =<PHONE_NUMBER>00000000000000000000001400000002005C0004000000000014000000001001010000000000051200000000001800000000E00102000000000005200000002002000000001400000000E001010000000000010000000000001400000000A001010000000000050C000000
destbytes =<PHONE_NUMBER>0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
after memmove
srcbytes  =<PHONE_NUMBER>00000000000000000000001400000002005C0004000000000014000000001001010000000000051200000000001800000000E00102000000000005200000002002000000001400000000E001010000000000010000000000001400000000A001010000000000050C000000
tempbytes =<PHONE_NUMBER>00000000000000000000001400000002005C0004000000000014000000001001010000000000051200000000001800000000E00102000000000005200000002002000000001400000000E001010000000000010000000000001400000000A001010000000000050C000000
destbytes =<PHONE_NUMBER>00000000000000000000001400000002005C0004000000000014000000001001010000000000051200000000001800000000E00102000000000005200000002002000000001400000000E001010000000000010000000000001400000000A001010000000000050C000000

Also, I found out that we can avoid using memmove and a temporary buffer entirely:
srcbuf = memoryview(finalsd)
destbuf = (ctypes.c_ubyte * length).from_address(securitydescriptor)
destbuf[:] = srcbuf[:]

This is what I get when I compare the result of running (Get-Acl).Acccess in powershell:
################### real directory                    | dokan-exposed MemoryFS
------------------------------------------------------|------------------------
FileSystemRights  : FullControl                       |  -536870912
AccessControlType : Allow                             |  Allow
IdentityReference : CREATOR OWNER                     |  Everyone
IsInherited       : False                             |  False
InheritanceFlags  : ContainerInherit, ObjectInherit   |  None
PropagationFlags  : InheritOnly                       |  None
                                                      |
FileSystemRights  : FullControl                       |  -1610612736
AccessControlType : Allow                             |  Allow
IdentityReference : NT AUTHORITY\SYSTEM               |  NT AUTHORITY\RESTRICTED
IsInherited       : False                             |  False
InheritanceFlags  : ContainerInherit, ObjectInherit   |  None
PropagationFlags  : None                              |  None
                                                      |
FileSystemRights  : FullControl                       |  268435456
AccessControlType : Allow                             |  Allow
IdentityReference : BUILTIN\Administrators            |  NT AUTHORITY\SYSTEM
IsInherited       : False                             |  False
InheritanceFlags  : ContainerInherit, ObjectInherit   |  None
PropagationFlags  : None                              |  None
                                                      |
FileSystemRights  : Modify, Synchronize               |  -536870912
AccessControlType : Allow                             |  Allow
IdentityReference : BUILTIN\Users                     |  BUILTIN\Administrators
IsInherited       : False                             |  False
InheritanceFlags  : ContainerInherit, ObjectInherit   |  None
PropagationFlags  : None                              |  None

The access rights being displayed as integers seems odd, I guess that means the value doesn't match any bitwise combination of any valid enumerators?

Great @melak47 ! I will try to see if the string is not incorrect.
I get an error when try to use your code on python 2:
  File "fs\expose\dokan\__init__.py", line 826, in GetFileSecurity
    srcbuf = memoryview(finalsd)
TypeError: cannot make memory view because object does not have the buffer interface

So I installed py 3 and :
>>> from fs.expose import dokan
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\liryna\Desktop\pyfilesystem\fs\__init__.py", line 22, in <module>
    from fs import errors
  File "C:\Users\liryna\Desktop\pyfilesystem\fs\errors.py", line 220
    except ResourceNotFoundError, e:

What am I doing wrong 😄 ?

@Liryna I think we'll need a wrapper or something to switch between buffer and memoryview for python 2 and 3.
As for the other error, I'm not sure, but here are all my changes:
GetFileSecurity.patch.txt

Thank you 👍 but I still dont understand how pyfs can build in python 3. Everywhere except use a comma when as should be used.

I used pip to install it, and there appears to be a build step that runs 2to3.py on the sources :)

Seems like I need to do the same so 😄
How do you use pip ? do you build a package first ? (I only run python setup.py install for now)

pip uses the same setup.py, I don't know if there is a difference in the end result.
You can use pip install . to install, and pip install -U . to upgrade after making changes.
(Normally, you could use -e to get an editable install, but the extra build step means that doesn't really work.)

Ah didn't know this !
but 2to3 does not seems to be run on my side 😞
>>> from fs.memoryfs import MemoryFS
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\liryna\AppData\Local\Programs\Python\Python35-32\lib\site-packages\fs\__init__.py", line 22, in <module>
    from fs import errors
  File "C:\Users\liryna\AppData\Local\Programs\Python\Python35-32\lib\site-packages\fs\errors.py", line 220
    except ResourceNotFoundError, e:
                                ^
SyntaxError: invalid syntax

Well @melak47 the test I wanted to do is to change the sddl string to the value of my desktop folder for exemple. You can get the value by running CACLS . /S in the folder. The only issue is I think that there is users ID so it is not generic.

This is a sddl that I create that give all acces to everyone and system: "D:PAI(A;OICI;FA;;;WD)(A;OICI;FA;;;SY)"

No luck with either your universal string nor with one copied from another directory.

(Get-Acl).Access now reports:
FileSystemRights  : FullControl
AccessControlType : Allow
IdentityReference : Everyone
IsInherited       : False
InheritanceFlags  : ContainerInherit, ObjectInherit
PropagationFlags  : None

FileSystemRights  : FullControl
AccessControlType : Allow
IdentityReference : NT AUTHORITY\SYSTEM
IsInherited       : False
InheritanceFlags  : ContainerInherit, ObjectInherit
PropagationFlags  : None

The rights have changed to FullControl, which is good.
However, the inheritance properties have changed, maybe that is a problem?
Can you create one with FullControl and no inheritance?

Here it with inheritance D:AI(A;OICI;FA;;;WD)(A;OICI;FA;;;SY)(A;OICIID;FA;;;SY)(A;OICIID;FA;;;BA)(A;OICIID;FA;;;S-1-5-21-88962774-469885600-2606274075-1001)
but this should be removed (A;OICIID;FA;;;S-1-5-21-88962774-469885600-2606274075-1001) it is my user id.
I don't know if removing it would break it 😢

Still no change 😢

This superuser.com post mentions that the Authenticated Users group is necessary for those context menu entries to appear!

Ok so it should be this "D:PAI(A;OICI;FA;;;AU)(A;OICI;FA;;;WD)(A;OICI;FA;;;SY)" (everyone, system and authenticated users have full controle)
But didn't work 😢 for me

On a side note. When it you rd the drive it always looks like a 200gb drive with 100gb used no matter what the FS is that you mount

Mount not rd

@chibill I am not familiar enough with pyfilesystem and whether it supports reporting free space, but I think "free space" is hard to determine in quite a few cases (FTP, Memory...).
@melak47 2to3 has a fixer for the conversion between buffer/memoryview that needs to be manually enabled because the semantics are not exactly identical. https://docs.python.org/3.6/library/2to3.html#2to3fixer-buffer
Maybe you can try this one.

pyfilesystem and whether it supports reporting free space

It's 'optional', see http://docs.pyfilesystem.org/en/latest/base.html#fs.base.FS.getmeta
But I have no idea whether Dokan is using that or just always applying some kind of "default values".

@Rondom that seems useful, but I can't quite figure out how to enable exactly that fixer in setup.py.
It lives in lib2to3/fixes/fix_buffer.py, but adding extra["use_2to3_fixers"] = ["lib2to3.fixes.fix_buffer"] doesn't work (ImportError: No module named 'lib2to3.fixes.fix_buffer.fix_apply'; 'lib2to3.fixes.fix_buffer' is not a package).
Enabling all the fixers with extra["use_2to3_fixers"] = ["lib2to3.fixes"] appears to hang the setup, consuming lots of CPU and memory, and I haven't seen it finish yet...

@melak47 That's one more reason to use ctypes instead of pywin32 :)

@melak47 I have try to reproduce the behavior on the C dokan mirror. I began to return STATUS_NOT_SUPPORTED in getfilesecurity. I got the same behavior. It only proposed to create a folder with admin rights.
Therefore I have rewritten and tested different sddl string. You were right, only Authenticated Users rights are needed.
Only the sddl is needed D:PAI(A;OICI;FA;;;AU) and I have correctly tested it !
Here is my C code:
  if (dwAttrib == INVALID_FILE_ATTRIBUTES ||
    !(dwAttrib & FILE_ATTRIBUTE_DIRECTORY))
    return STATUS_NOT_SUPPORTED;

  PSECURITY_DESCRIPTOR SecurityDescriptorTmp = NULL;
  ULONG               Size = 0;
  if (!ConvertStringSecurityDescriptorToSecurityDescriptor(
    L"D:PAI(A;OICI;FA;;;AU)",
    SDDL_REVISION_1, &SecurityDescriptorTmp, &Size)) {
    DWORD error = GetLastError();
    DbgPrint(L"\terror code 1 = %d\n\n", error);
    return DokanNtStatusFromWin32(error);
  }

  LPTSTR pStringBuffer = NULL;
  if (!ConvertSecurityDescriptorToStringSecurityDescriptor(
    SecurityDescriptorTmp, SDDL_REVISION_1, *SecurityInformation,
    &pStringBuffer, NULL)) {
    DWORD error = GetLastError();
    DbgPrint(L"\terror code 2 = %d\n\n", error);
    return DokanNtStatusFromWin32(error);
  }

  LocalFree(SecurityDescriptorTmp);
  SecurityDescriptorTmp = NULL;
  Size = 0;
  if (!ConvertStringSecurityDescriptorToSecurityDescriptor(
    pStringBuffer, SDDL_REVISION_1, &SecurityDescriptorTmp,
    &Size)) {
    DWORD error = GetLastError();
    DbgPrint(L"\terror code 3 = %d\n\n", error);
    return DokanNtStatusFromWin32(error);
  }

  if (Size > BufferLength)
  {
    *LengthNeeded = Size;
    DbgPrint(L"\tlength needed %d %d\n", Size, BufferLength);
    return STATUS_BUFFER_OVERFLOW;
  }

  memcpy(SecurityDescriptor, SecurityDescriptorTmp, Size);
  *LengthNeeded = Size;

  if (pStringBuffer != NULL)
    LocalFree(pStringBuffer);
  if (SecurityDescriptorTmp != NULL)
    LocalFree(SecurityDescriptorTmp);

I had not time for now to test to convert to python.
But if the sddl still dont work, it mean we dont give the correct informations in the describtor buffer.

I tried this:
    @handle_fs_errors
    def GetFileSecurity(self, path, securityinformation, securitydescriptor, securitydescriptorlength, neededlength, info):
        try:
            path = self._dokanpath2pyfs(path)
            if self.fs.isdir(path):
                sd = ConvertStringSecurityDescriptorToSecurityDescriptor('D:PAI(A;OICI;FA;;;AU)', SDDL_REVISION_1);
                ssd = ConvertSecurityDescriptorToStringSecurityDescriptor(sd, SDDL_REVISION_1, securityinformation[0]);
                print "ssd ==", ssd
                finalsd = ConvertStringSecurityDescriptorToSecurityDescriptor(ssd, SDDL_REVISION_1);

                length = finalsd.GetLength()
                neededlength[0] = length
                if length <= securitydescriptorlength:
                    srcbuf = memoryview(finalsd)
                    destbuf = (ctypes.c_ubyte * length).from_address(securitydescriptor)
                    tempbuf = (ctypes.c_ubyte * length).from_buffer_copy(srcbuf)
                    ctypes.memmove(destbuf, tempbuf, length);
                    # destbuf[:] = srcbuf
                    return STATUS_SUCCESS
                else:
                    return STATUS_BUFFER_OVERFLOW
        except:
            import traceback
            traceback.print_exc()
        return STATUS_NOT_IMPLEMENTED

I tried both a MemoryFS and an OSFS, and both memmove and slice syntax - still no change for me.
This prints ssd == D:PAI(A;OICI;FA;;;AU, so the descriptor converts back to a string exactly the same.
I'm testing on Windows 10  x64 build 14393 with the Dokan 1.0.0 release.

I am still unable to use pyfs on python 3 😢
Always get the error during build time:
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from fs.memoryfs import MemoryFS
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "E:\islog\dev\app\dokan\pyfilesystem\fs\__init__.py", line 22, in <module>
    from fs import errors
  File "E:\islog\dev\app\dokan\pyfilesystem\fs\errors.py", line 220
    except ResourceNotFoundError, e:

I understand it is because It is not converted properly to py3 but no idea how to force it 😢

except ResourceNotFoundError, e is an old syntax form Python <=2.5.
except ResourceNotFoundError(e) should work on 2.6 and later. I doubt we support Python 2.5, or do we?

@Liryna I have an idea. Are you running your python interpreter in the pyfilesystem source folder?
In that case import fs will see the directory fs in the current directory, which has the python2 sources,  and import that instead of the installed version from e.g. C:\Python35\Lib\site-packages\fs that was processed by 2to3.

@melak47 it was exactly that 😄 ! Thank you !

Like @melak47 said, the rights are correctly set 😢
I have no other ideas on this sorry guys. I tried to reproduce the mirror volum settings but nothing improved the behavior.
@lurch do you think this could be merged without the "New" menu item ?

I'd obviously prefer to merge a fully-working version, but it sounds like that's not going to be possible unless somebody intimately familiar with both low-level Win32 API code and Python ctypes takes a look at it :-/
Could you update the Dokan-expose documentation to explain this menu-limitation (and any possible workarounds)? And I guess http://pyfilesystem.readthedocs.io/en/latest/expose/dokan.html needs to mention a different Dokan download-URL too ;-)  (and maybe you could also show mounting to a path)

Yeah the mounting on windows you need to use say "Q://" (Might be the other slashes typing from memory)

@Liryna Can you compare the permissions of the pyfilesystem and dokan mirror volumes (say with SACL again)?

How do you use the mirror volume on dokan? (1.0.0)

https://github.com/dokan-dev/dokany/wiki/Use Mirror example

@chibill What do you mean by the wiki is not up to date ?
https://github.com/dokan-dev/dokany/wiki/Use-Mirror-example
There was just Mirror.exe is embedded with dokan install in Dokan\DokanLibrary-1.0.0\sample\mirror when the option Install Dokan development files is checked during the installation. missing.
( @Rondom always faster 😄 )
@lurch I will make the changes tomorrow 👍
@melak47 I don't have my environment with me 😢 I give will give you my last patch (include 90% of your code and I will give you the SACL exact).
I looked in the property window in my test and it was correctly showing the Authenticated Users rights full access 😢

Well It was wrong till you say its install WHEN you check the option.

I've managed to do this by providing the security descriptor of some existing directory (current user's home directory in my example) through the call to GetFileSecurity function.
https://gist.github.com/happy-monk/ad7c210279fb25409d54e1b39038ef8e
Apparently, security descriptor is tricky thing to make, but the GetFileSecurity function makes them right for sure.

Well done @happy-monk ! Lets do this 😃 !
Even full ctypes like you wanted ❤
Just a question, What does return os.path.expanduser('~') in case of the user is local system ? (service application)

Just a question, What does return os.path.expanduser('~') in case of the user is local system ? (service application)

It return C:\Windows\System32\config\systemprofile so for python services using dokan, it is going to be the same behavior. There is only SYSTEM and Admin rights set here.
I have successfully added integrated @happy-monk code ❤ it work perfectly !
PR has been updated @lurch and I made the changes that you requested 👍

So just to clarify - does this now mean that all outstanding problems with Dokan-expose have been fixed? :smiley:
And there's no dependency on win32security?

Yes all problmes reported here has been fixed.
There is only the case where python is started with system local account (service application), the menu item will not work (create folder with admin rights only proposed).
And no dependency to win32security 👍

Yes all problmes reported here has been fixed.

Awesome! :+1:

There is only the case where python is started with system local account

I think that's a pretty unlikely scenario with pyfilesystem, so we can probably ignore it?

tell me if still some changes remaining

There's still a few possible issues outstanding, and because of the length and complexity of this thread I'm not sure if GitHub is reporting them properly:
https://github.com/PyFilesystem/pyfilesystem/pull/258/files#r84283784
https://github.com/PyFilesystem/pyfilesystem/pull/258/files#r84280582
https://github.com/PyFilesystem/pyfilesystem/pull/258/files#r80446904
https://github.com/PyFilesystem/pyfilesystem/pull/258/files#r84295828
https://github.com/PyFilesystem/pyfilesystem/pull/258/#discussion_r84297844
https://github.com/PyFilesystem/pyfilesystem/pull/258/#discussion_r84298242
Thanks to you and everyone else for all your hard work on this :heart:  I'll do my best to test and merge it over the weekend.
It might be fun to try exposing a pyfilesystem OSFS with dokan-expose, and then wrapping the exposed FS in another OSFS, and attempting to run the OSFS unit-tests over it ;-)

I added the param but I don't get how to give acces to user config

There's an FSOperationsClass kwarg in the mount function - is that what you're looking for?
http://docs.pyfilesystem.org/en/latest/expose/dokan.html#fs.expose.dokan.mount

Ok I see ! so securityfolder can be set in argument dokan.mount(fs, "Q:\\", securityfolder="C:\\") if understood correctly how it work.

I haven't tried this myself, but that's not the way I understood it. Looking at the code in https://github.com/Liryna/pyfilesystem/blob/master/fs/expose/dokan/init.py#L942 I think you might need to do....
Ah! When I first saw your example I thought it was wrong, and you'd need to pass in your own FSOperations sub-class, but looking at the code in even more detail, it seems that you're right, because the same **kwds given to the mount call, also get passed through to the FSOperations constructor! (after popping off the optional numthreads and flags options)

securityfolder doc added: https://github.com/PyFilesystem/pyfilesystem/pull/258/commits/8925c1c5a10afc13b5c603a4d3c4c08435a4d1e8

Currently it does not work. (From the repo that the PR is against.)

mp = dokan.mount(fs,"Q://")
Traceback (most recent call last):
File "", line 1, in
File "C:\Python27\lib\site-packages\fs\expose\dokan_init_.py", line 966, in mount
raise OSError("the dokan library is not available")
OSError: the dokan library is not available

Yet I can import get the dokan dll from ctypes jsut fine

@chibill There was a missing for python 2
I added it in the branch, you can pull and run it.
I made my last test on python 3 & 2. everything work great 👍

@Liryna I think, it would be better if you did this the other way around. libdokan heavily uses objects from ctypes, so it's reasonable to user import * there. dokan.__init__ uses them only occasionally, so it's better to use import ctypes there to not pollute the namespace with unnecessary names.

Good for me so 👍

@happy-monk Good idea.

@ht35268 I think, the main purpose of libdokan as the separate module is to isolate all ctypes stuff.
@Liryna Good, but your two last commits pretty much cancel each other. So, maybe, you should squash them together? :) (If you did not plan to squash the whole PR at some point).

@happy-monk I am going to rebase all the PR when @lurch say the GO 👍

Whats the status on this? Is it basically just waiting on a rebase and then it can me pulled?

Wait @lurch approval, I rebase and merge it.
Otherwise this has been tested by a couple of people and no issue has been found so far 👍

I have been using it from your repo and have found no errors that is why I am asking. (Want to be able to just install it normally.)

Oh Ok ! Thats a good news 😄 !

@lurch @willmcgugan any news on this ?
There seems to be already a couple of users using it. It would be great to have it in the pyfs master 👍

Any news on this PR?

Yeah. I am still using what I cloned from the other one.
On Wed, Dec 7, 2016 at 3:27 PM, Andreas Gnau<EMAIL_ADDRESS>wrote:

Any news on this PR?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PyFilesystem/pyfilesystem/pull/258#issuecomment-265578415,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACCNyUazgGQeXtk-fwOA_QJhK70oviV8ks5rFyS0gaJpZM4JYzKp
.

@willmcgugan So, would you prefer this PR to be submitted to pyfilesystem2?

Really looking forward to having this in master!

.? Any idea about when this might be merged?

Thanks for the work all. I don't have access to a Windows machine, but if you are all happy with this (@lurch?) I'll go ahead and merge.

Thanks @willmcgugan I think this can be merged. It has been tested by a couple of users already. @lurch does not look to have the time to test it.
Like I said before, the current implementation of Dokan in the master is extremely old and can no longer be used.
Here I am proposing a working wrapper of the new Dokan version 👍

@willmcgugan Tested and secure.
I was wondering if we should drop support for Python 2

Ok, if everyone is confident. Thanks for your work!


CSS is not loading properly when refreshing page in Angular 8

In my angular application, I have 2 routes - /login and /dashboard.
If I navigate from login to dashboard route CSS is loading properly in the dashboard screen.

If I do refresh in the dashboard (or directly opens dashboard route) some of the CSS is not loading properly.

I checked in Console and Network tabs. I didn't get any errors and warnings.
I need your help to resolve this issue.

Thanks for reporting this issue. However, you didn't provide sufficient information for us to understand and reproduce the problem. Please check out our submission guidelines to understand why we can't act on issues that are lacking important information.
If the problem persists, please file a new issue and ensure you provide all of the required information when filling out the issue template.


Fix UWP UnauthorizedExceptions

Trying to use this in UWP currently throws an UnauthorizedAccessException, as trying to read a StorageFile directly via its filepath isn't allowed. Instead, you have to get the stream from the StorageFile itself.

Great, thanks!


Checksum extension: use Multihash

Related Issue(s): #503
Proposed Changes:
extension(checksum): update checksum extension to use Multihash
Instead of defining the hashing algorithm within the key, use a self-identifiable hash
(Multihash). This way the stac spec doesn't need to introduce a long list of supported
hash algorithms, but can rely on the ones specified in Multicodec.
I also changed the examples to use BLAKE2 instead of MD5. While MD5 is still in wide
use, I want to encourage people to update to newer hash function, which makes also
sense if you want to check the integrity of untrusted sources.
Closes #503.
PR Checklist:

[ ] This PR has no breaking changes.
[x] I have added my changes to the CHANGELOG or a CHANGELOG entry is not required.
[ ] API only: I have run npm run generate-all to update the generated OpenAPI files.

I just rebased it on-top of the current dev branch.


Added installation and CJS usage examples

It takes an extra step to dig into tests and figure out how the module works with CJS.
It's nice to have a copy-paste friendly npm/yarn install command to give the package a try quickly

shit ... it looks like I've forgotten to use --no-default flag for ascjs transformation ... this PR shouldn't be needed as the module exports only the default and nothing else.

actually ... I forgot this module also exports serialize/unserialize helpers ... will have a look at this PR, thanks!


Replace TSlint with ESlint

added eslitn config
added eslint packages
remove tslint config
update deprecated packages

Hello? any new progress with this mr?


Crop overscan in Extract Review

Description

added ability to define Overscan Crop for input in ExtractReview plugin
input field is string where is possible to define how crop will work

value may cause that resolution will be expanded instead of cropped
crop may be explicit or relative, in pixels or percents
added special percent case -10%+ (10 is random number) which says that input resoltuion is 110% of expected output (input width 2200px -> output width 2000px)
value may containt 2 values one for width second for height
all values are center related == +300px will add 150px on both sides and keep center in center
filters should only crop or expand resolution (not rescale)

Example variants with description

String
Output
Description

2200px
Empty string keep resolution unchanged.

50%
1100px
Crop 25% of input width on left and right side.

300px
300px
Keep 300px in center of input and crop rest on left and right.

300
300px
Values without units are used as pixels (px).

0px
0px
Invalid value.

+0px
2200px
Keep resolution unchanged.

+300px
2500px
Add black pillars of 150px width on left and right side.

-300px
1900px
Crop 150px on left and right side

+10%
2420px
Add black pillars of 5% size of input on left and right side.

-10%
1980px
Crop 5% of input size by on left and right side.

-10%+
2000px
Input width is 110% of output width.

Closes https://github.com/pypeclub/client/issues/53#issuecomment-846796817

Syntax for reversing the overscan using -10%+ us not working. Instead it actually expands the result by 10% while keeping the same resolution.

see comment about the -+ syntax.

Swapped +/- in code. Fixed.

Secondly Please move the crop settings above the resolution

Done

Lastly, can we make 0 value do nothing instead of being invalid?

Done. Additional question, what if output is smaller than 0px? (Small chance but possible...)


⚠ Minecraft has degraded performance

In d17d97c, Minecraft (https://www.minecraft.net) experienced degraded performance:

HTTP code: 200
Response time: 1028 ms

Resolved: Minecraft performance has improved in f0d13b0.


Add CMake flag CMAKE_BUILD_TYPE=Release

The default value of CMAKE_BUILD_TYPE is debug, it will cause some problem when using CUDA (some operators with broadcast cannot do backward).
Now add it to docs.

Thanks for your contribution, but shouldn't we rather fix the bug instead of working around it?

Thanks for your contribution, but shouldn't we rather fix the bug instead of working around it?

This flag will be passed to nvcc, whose behavior is quiet different between debug and no-debug mode. I check the code that cannot work well without the compiler flag, but I can't find the bug.

@DickJC123 could you maybe assist on this matter?
Also, could you provide detailed instructions (ideally a docker container) to reproduce this?

Simply use mx.np.divide do broadcast computation and then backward, the error message ends with:
too many resources requested for launch

Could you also provide the version of your toolchains? Nvcc, cuda, cudnn, gcc, etc

EC2 Deeplearning AMI 18.1
OS: Ubuntu 16.04
CMake: 3.13.3
CUDA: 10.0
CUDNN: 7.4.1
nvcc: 10.0.130
gcc: 5.4.0

#15747

@marcoabreu The default cmake build flag if not specified will not use the compiler optimization. We need to explicitly specify CMAKE_BULD_TYPE=Release. I suggest we do something in CMakeList.txt as the following: https://blog.kitware.com/cmake-and-the-default-build-type/

I know and that's fine. Unless you explicitly compile with release, you don't get release.
But this PR is trying to hide an error and I think we should rather address the error it revealed.

I find that all install script and ci use this flag, as well as windows build tutorial use release mode. We should make it consistent.
For ubuntu and other unix-like OS are used most, the tutorial should have it to make sure users get no-bug MXNet.

There are in fact totally valid use cases to not use release compilation. So I wouldn't dismiss the other options.

There are in fact totally valid use cases to not use release compilation. So I wouldn't dismiss the other options.

CMake build without 'release' flag cannot pass all unit test in test_operator_gpu.py in my try.

Interesting, which ones are failing? Can we fix them?

Just FYI,  CMAKE_BULD_TYPE=Release is specified when running CI.
https://github.com/apache/incubator-mxnet/blob/master/ci/docker/runtime_functions.sh#L858

@marcoabreu There are two problems reflected here. IMHO, we should not tangle them.

The Release build type is not set as the default value for cmake build config. Considering most of the users would just adopt the default settings we provide, this problem should have been corrected to make the performance of the binaries produced by make and cmake comparable. This PR is submitted to address this problem.

The lib built using either current cmake (without Release build type) or make with DEBUG=1 would fail on reduce-like operators on GPUs, sum for example. This problem has been observed many times by various users. That part of code was owned by Nvidia folks. The algorithm itself is so sophisticated that it requires their expertise to dive in. We should track this issue in another thread, instead of requesting the contributors fixing the build type problem to look into the problem that is out of their domain knowledge.

@marcoabreu IMHO the default settings for building from source should be as close to the ones used for the release versions as possible. DEBUG is more of a choice for our developers, isn't it?
Also considering the fact that our CI does not even test the DEBUG build mode and most other builds are not using DEBUG as the default, I would consider this PR a step toward a more consistent settings for the different builds.
I would agree that we need to fix "bugs", let's take a look at this "bug" now. I've personally encountered and fixed several of this "bug" myself. So from my previous experience, this error @hgt312 encountered could be possibly caused by using too large blocks/too many registers/too much shared memory according to this thread on CUDA user forum. So now let's examine each one of them:

Could we have different number of GPU thread blocks because we changed the build? NO, it's related with the input size, which was not changed between the 2 test runs with the same test code. Also we mostly use very small tensors for testing, so this could not be the cause.
Could we use different numbers of registers because we changed the build? YES! It's possible that without proper optimization, a vanilla compilation of a complicated GPU kernel could lead to excessive usage of registers.
Could we use too much shared memory because we changed the build? NO, the amount of shared memory needed only depends on the input data's shape and type.

So now we can see that the most possible cause of this is the second one: we may have some complicated GPU kernels that require rewrites (maybe splitting into 2 or more smaller kernels and do multiple launches, or some optimizations to make register usages lower). However, technically this is not "bug" rather than us hitting limitations put on us by the tools and the hardwares we are using. So either we make everything work for everyone (especially the few developers who use DEBUG and run on older GPUs) even when compiler optimizations are poorly done or when the code is run on ancient hardware (at a cost), or, we aim for a working and performant code compiled with proper optimizations for nearly all of our users (who may not even be aware of the build settings), which one would you prefer? Or do you think that this needs a community consensus on it?
@hgt312 Would you please share the failures so that we know which operators we need to take a look at? So that once the decision on the tradeoff has been made we could take further actions accordingly.

A sample error message:
(base) ubuntu@ip-172-31-16-49:~/incubator-mxnet$ nosetests -s --verbose tests/python/gpu/test_operator_gpu.py:test_np_sum
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=342263604 to reproduce.
test_operator_gpu.test_np_sum ... [08:06:44] ../src/base.cc:84: Upgrade advisory: this mxnet has been built against cuDNN lib version 7401, which is older than the oldest version tested by CI (7600).  Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.
[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1216105730 to reproduce.
ERROR

======================================================================
ERROR: test_operator_gpu.test_np_sum
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 177, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/util.py", line 307, in _with_np_shape
    return func(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/util.py", line 491, in _with_np_array
    return func(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_numpy_op.py", line 264, in test_np_sum
    assert_almost_equal(y.asnumpy(), expected_ret, rtol=1e-3 if dtype == 'float16' else 1e-3,
  File "/home/ubuntu/incubator-mxnet/python/mxnet/ndarray/ndarray.py", line 2504, in asnumpy
    ctypes.c_size_t(data.size)))
  File "/home/ubuntu/incubator-mxnet/python/mxnet/base.py", line 254, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [08:06:54] /home/ubuntu/incubator-mxnet/src/operator/nn/././../tensor/./broadcast_reduce-inl.cuh:528: Check failed: err == cudaSuccess (7 vs. 0) : Name: reduce_kernel_M1 ErrStr:too many resources requested for launch
Stack trace:
  [bt] (0) /home/ubuntu/incubator-mxnet/python/mxnet/../../build/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x32) [0x7f81a9b7fb82]
  [bt] (1) /home/ubuntu/incubator-mxnet/python/mxnet/../../build/libmxnet.so(void mxnet::op::broadcast::ReduceImpl<mxnet::op::mshadow_op::sum, 2, float, mshadow::half::half_t, mshadow::half::half_t, mxnet::op::mshadow_op::identity>(CUstream_st*, mxnet::TBlob const&, mxnet::OpReqType, mxnet::TBlob const&, mshadow::Tensor<mshadow::gpu, 1, char> const&, mxnet::op::broadcast::ReduceImplConfig<2> const&)+0x820) [0x7f81aa184e10]
  [bt] (2) /home/ubuntu/incubator-mxnet/python/mxnet/../../build/libmxnet.so(void mxnet::op::broadcast::Reduce<mxnet::op::mshadow_op::sum, 2, mshadow::half::half_t, mxnet::op::mshadow_op::identity, true>(mshadow::Stream<mshadow::gpu>*, mxnet::TBlob const&, mxnet::OpReqType, mshadow::Tensor<mshadow::gpu, 1, char> const&, mxnet::TBlob const&)+0x539) [0x7f81aa187eb9]
  [bt] (3) /home/ubuntu/incubator-mxnet/python/mxnet/../../build/libmxnet.so(void mxnet::op::ReduceAxesComputeImpl<mshadow::gpu, mxnet::op::mshadow_op::sum, true, false, mxnet::op::mshadow_op::identity>(mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, mxnet::TShape const&)+0x13e9) [0x7f81aa868649]
  [bt] (4) /home/ubuntu/incubator-mxnet/python/mxnet/../../build/libmxnet.so(void mxnet::op::NumpyReduceAxesCompute<mshadow::gpu, mxnet::op::mshadow_op::sum, true, false, mxnet::op::mshadow_op::identity>(nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)+0x4ac) [0x7f81aa97a26c]
  [bt] (5) /home/ubuntu/incubator-mxnet/python/mxnet/../../build/libmxnet.so(mxnet::imperative::PushFCompute(std::function<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&)::{lambda(mxnet::RunContext)#1}::operator()(mxnet::RunContext) const+0x2a6) [0x7f81ac1cdc16]
  [bt] (6) /home/ubuntu/incubator-mxnet/python/mxnet/../../build/libmxnet.so(std::_Function_handler<void (mxnet::RunContext), mxnet::imperative::PushFCompute(std::function<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&)::{lambda(mxnet::RunContext)#1}>::_M_invoke(std::_Any_data const&, mxnet::RunContext&&)+0x17) [0x7f81ac1cde67]
  [bt] (7) /home/ubuntu/incubator-mxnet/python/mxnet/../../build/libmxnet.so(+0x39d3f4e) [0x7f81ac127f4e]
  [bt] (8) /home/ubuntu/incubator-mxnet/python/mxnet/../../build/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0x5cf) [0x7f81ac13418f]

-------------------- >> begin captured logging << --------------------
root: INFO: NumPy-shape semantics has been activated in your code. This is required for creating and manipulating scalar and zero-size tensors, which were not supported in MXNet before, as in the official NumPy library. Please DO NOT manually deactivate this semantics while using `mxnet.numpy` and `mxnet.numpy_extension` modules.
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=342263604 to reproduce.
root: INFO: NumPy-shape semantics has been activated in your code. This is required for creating and manipulating scalar and zero-size tensors, which were not supported in MXNet before, as in the official NumPy library. Please DO NOT manually deactivate this semantics while using `mxnet.numpy` and `mxnet.numpy_extension` modules.
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1216105730 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 9.612s

FAILED (errors=1)

@marcoabreu As you can see from the discussion, sufficient technical justification has been presented for the change in this PR. I will move forward if no more questions/concerns in 24 hours.

I'd like to get an opinion from some Nvidia engineers first. So far, we are making assumptions about these kernels differing in such a significant matter that we run into limits. Since we have the advantage of having people with detailed knowledge about the ins and outs of GPUs, I'd like to have them consulted first.

@ptrendx @DickJC123 could you chime in please?

It makes sense to use release build by default, though we should still document the usage of debug build in developer guide.
The debug build issue should definitely be investigated. However, given that the debug build has issues regardless of the changes in this PR, the investigation of the bug should not be a blocker for this change.
#9516 seems to be an appropriate issue for investigation on nvcc debug mode. @hgt312 @reminisce @haojin2 it would be great if you could document what you've seen in that issue. @ptrendx @DickJC123 feel free to comment there with any insight you have.
@marcoabreu does that sound good? Do you have other technical reasons that make you believe the PR should be stopped?

Yeah that sounds like a good way to move forward. Could you add the issue as release requirement for the 1.6 release please? After that, we're good to move forward with this PR.

The too many resources requested for launch error happens most often because the number of registers required  for the kernel exceeded the number of registers available. The register file on the GPU has some capacity that is shared by all threads in a SM (streaming multiprocessor), so the more registers is used, less number of threads can be actually launched. The problem comes from the fact that the number of threads launched is a value known only at runtime, not at compile time, so the compiler cannot do the analysis to limit the number of used registers / spill some to global memory. Debug build uses more registers than the release build, so that is where you hit the error in that particular kernel. This problem can be solved by telling the compiler what is the maximum number of threads and blocks that will be launched per SM via launch bounds. Inserting proper launch bounds should be a very easy change, if you have any problem applying it just tell us which exact kernel is giving this error and we can make a PR for it as well.

@ptrendx Thanks for the detailed analysis. That's very helpful. The kernel currently throws the error is reduce_kernel_M1. That would be the best if you guys can apply the fix since you guys are experts in this.
@marcoabreu I have created an issue for tracking the progress of fixing this problem. Please kindly consider unblocking this PR from being merged.

@reminisce Ok, I assigned myself to that issue.

Thanks everyone!

@ptrendx Thank you for helping resolve this issue.


Fixed placeholder search not working right after adding a new index w…

Pull Request
Related issue
Fixes #300
What does this PR do?

Fixes a bug where the placeholder search is not working correctly after you add a document with a new start.

PR checklist
Please check if your PR fulfills the following requirements:

[x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
[x] Have you read the contributing guidelines?
[x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!

Hey @vaibhav135, thanks a lot for your contribution!  It's refreshing the cache which is exactly what we needed :)
I created another PR that re-triggeres the getIndexes documents whenever you make a search on an index with 0 documents and I'm merging your PR in this branch.

bors try

bors merge


Error on Homebridge

Hi there.
I got this error when I've tried to add to Homebridge:
[12/1/2021, 9:42:14 AM] [Xiaomi Mi Robot Vaccum-Mop P] INF getFirmware | viomi.vacuum.v8 | Firmwareversion is 3.5.3_0017
[12/1/2021, 9:42:17 AM] [homebridge-xiaomi-roborock-vacuum] This plugin slows down Homebridge. The read handler for the characteristic 'Serial Number' was slow to respond! See https://git.io/JtMGR for more info.
[12/1/2021, 9:42:23 AM] [homebridge-xiaomi-roborock-vacuum] This plugin slows down Homebridge. The read handler for the characteristic 'Serial Number' didn't respond at all!. Please check that you properly call the callback! See https://git.io/JtMGR for more info.
[12/1/2021, 9:42:24 AM] [Xiaomi Mi Robot Vaccum-Mop P] ERR getSerialNumber | Failed getting the serial number. Error: Call to device timed out
    at DeviceInfo._retryOnTimeout (/usr/lib/node_modules/homebridge-xiaomi-roborock-vacuum/miio/lib/device_info.js:295:29)
    at DeviceInfo.call (/usr/lib/node_modules/homebridge-xiaomi-roborock-vacuum/miio/lib/device_info.js:227:12)
    at module.exports.getSerialNumber (/usr/lib/node_modules/homebridge-xiaomi-roborock-vacuum/miio/lib/devices/viomivacuum.js:141:20)
    at XiaomiRoborockVacuum.getSerialNumber (/usr/lib/node_modules/homebridge-xiaomi-roborock-vacuum/index.js:842:28)
    at XiaomiRoborockVacuum.initializeDevice (/usr/lib/node_modules/homebridge-xiaomi-roborock-vacuum/index.js:690:24)
    at XiaomiRoborockVacuum.connect (/usr/lib/node_modules/homebridge-xiaomi-roborock-vacuum/index.js:770:7) {
  code: 'timeout'
}
[12/1/2021, 9:42:24 AM] [Xiaomi Mi Robot Vaccum-Mop P] STA getDevice | Serialnumber: Unknown
[12/1/2021, 9:42:24 AM] [Xiaomi Mi Robot Vaccum-Mop P] INF getFirmware | viomi.vacuum.v8 | Firmwareversion is 3.5.3_0017
[12/1/2021, 9:42:24 AM] [Xiaomi Mi Robot Vaccum-Mop P] STA getDevice | Firmwareversion: 3.5.3_0017
[12/1/2021, 9:42:24 AM] [Xiaomi Mi Robot Vaccum-Mop P] INF changedCleaning | viomi.vacuum.v8 | Cleaning is ON.
[12/1/2021, 9:42:24 AM] [Xiaomi Mi Robot Vaccum-Mop P] MON changedCharging | viomi.vacuum.v8 | ChargingState is now false
[12/1/2021, 9:42:24 AM] [Xiaomi Mi Robot Vaccum-Mop P] INF changedCharging | viomi.vacuum.v8 | Charging is cancelled
[12/1/2021, 9:42:24 AM] [Xiaomi Mi Robot Vaccum-Mop P] INF changedCharging | viomi.vacuum.v8 | Robot not anymore in dock.
[12/1/2021, 9:42:24 AM] [Xiaomi Mi Robot Vaccum-Mop P] MON changedSpeed | viomi.vacuum.v8 | FanSpeed is now 3%
[12/1/2021, 9:42:24 AM] [Xiaomi Mi Robot Vaccum-Mop P] INF changedSpeed | viomi.vacuum.v8 | Speed was changed to 3% (Turbo), for HomeKit 100%
[12/1/2021, 9:42:24 AM] [Xiaomi Mi Robot Vaccum-Mop P] INF changedPause | viomi.vacuum.v8 | Paused possible
[12/1/2021, 9:42:24 AM] [Xiaomi Mi Robot Vaccum-Mop P] ERR getSerialNumber | Failed getting the serial number. Error: Call to device timed out
    at DeviceInfo._retryOnTimeout (/usr/lib/node_modules/homebridge-xiaomi-roborock-vacuum/miio/lib/device_info.js:295:29)
    at DeviceInfo.call (/usr/lib/node_modules/homebridge-xiaomi-roborock-vacuum/miio/lib/device_info.js:227:12)
    at module.exports.getSerialNumber (/usr/lib/node_modules/homebridge-xiaomi-roborock-vacuum/miio/lib/devices/viomivacuum.js:141:20)
    at XiaomiRoborockVacuum.getSerialNumber (/usr/lib/node_modules/homebridge-xiaomi-roborock-vacuum/index.js:842:28)
    at module.exports (/usr/lib/node_modules/homebridge-xiaomi-roborock-vacuum/lib/callbackify.js:5:20) {
  code: 'timeout'
}

It showed moments before, but now has vanished from home app, and presenting this error.
I've configured it as a 'bridge' into homebridge.
Any ideas??

By the way:
My robot is the "Xiaomi Mi Robot Vacuum-Mop P"

Hi! Yeah... it's a known error for Viomi robots... I'll try to make this error silent while we figure out the right way to retrieve their serial number

Right...
But two questions:

It's normal, due this error, the plugin not work properly?
It's correct to assign this plugins as a "bridge", to run isolated from all Homebridge system?

Tks


Shared LinkConditionerConfig

Does LinkConditionerConfig belong to SharedConfig?
As I understand SharedConfig is a set of configuration that absolutely needs to be in sync between server and client in order for the network communication to work properly.
Link condition does not belong to this category. Usually this is the other way - server and client have really different network link reliability. It should be possible to configure different link condition in NaiaServer and NaiaClient.

If you agree, I can prepare a PR with this change.

Hey Tomasz! Hey thanks for looking through naia in such detail, I love getting this feedback!
Yes, eventually I would like the LinkConditionerConfig to be specific to either the Client OR Server, like you say. The reason that it's in the SharedConfig right now is that the LinkConditioner currently buffers their INCOMING messages, but not OUTGOING messages. If both the Client & Server share the same link conditions, then we don't have to worry about buffering outgoing messages, because it'll be buffered once it arrives.
I would definitely like to change this, but opted for this solution as I was getting frustrated and wanted to move on to the clientside prediction stuff 👍  If you can add buffering to outgoing messages in naia-socket, then feel free to change this!
I will get to this eventually, if not.


Make kubeip compatible with Workload Identity

Is your feature request related to a problem? Please describe.
Workload Identity is the recommended way for workloads running on Google Kubernetes Engine (GKE) to access Google Cloud services without a Google Service Account key mounted.
Describe the solution you'd like
Right now, a Google Service Account key should be generated and stored in a Kubernetes secret which is then used by kubeip. Using Workload Identity no key will be used and kubeip's Kubernetes service account will be able to impersonate an IAM service account, giving kubeip the required permissions.
Describe alternatives you've considered
Mounting the Service Account key in JSON format is the only supported method right now
Additional context

More information about Workload Identity: https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity

That would be pretty cool

I consider this a must-have. Distributing key files is not something we should need to do today.

I think it already supports it, by virtue of using the Google SDK and its default behaviour.
Just removing the GOOGLE_APPLICATION_CREDENTIALS environment label from the deployment will mean it falls back to the default way of working, which includes going to the Metadata server, which is what Workload Identity relies on.


In the inspect the layer button overlays the results

Describe the bug
In the inspect the layer button overlays the results
To Reproduce
Steps to reproduce the behavior:

open the plugin
Make sure the tokens are applied to the elements
navigate to inspect
select the layer button and see the layer button overlay

Expected behavior
when interacting with the layer button it should not overlay
Screenshots or Screencasts

Still reproducible in V 2.2.2


Has this ever worked?

I attempted to follow this, but the examples on the page are incomplete, and the linked repo did not contain this method of deployment, at least in any form I could discern. I've attempted to duplicate what I see in this document but it's failing, additionally it appears that there are either errors, or omissions. Can we either get the templates re-written in a working fashion, or a link to where these can be found online?
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bb5b79f5-7069-cc1c-819d-81e1aa90a498
Version Independent ID: df9cebf2-43d3-869a-d591-aedfe75f59f4
Content: Implement a property transformer and collector in an Azure Resource Manager template
Content Source: docs/building-blocks/extending-templates/collector.md
Service: guidance
GitHub Login: @petertay
Microsoft Alias: pnp

So, I got bored last night and got this working, I had some issues with the deployment() function so i skipped out of it, as there is potentially either a bug in visual studio around deployment() or an issue with the documentation.
https://github.com/jeffpatton1971/arm-property-transform-collector/tree/v1.1
This is a working version of the code on the transformer collector doc, as written.

I had commented a working version of this but it was before they switched over to using GitHub :)
https://github.com/pckls/azure-resource-manager-templates/tree/master/property-transformer-and-collector


[sw/silicon_creator] Run clang-format over shutdown code

Run clang-format over the code to reduce irrelevant changes in future PRs.

How come CI didn't catch that when merging?

The // clang-format off attribute didn't used to be properly terminated in the unit tests (it was being appended to a macro). I'm not sure about the change in the shutdown.c file though (might be my local clang-format installation behaving slightly differently).

(clang-format only runs on the diff - so re-enabling it doesn't automatically force the rest of the file to be reformatted)


alpha.png not found

When trying to build with "make test" i get the following:
./tests/hicolor.test

==== data-integrity-2.1 alpha roundtrip FAILED
==== Contents of test case:

    hicolor quant alpha.png alpha-q.png
    exec gm compare -metric rmse alpha.png alpha-q.png

---- Test generated error; Return code was: 1
---- Return code should have been one of: 0 2
---- errorInfo: source image "alpha.png" doesn't exist
    while executing
"exec {*}$::hicolorCommand {*}$args"
    (procedure "hicolor" line 2)
    invoked from within
"hicolor quant alpha.png alpha-q.png"
    ("uplevel" body line 2)
    invoked from within
"uplevel 1 $script"
---- errorCode: CHILDSTATUS 2909 1
==== data-integrity-2.1 FAILED

hicolor.test:   Total   24      Passed  23      Skipped 0       Failed  1
make: *** [Makefile:26: test] Error 1

Thanks!


Added GitVio to Open-Source

I have added GitVio to Open-Source Projects.

@devXprite please resolve the merge conflict
__
  Summary: :hammer_and_wrench: This code needs a fix  |  Tags: Not reusable

Thanks @kaiwalyakoparkar for approving.