Events not firing

I'm trying to pin down exactly what's happening here. The specs don't pass, I ran it in debug mode:
creating Makefile
CFLAGS='-isysroot /Developer/SDKs/MacOSX10.6.sdk -mmacosx-version-min=10.6 -mdynamic-no-pic -std=gnu99 -Os -pipe -Wmissing-prototypes -Wreturn-type -Wmissing-braces -Wparentheses -Wswitch -Wunused-function -Wunused-label -Wunused-parameter -Wunused-variable -Wunused-value -Wuninitialized -Wunknown-pragmas -Wshadow -Wfour-char-constants -Wsign-compare -Wnewline-eof -Wconversion -Wshorten-64-to-32 -Wglobal-constructors -pedantic' /usr/bin/gcc -isysroot /Developer/SDKs/MacOSX10.6.sdk -mmacosx-version-min=10.6 -mdynamic-no-pic -std=gnu99 -dead_strip -framework CoreServices -D DEBUG=true -o '/users/dave/code/temp/rb-fsevent/bin/fsevent_watch' fsevent/fsevent_watch.c
fsevent_watch compiled
.
append_path called for: /users/dave/code/temp/rb-fsevent/spec/fixtures/custom 'path
  resolved path to: /Users/dave/code/temp/rb-fsevent/spec/fixtures/custom 'path

config.sinceWhen<PHONE_NUMBER>3709551615
config.latency      0.300000
config.flags        00000000
config.paths
  /Users/dave/code/temp/rb-fsevent/spec/fixtures/custom 'path

FSEventStreamRef @ 0x1001085c0:
   allocator = 0x7fff709faee0
   callback = 0x100001522
   context = {0, 0x0, 0x0, 0x0, 0x0}
   numPathsToWatch = 1
   pathsToWatch = 0x7fff709faee0
        pathsToWatch[0] = '/Users/dave/code/temp/rb-fsevent/spec/fixtures/custom 'path'
   latestEventId = -1
   latency = 300000 (microseconds)
   flags = 0x00000000
   runLoop = 0x0
   runLoopMode = 0x0

F
append_path called for: /users/dave/code/temp/rb-fsevent/spec/fixtures/custom 'path
  resolved path to: /Users/dave/code/temp/rb-fsevent/spec/fixtures/custom 'path

config.sinceWhen<PHONE_NUMBER>3709551615
config.latency      0.300000
config.flags        00000000
config.paths
  /Users/dave/code/temp/rb-fsevent/spec/fixtures/custom 'path

FSEventStreamRef @ 0x1001085c0:
   allocator = 0x7fff709faee0
   callback = 0x100001522
   context = {0, 0x0, 0x0, 0x0, 0x0}
   numPathsToWatch = 1
   pathsToWatch = 0x7fff709faee0
        pathsToWatch[0] = '/Users/dave/code/temp/rb-fsevent/spec/fixtures/custom 'path'
   latestEventId = -1
   latency = 300000 (microseconds)
   flags = 0x00000000
   runLoop = 0x0
   runLoopMode = 0x0

append_path called for: /users/dave/code/temp/rb-fsevent/spec/fixtures
  resolved path to: /Users/dave/code/temp/rb-fsevent/spec/fixtures

config.sinceWhen<PHONE_NUMBER>3709551615
config.latency      0.500000
config.flags        00000000
config.paths
  /Users/dave/code/temp/rb-fsevent/spec/fixtures

FSEventStreamRef @ 0x1001085e0:
   allocator = 0x7fff709faee0
   callback = 0x100001522
   context = {0, 0x0, 0x0, 0x0, 0x0}
   numPathsToWatch = 1
   pathsToWatch = 0x7fff709faee0
        pathsToWatch[0] = '/Users/dave/code/temp/rb-fsevent/spec/fixtures'
   latestEventId = -1
   latency = 500000 (microseconds)
   flags = 0x00000000
   runLoop = 0x0
   runLoopMode = 0x0

F
append_path called for: /users/dave/code/temp/rb-fsevent/spec/fixtures
  resolved path to: /Users/dave/code/temp/rb-fsevent/spec/fixtures

config.sinceWhen<PHONE_NUMBER>3709551615
config.latency      0.500000
config.flags        00000000
config.paths
  /Users/dave/code/temp/rb-fsevent/spec/fixtures

FSEventStreamRef @ 0x1001085e0:
   allocator = 0x7fff709faee0
   callback = 0x100001522
   context = {0, 0x0, 0x0, 0x0, 0x0}
   numPathsToWatch = 1
   pathsToWatch = 0x7fff709faee0
        pathsToWatch[0] = '/Users/dave/code/temp/rb-fsevent/spec/fixtures'
   latestEventId = -1
   latency = 500000 (microseconds)
   flags = 0x00000000
   runLoop = 0x0
   runLoopMode = 0x0

append_path called for: /users/dave/code/temp/rb-fsevent/spec/fixtures
  resolved path to: /Users/dave/code/temp/rb-fsevent/spec/fixtures

config.sinceWhen<PHONE_NUMBER>3709551615
config.latency      0.500000
config.flags        00000000
config.paths
  /Users/dave/code/temp/rb-fsevent/spec/fixtures

FSEventStreamRef @ 0x1001085e0:
   allocator = 0x7fff709faee0
   callback = 0x100001522
   context = {0, 0x0, 0x0, 0x0, 0x0}
   numPathsToWatch = 1
   pathsToWatch = 0x7fff709faee0
        pathsToWatch[0] = '/Users/dave/code/temp/rb-fsevent/spec/fixtures'
   latestEventId = -1
   latency = 500000 (microseconds)
   flags = 0x00000000
   runLoop = 0x0
   runLoopMode = 0x0

F
append_path called for: /users/dave/code/temp/rb-fsevent/spec/fixtures
  resolved path to: /Users/dave/code/temp/rb-fsevent/spec/fixtures

config.sinceWhen<PHONE_NUMBER>3709551615
config.latency      0.500000
config.flags        00000000
config.paths
  /Users/dave/code/temp/rb-fsevent/spec/fixtures

FSEventStreamRef @ 0x1001085e0:
   allocator = 0x7fff709faee0
   callback = 0x100001522
   context = {0, 0x0, 0x0, 0x0, 0x0}
   numPathsToWatch = 1
   pathsToWatch = 0x7fff709faee0
        pathsToWatch[0] = '/Users/dave/code/temp/rb-fsevent/spec/fixtures'
   latestEventId = -1
   latency = 500000 (microseconds)
   flags = 0x00000000
   runLoop = 0x0
   runLoopMode = 0x0

append_path called for: /users/dave/code/temp/rb-fsevent/spec/fixtures
  resolved path to: /Users/dave/code/temp/rb-fsevent/spec/fixtures

config.sinceWhen<PHONE_NUMBER>3709551615
config.latency      0.500000
config.flags        00000000
config.paths
  /Users/dave/code/temp/rb-fsevent/spec/fixtures

FSEventStreamRef @ 0x1001085e0:
   allocator = 0x7fff709faee0
   callback = 0x100001522
   context = {0, 0x0, 0x0, 0x0, 0x0}
   numPathsToWatch = 1
   pathsToWatch = 0x7fff709faee0
        pathsToWatch[0] = '/Users/dave/code/temp/rb-fsevent/spec/fixtures'
   latestEventId = -1
   latency = 500000 (microseconds)
   flags = 0x00000000
   runLoop = 0x0
   runLoopMode = 0x0

F
append_path called for: /users/dave/code/temp/rb-fsevent/spec/fixtures
  resolved path to: /Users/dave/code/temp/rb-fsevent/spec/fixtures

config.sinceWhen<PHONE_NUMBER>3709551615
config.latency      0.500000
config.flags        00000000
config.paths
  /Users/dave/code/temp/rb-fsevent/spec/fixtures

FSEventStreamRef @ 0x1001085e0:
   allocator = 0x7fff709faee0
   callback = 0x100001522
   context = {0, 0x0, 0x0, 0x0, 0x0}
   numPathsToWatch = 1
   pathsToWatch = 0x7fff709faee0
        pathsToWatch[0] = '/Users/dave/code/temp/rb-fsevent/spec/fixtures'
   latestEventId = -1
   latency = 500000 (microseconds)
   flags = 0x00000000
   runLoop = 0x0
   runLoopMode = 0x0

Failures:

  1) FSEvent should work with path with an apostrophe
     Failure/Error: @results.should == [custom_path.to_s + '/']
       expected: ["/users/dave/code/temp/rb-fsevent/spec/fixtures/custom 'path/"]
            got: [] (using ==)
       Diff:
       @@ -1,2 +1,2 @@
       -["/users/dave/code/temp/rb-fsevent/spec/fixtures/custom 'path/"]
       +[]
     # ./spec/rb-fsevent/fsevent_spec.rb:30:in `block (2 levels) in <top (required)>'

  2) FSEvent should catch new file
     Failure/Error: @results.should ==<EMAIL_ADDRESS>+ '/']
       expected: ["/users/dave/code/temp/rb-fsevent/spec/fixtures/"]
            got: [] (using ==)
       Diff:
       @@ -1,2 +1,2 @@
       -["/users/dave/code/temp/rb-fsevent/spec/fixtures/"]
       +[]
     # ./spec/rb-fsevent/fsevent_spec.rb:40:in `block (2 levels) in <top (required)>'

  3) FSEvent should catch file update
     Failure/Error: @results.should == [@fixture_path.join("folder1/").to_s]
       expected: ["/users/dave/code/temp/rb-fsevent/spec/fixtures/folder1/"]
            got: [] (using ==)
       Diff:
       @@ -1,2 +1,2 @@
       -["/users/dave/code/temp/rb-fsevent/spec/fixtures/folder1/"]
       +[]
     # ./spec/rb-fsevent/fsevent_spec.rb:49:in `block (2 levels) in <top (required)>'

  4) FSEvent should catch files update
     Failure/Error: @results.should == [@fixture_path.join("folder1/").to_s, @fixture_path.join("folder1/folder2/").to_s]
       expected: ["/users/dave/code/temp/rb-fsevent/spec/fixtures/folder1/", "/users/dave/code/temp/rb-fsevent/spec/fixtures/folder1/folder2/"]
            got: [] (using ==)
       Diff:
       @@ -1,3 +1,2 @@
       -["/users/dave/code/temp/rb-fsevent/spec/fixtures/folder1/",
       - "/users/dave/code/temp/rb-fsevent/spec/fixtures/folder1/folder2/"]
       +[]
     # ./spec/rb-fsevent/fsevent_spec.rb:61:in `block (2 levels) in <top (required)>'

Finished in 13.4 seconds
5 examples, 4 failures

I'm running ruby-1.9.2-p180
Mac OSX 10.6.6
I'm not sure what other information would be helpful. If i build the debug binary and run it on a directory i'm not seeing any events firing.
Any ideas?

@andreyvit - the final response from apple is that this isn't a bug because the fsevents table is case sensitive. i think the point of the bug report, being that path resolution apis weren't necessarily returning the correct case for a path, was completely ignored (so client and fseventsd can still have different paths). re-submitting...
in the meantime, i added file reference url resolution as the default 10.6+ strategy. no idea if that'll be more consistent. sigh.
https://github.com/ttilley/fsevent_watch/commit/e249e0fa3cf7c9b7e774524bb1d130c6b334505c#L1R49

The wording even sounds like a "go fuck yourself"...

The equivalence of what is happening is:

fsevents_tool -files ./test
mv ./test ./bar
echo "" > bar/test

Nobody should or would expect events to arrive on test/test.
The user workaround is as follows:

Do not rename watched paths.  (mv /User/foo/bar /Users/foo/BAR)
Watch the parent directory instead of the directory you wish to rename.

/me slams head into desk repeatedly

I guess they've completely misunderstood the bug report. Can you post a clarifying comment for them? Or is there no way to follow up? (Haven't ever posted on RADAR myself.)

Would you still like to keep my directory in science experiment mode? Or can I move things around? :)

Well, personally, I would be super-happy if you could download LiveReload 2 app (http://livereload.com/), add your directory there and see if it triggers the error. If you choose to do so, you can report the result to<EMAIL_ADDRESS>to avoid spamming people here.
Other than that, there hasn't been any development on this topic, so I guess no — feel free to delete/move it. Thanks!

@agibraiter - my bug report has stalled. 100%. after clarifying the issue, any questions about the status of the bug go ignored. the utility they mention in their bug reproduction example (for the incorrect bug) doesn't actually exist, at least publicly. sending the raw contents of /.fseventsd/ would be a potential risk for you, and painful for us, as it would include binary representations of all events, coalesced to a degree. I don't know of an easy way to look at the raw HFS+ inode data alongside fork data to determine potential on-disk issues.
The only thing I can think of is something that using the underlying internal-only fsevent device API that fseventsd itself uses, and censoring that yourself to only include data you want to share. Something like: http://www.osxbook.com/software/fslogger/
As far as I can guess, there should STILL be an fsevent notification somewhere... it's just incorrect in regards to case. If you can include ls output of the directory, stat -x of the directory, and a snippet of output from something like fslogger while editing files under the broken directory, that might be enough info to push forward the bug report with apple.

...not that it'd particularly help us much, but i'd really like to see this fixed in macos.

First bug report was March 30, 2011. Last bug report was October 25, 2011.
The majority of reporters had a problem with a path underneath a dropbox sync point. Dropbox doesn't have a UI for performing updates. Growl 1.3 was released on the mac app store around November 9, breaking dropbox notification support. An updated version of dropbox was available not long after, but that may have been the first time a lot of people had ever updated. It's a bit suspicious to me that bug reports stopped coming in right when a bunch of people had a reason to update dropbox... which doesn't use the public fsevents api, instead opting to use the private api.
Given the details and timeframe I'm happy to consider this bug unresolvable and dead. Unresolvable because it's looking likely that the issue isn't with macos or my own software, and dead because I haven't had any recent reports and I'd really rather stab myself in the face than spend more time on this.
...Since I was desperately waiting to resolve this issue before making another release, expect one in the hopefully not too distant future that also resolves the issue with installing without xcode.

Hehe, I've recently had two users encounter this bug (for whom LiveReload has detected it and referred them to me). Both on Dropbox. Of course, I have no idea if they have updated Dropbox recently or not. (Does it even require manual updating? I don't think it ever asked me about updates.)

I know this is closed, but I recently ran into this issue in my project folder. I have Dropbox installed, but my project folder is not in Drobox. I renamed my project folder from "sites" to "Sites" and everything started to work fine. It still works after renaming it back to "sites".

@mstalker - frustrating, isn't it? i originally thought this was completely unrelated to dropbox because the problem wasn't always under the dropbox folder, but it does appear that dropbox (specifically /Library/DropboxHelperTools/Dropbox_VERSION/dbfseventsd) is still involved in triggering the state where this problem occurs. I'd be lying if I said I fully understood the scenario, but upgrading dropbox has improved the situation dramatically. If you don't mind, can you look and see what version is being run? Locally I have u501.
I recently discovered that dropbox for linux can also cause problems on that platform. How it manages to do that, I have no idea. Compass.app has a bug report that FS notifications (via inotify) simply don't work on ubuntu under the dropbox folder.

I just faced this issue (LiveReload warned me) and I don't even have Dropbox installed :|
I don't use it, and have never on my mac (latest Lion)
What do you say about that?

I say you are a unique and fascinating specimen! It'd be incredibly useful
to take a peek at your filesystem if that is at all acceptable. I need to
take my poor cat to the vet in a few hours and, erm... forgot to go to
sleep last night, but should be otherwise available. My AIM handle is
YourTravis and I'm on irc.freenode.net as ttilley most of the time (usually
in #growl #ruby-lang #macruby #jruby #guard (the guard channel is probably
best for this)).

          *

Travis Tilley
(201)676-0873
On Sat, Feb 4, 2012 at 3:19 AM, Dheeraj Kumar <
<EMAIL_ADDRESS>
wrote:

I just faced this issue (LiveReload warned me) and I don't even have
Dropbox installed :|
I don't use it, and have never on my mac (latest Lion)
What do you say about that?

Reply to this email directly or view it on GitHub:
https://github.com/thibaudgg/rb-fsevent/issues/10#issuecomment-3808995

Sure, you can inspect it. Is it okay to move the files and just keep the empty folder? I'll contact you on IRC.

Public ann: @bdkjones (the author of CodeKit) took his shot at getting Apple Developer Support fix this problem. So far, the progress is mixed, but we did reveal some additional data.
We would again very much like to poke around on a machine with a broken folder. If you are affected by this bug and can afford to leave the folder unfixed for a few days and can let us screenshare into your machine, please ping me!

After days of research, I have narrowed down the cause of this issue. It IS Apple's fault and it IS something they have to fix; there is no workaround for us.
FSEvents is case-sensitive. It treats /Users/folder and /Users/Folder as two separate paths. Moreover, when you start an FSEvents stream, the path you pass in is "standardized" to the value that realpath() returns. However, this value does not always match what the kernel writes to /dev/fsevents, which is the exact root cause of this bug.
Here is a screenshot of my ongoing conversation with Apple Developer Technical Support:

Apple, quite idiotically, has not yet acknowledged that this is something they should fix. Even if they do acknowledge that, it could be years before a fix makes it into OS X.
What We Need
In the meantime, I propose that we focus on ways to make the kernel report events using the same path that realpath() returns. So, from the example above, the kernel is reporting events for /Folder/database but realpath() has that path as /Folder/Database.
Since FSEvents always uses the realpath() variation, there must be some sequence of actions we can take (programmatically) on Database that will cause the kernel to start using the same capitalization as realpath().
We just need to discover what those actions are. I have asked Apple DTS for ideas. But it SHOULD be possible to "fix" broken folders somehow.

@bdkjones This is all that's needed:
BrokenFolder="/some/path/to/database"
ParentOfBrokenFolder="$(dirname "$BrokenFolder")"
NameOfBrokenFolder="$(basename "$BrokenFolder")"
cd "$ParentOfBrokenFolder"
mv "$NameOfBrokenFolder" "$NameOfBrokenFolder.broken"
mkdir "$NameOfBrokenFolder"
mv "$NameOfBrokenFolder.broken"/* "$NameOfBrokenFolder.broken"/.* "$NameOfBrokenFolder.broken/"
rmdir "$NameOfBrokenFolder.broken"

...properly implemented in native code and adjusted to preserve the permissions and other attributes of the broken folder.

...My problem is that I'm not very comfortable doing that to the /Users folder, and also, as we may have figured, my detection method has false positives. Btw, would be nice to hear your latest thoughts on those false positives — the public would greatly benefit from a reliable way to detect this case.

I would definitely have reservations about using the above method on /Users. First, you'd have to be running as root (which immediately rules out App Store apps). Second, if anything went wrong you'd nuke the user's entire Mac. Third, even if this process completed 100% correctly, it's impossible to say what side effects would happen. Nothing on the system expects the /Users folder to temporarily disappear... it just seems like stuff we can't even imagine would flip the hell out.
This is true of other common folders like ~/Documents and ~/Dropbox, etc. Who knows what apps are using files that are located in those folders and what those apps will do if those files momentarily move as the root folder is replaced?
What I'm wondering is if there's not some less brute-force method that will achieve the same result. Modifying the inode data, changing permissions and then changing them back, some stupid-low-level HFS call... anything like that.

On the other hand, dozens of users have successfully completed this sequence using sudo on their macs following my instructions, and everything is working fine. For Mac App Store app, a pointer to a non-Mac App Store fixing tool would work. And a lot of people have the problem on a fairly mundane folder like ~/Projects.
But yes, sometimes a less dangerous action like renaming the folder and renaming it back, or even something as simple as rebooting the Mac, fixes the issue. The problem is that no other step has been known to be reliable so far.
One of the interesting steps that works intermittently is repairing the permissions using Disk Utility. I wonder what exactly that command does, but I imagine it's a bunch of chmod's. So perhaps for some folders, any inode change fixes the problem, and for some other folders, you need to nuke the inode completely.

Yea. I've asked Apple DTS for their recommendation on a programatic way to "restore" the folder so that the kernel writes events to /dev/fsevents using the same casing as realpath(). I'll report what I hear from them. Hopefully the engineers can give us a simple, robust way to do this.
As for the false-positives issue: I have no idea. I just ran your tool and passed in my home directory, using the same casing that you see in the Finder. It found 13,800 broken folders. Interestingly, all of them appear to be ones that are somehow exposed to "networking".  Given that most folks who report the FSEvents bug to me are using Dropbox and other network-syncing clients and the folder most often affected is Dropbox, I'm guessing that whatever is corrupting the filesystem info to produce these "broken" folders more often affects those folders that are somehow exposed to networking.
This does not, however, explain how folders like /Users become affected.

So I grabbed the open source version of realpath() that Apple has released here: http://www.opensource.apple.com/source/Libc/Libc-498.1.7/stdlib/FreeBSD/realpath.c
I put this in my app and called it. The returned path matched the casing that:

The kernel is writing to /dev/fsevents
All of Apple's standard file-access APIs return.
The alias manager returned.

So, clearly, the open source version of realpath() is not what Apple is actually using in OS X. Because when I call the system's version of realpath() I get a result with one path component capitalized, even when I pass in a path that has that component lower-cased. And there is nothing in the open source implementation that would change the case of a letter.

I'm going to start writing a replacement for FSEvents. It shall be called FuckFSEvents and it will read directly from /Dev/fsevents in a case-insensitive manner—like a goddamn sane person would.

@bdkjones Wait a sec, that's a very old version of realpath.c; my understanding is that it simply doesn't do anything to the path that you pass, short for resolving symlinks and relative references.
I've previously sent you a different version of realpath from OS X 10.0, and it works in a completely different way, using getattrlist system call to obtain the actual name of the folder (which performs a vnop_getattrlist call into the non-open-source HFS+ driver).a
Now here's a question (which was why I spent time disassembling the FSEvents binary): if you force FSEvents to use a different implementation of realpath, will it cure the problem? I believe that the answer is yes, and I think that's an avenue worth exploring before you go writing a root tool.

Yes—if we could get FSEvents to watch the path as returned by the alias API or any of Apple's regular file path APIs, it would solve the problem. realpath() is returning capitalized Database as the watched folder name. Everything else is spitting out lowercase database.
So either realpath() is wrong, OR there is a bug down in the kernel or HFS+ driver. It is possible that realpath() is the correct path and that everything else is wrong.
But the Apple engineer has explained to me that, as far as the kernel is concerned, case-sensitivity does not matter. /Folder/Database and /Folder/database are the exact same path in kernel space. So I have no idea which one of these things is messed up.

Here is a bit of code that I just wrote that intercepts all call to realpath from the current process, including the ones made by FSEvents. Feel free to plug in your other implementation and see how it fairs on your system. It uses mach_override from mach_star.

How do you suggest we force fsevents to use a different implementation of realpath()?
 I just need events that are happening right now.

The kernel provides the PID of the process responsible for an event, but the FSEvents API does not expose that information. We could use this to ignore unwanted changes (like a branch-switch in Git).

The events as they are written to /dev/fsevents are REALLY nicely organized. Example: when a file is renamed or moved, the old path and the new path of the file appear in the SAME event. If you use the FSEvents API, you only get the old path. Discovering the new path is a bunch of work.

Using GCD, it's easy to build a daemon that reads from the kernel's I/O stream without blocking, then send event information to an app using the XPC framework.

Cool. I can give that a go in the morning (it's 2AM here), but I imagine it won't work because the call to realpath() isn't happening in my process; it's happening in the fseventsd process.
We'd need something like method-swizzeling, where the call to the function is overridden in ALL processes, system-wide.

because the call to realpath() isn't happening in my process; it's happening in the fseventsd process

I cannot say whether the call is also happening in the daemon, but I saw multiple calls to that function from within the FSEvents framework binary (i.e. in-process).

Updated FSEventsFix.c to include the BSD implementation of realpath and to call it properly.
I confirm that:

LiveReload still works.
The hooked realpath gets called from FSEventStreamCreate.

Now you only need to drop 3 files into your project, call FixFSEvents() from main() and tell us if it worked or not.

One more confirmation: if I change my realpath to always return an uppercase path, FSEventStreamCopyPathsBeingWatched reports the uppercase path as the one being monitored.
At this point, I expect a miraculous cure report from you tomorrow. :D
The up-to-date code is not in my features/fix-fsevents branch under mac/FSEventsFix, in particular, here's FSEventsFix.c. This will stay current with any further updates I make, until I marge and delete the branch.

I've updated the code with the latest version of mach_override (unfortunately, it's no longer just 2 files, but the previous one had a bug). I've also attempted to use dyld interposing and Facebook's fishhook for overriding, but without success so far.

 The broken folder on my Mac has suddenly stopped being broken. I'm looking for a new one that's still broken to test the mach_override approach.

Confirmed. The mach_override approach solves the problem.
Here is what happens without the override applied:

And once the override is applied, here is the result:

The trouble is that we can't ship the mach_override hack. The buggy version of realpath() does a bunch of stuff that's desirable, such as crossing mount points, etc. The old BSD version does not do these things.
My guess is that the root of this bug lies in the way Apple's newer realpath() function calls down into getattrlist to retrieve the name of each path component. That seems to be the smoking gun.

Personally, I'll be shipping it if I can pass Mac App Store validation.
But you make a good point; instead of the old version, we should take the latest one and simply ignore the part that uses the name coming from kernel.
And hey, we're finally making some real progress here! Thanks for your persistence, Bryan, everyone else has given up by now.

Hell, I gave up several years ago.
On Thursday, May 14, 2015, Andrey Tarantsov<EMAIL_ADDRESS>wrote:

Personally, I'll be shipping it if I can pass Mac App Store validation.
But you make a good point; instead of the old version, we should take the
latest one and simply ignore the part that uses the name coming from kernel.
And hey, we're finally making some real progress here! Thanks for your
persistence, Bryan, everyone else has given up by now.
—
Reply to this email directly or view it on GitHub
https://github.com/thibaudgg/rb-fsevent/issues/10#issuecomment-101931607
.

--
Sent from Gmail Mobile

Made two more changes:

Replaced my realpath implementation with the one from OS X 10.10 — it does everything except replacing names.
Got fishhook hooking to work — it's significantly safer than mach_override (it's even compatible with iOS App Store).

I'm very happy with how this works now, and definitely shipping this.
@ttilley, @thibaudgg Are you guys comfortable shipping a fix like this (overriding realpath implementation process-wide)? What do you need to include it? Should I extract the code into a separate repository?

Here we go: andreyvit/FSEventsFix. Now it's just two files, and you really only need one of them (FSEvents.c). Include the file into the app or library, and the dreaded FSEvents problem will be fixed.
No public API defined or symbols exported, so no concerns about possible name clashes. Multiple copies of FSEventsFix can co-exist in a single process, so it can be included into a library without reservations.

Nice work! What did you have to do to get Fishhook to work correctly?
Also: what do you think about the ability to toggle the replacement in and out? 95% of the time, there's no need to replace realpath() with our "fixed" version. So, to be safe, what if we only forced the replacement when we had to? We have the ability to detect whether a path is broken (just run Apple's realpath() and do a case-sensitive compare against the path obtained from an FSRef alias). So what if, every time we add a path to an FSEvents stream, we first check to see if it's broken and, only if it is, THEN we swap the functions?
This has the major advantage that once Apple fixes the problem, our code simply stops swapping the functions. No update required.

What did you have to do to get Fishhook to work correctly?

Remove a leading underscore in the symbol name. For some reason they're comparing imported symbols starting with the second character. :-)

So, to be safe, what if we only forced the replacement when we had to?

Probably a good idea, although I would question whether the added complexity is worth it.
Pros: like you've said, it's marginally safer, and it will auto-disappear when the bug gets fixed.
Cons: we have to continue running realpath against every single folder (or maybe even file, who knows) in the project, I was looking forward to removing that particular code. We have to also add the logic to re-try monitoring after we detect a broken folder.

No, we only have to run the check on the folder we tell fsevents to watch. Child items of that folder don't matter.
The check is fast and easy, since it only has to be done on a handful of folders (the ones we open an FSEvents stream on) and I think the added safety is worth it.
And the auto-disappearing benefit is definitely nice.
Perhaps as an optional way to use the fseventsfix library? I can fork it and implement it myself, but since you're far more familiar with the code it would go much faster for you.
Sent from my iPhone

On May 14, 2015, at 10:53, Andrey Tarantsov<EMAIL_ADDRESS>wrote:
What did you have to do to get Fishhook to work correctly?
Remove a leading underscore in the symbol name. For some reason they're comparing imported symbols starting with the second character. :-)
So, to be safe, what if we only forced the replacement when we had to?
Probably a good idea, although I would question whether the added complexity is worth it.
Pros: like you've said, it's marginally safer, and it will auto-disappear when the bug gets fixed.
Cons: we have to continue running realpath against every single folder (or maybe even file, who knows) in the project, I was looking forward to removing that particular code. We have to also add the logic to re-try monitoring after we detect a broken folder.
—
Reply to this email directly or view it on GitHub.

No, we only have to run the check on the folder we tell fsevents to watch. Child items of that folder don't matter.

Are you sure about that? I could swear I had a case where a subfolder was broken, but I also assumed it was common knowledge. Now that you question that, I'm no longer sure if it happened.

Perhaps as an optional way to use the fseventsfix library?

Sure, I'll add it.

I'm sure. I just tested it over here. I added the parent folder of CloudDocs to my app after verifying that CloudDocs is still broken. FSEvents without the fix applied returned events just fine for files inside the broken folder.
What the FSEvents daemon probably does is check each event that gets written to dev/fsevents by seeing if the event's path starts with any of the paths we tell the stream to watch (in case-sensitive way).
So there's no need to check child folders/files.

 Debugging this without having an example case to test with is impossible. I can see why Apple refused to acknowledge it as a bug for years.

The real kicker is that this isn't even a bug in FSEvents.
So I suppose we all need to file a new Radar against realpath(). If I write one up and post it on OpenRadar, could everyone here file a duplicate copy? I hear that doing that makes Apple prioritize the bug report a bit more.

FSEvents ought to be case-insensitive

This isn't necessarily true; OS X does support case-sensitive installs, and iOS always uses a case-sensitive file system. I'd say that without the realpath bug FSEvents works pretty well as it is.

Yep, that was a real strike of luck for all of us.

could everyone here file a duplicate copy?

Sure. I can even file a duplicate DTS ticket. :-) My dev program renewal time is approaching fast, and I still have 2 unused tickets. Let's prepare a good comprehensive guide on this bug and file away. @bdkjones, how about summarizing all our findings on a GitHub wiki or gist so that we can edit & extend it over time?

OS X does support case-sensitive installs

Yes. But not by default. 99.9% of the Macs out there are running case-insenstive installs. And, FSEvents should, in my opinion, match the behavior of whatever volume I open a stream on. But, the Apple guys tell me there are many edge cases (such as composed character sequences in non-English languages) that make implementing that a nightmare. To which my reply is, "You have $200 Billion Dollars in the bank. I think you can tackle it."

how about summarizing all our findings on a GitHub wiki

Will do that this evening and post a link here. I will also upload all the tools required to do the same debugging that I did, so Apple can replicate the process on their machines. I may even record a freaking screencast to walk through it, as long as I still have a broken folder on my Mac.

as long as I still have a broken folder on my Mac

Talk about a scarce resource. :-)
Of course, it would be super-awesome to be able to reproduce the bug on demand. Everything points to just having a ‘folder with a lot of changes’ (network sync), but I once tried that and couldn't get the bug to happen. Maybe it's something else — maybe it's ‘changes while power napping’, or smt like that.

it would be super-awesome to be able to reproduce the bug on demand.

Agreed. Dropbox and syncing clients do seem to aggravate the problem. But I don't have Dropbox installed on my Mac and the two folders that are broken for me are buried in the new Photos app data file and iCloud in ~/Library.
Worse, the broken folder in the Photos data file just magically repaired itself yesterday, after being broken for days. And I didn't take any explicit action to repair it. In fact, I purposefully avoided launching Photos or repairing my disk/permissions, etc. to avoid losing the broken folder.

@andreyvit @bdkjones if you think you can fix that directly in rb-fsevent please don't hesitate to submit a pull request I would merge it with joy :dancers:

VICTORY!
Apple has acknowledged this as a bug!
They want a radar on it immediately and they want to correct it quickly because it is, technically, file-system damage. Here is a screenshot of the response from the DTS Engineer:

As requested, here is the Wiki page describing everything we know, with links to all the required tools to diagnose and debug the problem:
https://github.com/bdkjones/fseventsbug/wiki/realpath()-And-FSEvents

@andreyvit Another excellent reason we need to implement the function-swapping only when required is that it's very, very likely we'll never know exactly when Apple ships a fix to this problem. They aren't going to put this type of low-level change in a set of release notes and no one is going to email us.
If we don't swap the functions only when needed, the only way we'll have to determine when the fix is no longer needed is by shipping a version of our apps without the swapping and waiting to see if anyone reports broken folders anymore.

Plus, Apple may enhance realpath() down the road, or change the way it operates entirely if the system ever moves off of HFS+, etc. At that point, once they've fixed this issue, we don't want to be swapping the functions any longer—it would probably break things.

@bdkjones Have you filed a new RADAR yet? I think it makes sense to dupe it verbatim.

After some thinking, I've changed how FSEventsFix works. Now it requires you to call FSEventsFixInstall() to install, and also provides FSEventsFixIsBroken(path) to check if a folder is broken.
Btw, I've realized that all the FSAlias stuff was basically returning exactly the same value we pass in, so I've removed it from the check.
I've also added versioning via FSEventsFixVersion. The current version is set to 0.9.0, although I intend to set it to 1.0.0 right after @bdkjones confirms he's happy with the library.
Exposing public symbols is not actually a problem because FSEventsFix will be used either as part of an application (under a single developer's control), an external helper tool (e.g. fsevents_watch in rb-fsevents) or a dynamically-loaded library (e.g. a Node.js native module).

The Apple engineer has responded and has given me a very serious warning about replacing the realpath() function.
He has requested that I not post screenshots of our conversation, but I will summarize his warnings here:

It is not possible to replace realpath() for just FSEvents. If we swap it out, we're doing so for ALL system frameworks. While FSEvents make work correctly, there's no telling what other frameworks will do.

Non-ASCII file names. The original BSD function assumed only ASCII. The replacement realpath() function was designed to handle non-ASCII filenames (which may be one reason they added the low-level calls down to the filesystem). There is no telling how our replacement realpath() might deal with non-ASCII characters.

HFS+ is one of the only filesystems to permit hard-links to directories. This was not accounted for in the original BSD realpath().

Kevin's Suggested Workaround
Kevin suggests that, once we identify a broken folder, we call rename() (the low-level API) to force both HFS+ storage structures to update their values and bring the lettercasing in sync. I am testing this approach now to see if it solves the problem. I will post results when I have them.

Okay, I've confirmed that using rename() will repair a broken folder as long as you first rename it to something temporary and then back to the correct name.
I've updated my Wiki entry with full details and exact steps, including additional information from Kevin: https://github.com/bdkjones/fseventsbug/wiki/realpath()-And-FSEvents

@bdkjones First, we're not replacing with the original BSD realpath, we're replacing it with exactly the same version the system uses, just without the kernel call.
Second, if we wanted to, it is of course possible to replace the implementation only for FSEvents, because the only call to realpath that matters to us happens synchronously from FSEventStreamCreate.
Third, let's not succumb to paranoia. Our implementation of realpath works fine, and I don't foresee any reason why the system frameworks would have a problem with it. I'm almost sure that for the purposes of our app, even the original BSD implementation of realpath would work just fine.
And finally, they're wrong about rename, it will not always repair a broken folder. For a number of users, yes, but for a lot more users it does not help, that's one of the certain things about this bug.

@thibaudgg I'll send a pull request once we come to some sort of conclusion here, but meanwhile, don't you think that it's a bit too active to be a closed bug? :-)

Fair enough. ;)

It's unfortunate that any possible fix to this, at the moment, will be a hack with its own downsides and possible negative effects. How would you only duck punch the call to realpath() via FSEventStreamCreate()? You'd permanently hook FSEventStreamCreate() with another hook that hooks realpath(), performs the original call, and then removes the hook before returning the result?
Hey bro, I heard you like method swizzling, so i swizzled your method so you can swizzle your method from within your swizzled method. (sorry, couldn't help myself)

@ttilley Actually I do like method swizzling, and generally I'm not at all afraid of shipping stuff like that. Adding a bit of maintenance cost, yes, but in this case it's clearly acceptable.
There are four options now:

Hook permanently, but use a global flag to choose between our and original implementations. Set the flag when calling FSEventStreamCreate, reset afterwards.
Hook permanently, but examine the call stack at run time to choose the implementation.
Hook for the duration of FSEventStreamCreate, and then unhook.
Hook permanently and consider it a gift to all the other code that could also be affected by the bug.

Generally, I'm not afraid of shipping stuff like that. Years of production experience. Have I ever told you a story of software RAID for RAM? Sometimes the way to get the job done is to inject code into another process, scan the memory for certain machine code patterns and replace it with a modified code, and then combat memory corruptions (caused by an erratic host process) by keeping multiple copies of your state, comparing the copies on each access. That is a bit of a dirty trick, but it's in production and happily in use by a huge enterprise.
I don't see the realpath fix as unfortunate or undesirable, globally, so I'm leaning towards option 4 for my app. I'm open to having option 1 or 3 available in the API, though.
I do have a concern, though, that sometimes the path passed in may not be in a  correct case to begin with, especially if the path may be coming from the command line (like in rb-fsevents case, or in case of LiveReload's URL scheme API). I'm thinking to change how our realpath works to attempt to restore the correct case by enumerating the directories and finding case-insensitive matches for path components. As a bonus, this would also more closely match the behavior of realpath.

Let's implement number 3. I think that's a fair balance between risk and reward. But I also think the API needs a way to determine if a path is broken (or is a subfolder of a broken folder) and only swap the functions if a broken folder is detected.
Sent from my iPhone

On May 16, 2015, at 03:37, Andrey Tarantsov<EMAIL_ADDRESS>wrote:
@ttilley Actually I do like method swizzling, and generally I'm not at all afraid of shipping stuff like that. Adding a bit of maintenance cost, yes, but in this case it's clearly acceptable.
There are four options now:
Hook permanently, but use a global flag to choose between our and original implementations. Set the flag when calling FSEventStreamCreate, reset afterwards.
Hook permanently, but examine the call stack at run time to choose the implementation.
Hook for the duration of FSEventStreamCreate, and then unhook.
Hook permanently and consider it a gift to all the other code that could also be affected by the bug.
Generally, I'm not afraid of shipping stuff like that. Years of production experience. Have I ever told you a story of software RAID for RAM? Sometimes the way to get the job done is to inject code into another process, scan the memory for certain machine code patterns and replace it with a modified code, and then combat memory corruptions (caused by an erratic host process) by keeping multiple copies of your state, comparing the copies on each access. That is a bit of a dirty trick, but it's in production and happily in use by a huge enterprise.
I don't see the realpath fix as unfortunate or undesirable, globally, so I'm leaning towards option 4 for my app. I'm open to having option 1 or 3 available in the API, though.
I do have a concern, though, that sometimes the path passed in may not be in a correct case to begin with, especially if the path may be coming from the command line (like in rb-fsevents case, or in case of LiveReload's URL scheme API). I'm thinking to change how our realpath works to attempt to restore the correct case by enumerating the directories and finding case-insensitive matches for path components. As a bonus, this would also more closely match the behavior of realpath.
—
Reply to this email directly or view it on GitHub.

Also: I think that the reason rename() failed for some folks is that they renamed a subfolder of a broken folder rather than the broken folder itself. I can't confirm that, of course, but the rename() operation going through a temporary folder name physically unlinks the filesystem object representing the old folder and creates a new object representing the new folder. This results in an update to both underlying HFS+ catalog stores.
rename() should always work, provided that it targets the actual broken folder, which may not be the last folder of a given path.

Bryan, that theory definitely makes sense in light of Apple's explanation, but they may be mistaken about the root cause or some other aspect.
LiveReload already finds and reports the root broken folder, not just the watched one. That's the reason why it can point to /Users.
And, very often, nothing short of deleting the offending folder and creating a new one helps.
I will, however, pay attention to specifically confirm it the next time it comes up.

Agreed.
But what I'd like to ship is this process:

Find a broken folder by comparing NSURL to realpath().
Implement a rename() if possible (unless it takes root access)
Repeat the check between NSURL and realpath() to verify the folder is fixed.
If the folder is still broken, use your library to swap the realpath() function before creating the FSEvents stream.
Once the stream is created, unhook the function and swap back in the original realpath().

All right, the new version of FSEventsFix allows the flow that Bryan has suggested, see the updated README. For the fixing part, it tries to do a rename into itself, which doesn't return an error, but I'm not actually sure if it does anything. I don't think I'll be making a lot of further changes; @bdkjones, feel free to extend _FSEventsFixAttemptRepair to your liking.
I don't do NSURL stuff when checking for broken folders, but if you really want to, you can pass the path through NSURL before calling FSEventsFix APIs.
(I've traced the internals of [NSURL initWithPath], which calls into CFURLCreateWithFileSystemPath, which is a huge function with tons of code paths, but for a normal absolute path it seems to simply create a file system representation of the path, presumably in a way similar to [NSString fileSystemRepresentation], and then create a URL from the file system representation by adding percent escapes. So using NSURL doesn't seem like a hugely valuable addition for me, besides, LiveReload's internals use NSURL almost everywhere, so the path will already be coming from that source.)

Since realpath() was wonky, I made rb-fsevent run CFURLCreateWithFileSystemPath(), chop off imaginary paths until I hit an existing path (being called with a path that doesn't exist yet is valid), convert to volId/inodeId pair, and ask what the system thinks the path for that should be (using CFURLCreateFileReferenceURL() followed by CFURLCreateFilePathURL()). This also should drop down to the filesystem to ask what the path should be, but I have no idea what calls are made in the background and I have nothing broken to test against.
I guess it might be worth adding some ObjC to the project just to make sure i'm going through a codepath documented to be "correct".
@thibaudgg - i'm thinking of dropping support for pre-10.9 in the main binary if i'm going to be adding a workaround. there was a local root exploit discovered in the system preferences frameworks that was only fixed in 10.10 and apple says would be too much work to fix elsewhere... effecting 10.7-10.9... so nobody in their right mind should be running anything other than 10.10 at this point (or 10.6 if you have an i386 mac, or 10.5 on ppc, but even then you're better off installing linux to get a secure and up-to-date system). recent xcode can't even target pre-10.9.

@ttilley In turn, it may be worth adding some CoreFoundation to FSEventsFix. Right now it's all at POSIX level. And I don't think [NSURL initWithPath:] can be considered more correct; it's a one-line wrapper around CFURLCreateWithFileSystemPath.

I can definitely confirm that calling rename() with identical from and to paths does NOT fix a broken folder. You have to go through a temporary-name step for that fix to work.

@bdkjones Good, thanks. But are you sure we want to silently rename things? Also, how many broken folders remain in your disposal? :)

Yea, I discussed that with the Apple guy. He says that the entire filesystem is designed to handle things moving around, even if the thing we're moving is /Users. Everything that has a file open in an affected folder has an open file descriptor, which lets them resolve the path to the filesystem object. He says there should be no problems renaming a folder to something temporary and then renaming it back.

I have zero broken folders left, sadly.

@bdkjones The scenario I'm worried about if someone else is watching the folder for changes, and when it goes away, it's registered as a change, and they immediately do something that recreates the missing folder.
Another scenario: you have a Git checkout running, and while waiting, you pre-emptively add the folder to LiveReload/CodeKit. Bam, your Git is complaining about a missing folder. (Or will it? Worth testing, but you can see why I'm worried.)

Yea, but in theory anything that's watching that folder for changes using FSEvents is also broken and won't receive any events. Of course, if they're watching a parent folder of the broken folder, they will. As will anything that's subscribed to changes using kQueues instead of FSEvents.
We could solve this by showing a dialog box to the user before our apps rename folders. Something like, "Hey, folder X is currently damaged and can't be watched. We can fix that, but you'll want to close any apps that are watching for changes in this folder first."

Yes, I would definitely want to put up a dialog box before renaming a folder that the user doesn't expect me to rename.
But in that case, I have to question whether I even want to. Given a choice between a silent approach that works (hooking) and displaying a scary confusing dialog box, I'll pick the former any time. If something breaks with a future OS X update, it's not like we cannot update our apps.

@bdkjones I think I'll remove the one-step check-and-repair method, and will stop recommending it. At the very least, it doesn't seem like something that rb-fsevents or another library should do silently, and they don't have the option of showing UI. You'll probably be the only one shipping the repair code, so I would appreciate if you take over maintenance of the repair method — and until you can contribute a working version, I'll remove the repair method from the headers.
FSEventsFixIsBroken and FSEventsFixCopyRootBrokenFolderPath stay, and are enough to identify and report the bug.

Good news: I just ran find-fsevents-bugs myself (stupid to do it this late, right?), and I have quite a few under ~/Dropbox.  (Probably. Still need to verify that it's true.)

...just fixed find-fsevents-bugs so that it doesn't descend into broken folders. This results in a 100x saner result list, containing only 4 folders in my case.
Unfortunately, they all are reported as Found (FSCopyAliasInfo), which means that realpath data isn't corrupted, but rather FSCopyAliasInfo data is. My understanding is that it means that these folders aren't broken from the point of view of FSEvents. :-(
Btw, ~/Pictures/Photos Library.photoslibrary/Database is on the list as well. Could be a semi-reproducible way to get the corruption.
Can everyone in this chat run find-fsevents-bugs as well? Be sure to update to the latest commit. You can download a pre-built x64 version here: http://cl.ly/3K3f0z1a3W0s

@ttilley I'm totally fine about dropping pre-10.9, go for it!

For the sake of completeness, here's find-fsevents-bugs report for my root directory: https://gist.github.com/andreyvit/905f8c8c815cfec63a09. I find it fascinating; in some cases (under /Applications), FSCopyAliasInfo is clearly the only correct data left, while both realpath data and the primary directory listing corrupted.

@andreyvit - my photos db is also broken. realpath() returns Pictures/Photos Library.photoslibrary/Database while my little CFURL dance used in rb-fsevent returns Pictures/Photos Library.photoslibrary/database.
Interestingly enough, both give me correct capitalization for parent folders (i passed in an all-caps path), which I didn't expect realpath to do. That is, until I applied the fseventfix workaround. Then all folders had their case unchanged, including the final folder. The result was PICTURES/PHOTOS LIBRARY.photoslibrary/DATABASE. That's not correct for either result.

ACTUALLY, all those folders are broken with regards to FSEvents! It seems that FSCopyAliasInfo gives the path reported by the kernel.
Additionally, it appears that doing [NSURL fileURLWithPath:path].fileReferenceURL.filePathURL is equivalent to the FSCopyAliasInfo call. And calling [NSURL initFileURLWithPath:].path doesn't change the case of the input string, so it simply does nothing.
So it actually violates our previous assumptions:

We assumed that all Apple APIs short for realpath give the same result. In reality, in my case, enumerating the directory via opendir/readdir, looking at it in Finder, doing ls and calling realpath give the same results, but calling FSCopyAliasInfo, creating a file reference URL and then resolving it, and dragging-and-dropping from Finder into LiveReload give a different result.

We (or at least I) assumed that realpath gives an incorrect result, but in reality either side may be broken. ~/Dropbox/Clients/KEY-TEC looks correct in Finder/realpath, but is reported in lowercase by FSCopyAliasInfo, while /Applications/Adobe Photoshop CC 2014/Adobe Photoshop CC 2014.app/Contents/Frameworks/UpdaterNotifications.framework looks lowercase in finder, but is reported correctly by FSCopyAliasInfo.

So FSEventsFix's current implementation of broken folder check is broken. I don't particularly want to use FSCopyAliasInfo, so I guess I'll use CFURL file reference stuff.

@bdkjones This is new information for your dev ticket, btw. In my case above, realpath() gave me what I see when I ls or open in Finder, whereas the CFURL file reference resolving gave me an incorrect result.

@ttilley Yes, I was operating under several incorrect assumptions, so need to fix the fix now. :-)
But hey, it looks like the broken folders are aplenty, and the Photos.apps is a blessing! It may be the reproduction method we've been looking for, and related to a new Apple's app to boot. @bdkjones, I'm sure Kevin is going to like all of this. Be sure to get him to run find-fsevents-bugs on his machine as well.

To get the icing on the cake, we just need to prove that Time Machine backups for the broken folders are also broken. My time machine disk is in the office, and I'm not; can anyone check?

Here's my action plan:

Change broken folder detection to use Travis's CFURL dance™.
Change the realpath shim to just call the original and apply CFURL dance™ on top.
PROFIT!

Here's an additional idea for rename fix: what if we only changed case when renaming? That has a very low probability of interfering with other apps because the file system is case-insensitive. And we also don't even need to generate a new name; because realpath != FSCopyAliasInfo, we can rename the path into whatever FSCopyAliasInfo returns and back.

Realpath() is the thing that's always broken. The kernel's output will always match the fsref approach.
Sorry; meant to answer that earlier.
Sent from my iPhone

On May 18, 2015, at 13:21, Andrey Tarantsov<EMAIL_ADDRESS>wrote:
ACTUALLY, all those folders are broken with regards to FSEvents! It seems that FSCopyAliasInfo gives the path reported by the kernel.
Additionally, it appears that doing [NSURL fileURLWithPath:path].fileReferenceURL.filePathURL is equivalent to the FSCopyAliasInfo call. And calling [NSURL initFileURLWithPath:].path doesn't change the case of the input string, so it simply does nothing.
So it actually violates our previous assumptions:
We assumed that all Apple APIs short for realpath give the same result. In reality, in my case, enumerating the directory via opendir/readdir, looking at it in Finder, doing ls and calling realpath give the same results, but calling FSCopyAliasInfo, creating a file reference URL and then resolving it, and dragging-and-dropping from Finder into LiveReload give a different result.
We (or at least I) assumed that realpath gives an incorrect result, but in reality either side may be broken. ~/Dropbox/Clients/KEY-TEC looks correct in Finder/realpath, but is reported in lowercase by FSCopyAliasInfo, while /Applications/Adobe Photoshop CC 2014/Adobe Photoshop CC 2014.app/Contents/Frameworks/UpdaterNotifications.framework looks lowercase in finder, but is reported correctly by FSCopyAliasInfo.
So FSEventsFix's current implementation of broken folder check is broken. I don't particularly want to use FSCopyAliasInfo, so I guess I'll use CFURL file reference stuff.
—
Reply to this email directly or view it on GitHub.

Time machine is unlikely to be affected. It reads the database that the fseventsd daemon writes to disk. It doesn't try to handle events as they occur; it just goes back and reads every recorded event from the database when it's time to make a backup.
Sent from my iPhone

On May 18, 2015, at 13:31, Andrey Tarantsov<EMAIL_ADDRESS>wrote:
To get the icing on the cake, we just need to prove that Time Machine backups for the broken folders are also broken. My time machine disk is in the office, and I'm not; can anyone check?
—
Reply to this email directly or view it on GitHub.

Kevin has already found many broken folders on systems at Apple. So we're looking good there!
Sent from my iPhone

On May 18, 2015, at 13:28, Andrey Tarantsov<EMAIL_ADDRESS>wrote:
@ttilley Yes, I was operating under several incorrect assumptions, so need to fix the fix now. :-)
But hey, it looks like the broken folders are aplenty, and the Photos.apps is a blessing! It may be the reproduction method we've been looking for, and related to a new Apple's app to boot. @bdkjones, I'm sure Kevin is going to like all of this. Be sure to get him to run find-fsevents-bugs on his machine as well.
—
Reply to this email directly or view it on GitHub.

No. Renaming with just a case-change does not fix the folder. I attempted that over here and verified the folder was still broken.
Sent from my iPhone

On May 18, 2015, at 13:52, Andrey Tarantsov<EMAIL_ADDRESS>wrote:
Here's my action plan:
Change broken folder detection to use Travis's CFURL dance™.
Change the realpath shim to just call the original and apply CFURL dance™ on top.
PROFIT!
Here's an additional idea for rename fix: what if we only changed case when renaming? That has a very low probability of interfering with other apps because the file system is case-insensitive. And we also don't even need to generate a new name; because realpath != FSCopyAliasInfo, we can rename the path into whatever FSCopyAliasInfo returns and back.
—
Reply to this email directly or view it on GitHub.

The CFURL approach is going to match what the kernel spits out to /dev/FSEvents. In my testing, realpath() is always wrong.
Sent from my iPhone

On May 18, 2015, at 13:25, Travis Tilley<EMAIL_ADDRESS>wrote:
@bdkjones This is new information for your dev ticket, btw. In my case above, realpath() gave me what I see when I ls or open in Finder, whereas the CFURL file reference resolving gave me an incorrect result.
—
Reply to this email directly or view it on GitHub.

@bdkjones I had assumed that Finder would always show the correct case. What you said earlier suggested this. But Finder and realpath agree locally, with CFURL file reference resolving giving a different result. So not all Apple APIs end up returning the correct result then.

Nope. Finder may be incorrect. The kernel accesses the file system the same way CFURL does, so that's what we need. Realpath goes a different route.
In my testing, we just need FSEvents to subscribe to the path that we get with CFURL.
Sent from my iPhone

On May 18, 2015, at 14:07, Travis Tilley<EMAIL_ADDRESS>wrote:
@bdkjones I had assumed that Finder would always show the correct case. What you said earlier suggested this. But Finder and realpath agree locally, with CFURL file reference resolving giving a different result. So not all Apple APIs end up returning the correct result then.
—
Reply to this email directly or view it on GitHub.

That find-bugs tool will report mismatches one way (realpath is wrong) or another (copyAlias is wrong) depending on what case you pass into the tool.
I was able to get it to report a broken folder both ways, simply by varying the case of the input path I fed into the tool.
Sent from my iPhone

On May 18, 2015, at 12:38, Andrey Tarantsov<EMAIL_ADDRESS>wrote:
For the sake of completeness, here's find-fsevents-bugs report for my root directory: https://gist.github.com/andreyvit/905f8c8c815cfec63a09. I find it fascinating; in some cases (under /Applications), FSCopyAliasInfo is clearly the only correct data left, while both realpath data and the primary directory listing corrupted.
—
Reply to this email directly or view it on GitHub.

@bdkjones ‘Correct’ is a relative term here.
Here's my definitions:
Grade-A correctness: the case that has been used to create the folder (assuming that it has never been renamed. And hey, it's extremely unlikely that a mixed-case name could be erroneously produced from a lowercase name, so the mixed-case name is likely always correct.
Grade-B correctness: the case reported by Finder, ls and other enumeration APIs, because that's what I see (and because in most cases this matches the name I originally used, although Adobe guys may beg to differ).
From what I see on my system, grade-A correctness can be on either side, and grade-B correctness is always on realpath's side.
Now, from FSEvents' point of view, FSRef may always be the ‘correct’ one, and realpath the ‘incorrect’ one, but I think we should rather say that kernel uses an incorrect FSRef case when writing to /dev/fsevents, and so FSEvents needs an incorrect FSRef case to work.

That find-bugs tool will report mismatches one way (realpath is wrong) or another (copyAlias is wrong) depending on what case you pass into the tool.

True for the folder you pass in, but the descendant paths use the case returned by readdir, which is made much more clear in the updated output format.
Perhaps from the kernel's point of view one of my prior clients is called ‘key-tec’, but I personally called them ‘KEY-TEC’, and Dropbox has a folder called ‘KEY-TEC’, and readdir returns  ‘KEY-TEC’, and realpath returns  ‘KEY-TEC’, only the NSURL dance™, FSCopyAliasInfo and some unspecified parts of the kernel insist on ‘key-tec’, and I'll be excused for thinking that value highly incorrect. :-)

@bdkjones But I see your point; for FSEvents, we always need the result of the CFURL dance™, which is already on my action plan. I think we don't have an actual disagreement in this discussion. :-)

Yea. It's also worth pointing out that I had just two broken folders to test, so by no means is my testing exhaustive—it's quite possible the folders are breaking in other ways.
However, Kevin implied that the path obtained through NSURL (which is just a wrapper on CFURL, which itself is just a wrapper on CFString) should be "The One True Path" as far as the kernel is concerned. That is, CFURL should always match what the kernel spits out to /dev/fsevents.
Whether or not that output is actually the "Correct" path is above our pay grade.  All we care about is getting FSEvents to use the exact path that the kernel is going to spit out.

@andreyvit playing with fishhook locally, it doesn't seem to rebind within the current scope? if I rebind symbols, then call realpath immediately afterward, it calls the original realpath and not the replacement. did I break it? or does replacement only work for external references?
Anyways, here's a fairly naive implementation of realpath() on top of CFURL.
// naive implementation of realpath on top of CFURL
// NOTE: doesn't quite support the full range of errno results one would
// expect here, in part because some of these functions just return a boolean,
// and in part because i'm not dealing with messy CFErrorRef objects and
// attempting to translate those to sane errno values.
// NOTE: the OSX realpath will return _where_ resolution failed in resolved_name
// if passed in and return NULL. we can't properly support that extension here
// since the resolution happens entirely behind the scenes to us in CFURL.
static char* CFURL_realpath(const char *restrict file_name, char *restrict resolved_name)
{
  char* resolved;
  CFURLRef url1;
  CFURLRef url2;
  CFStringRef path;

  if (file_name == NULL) {
    errno = EINVAL;
    return (NULL);
  }

  #if __DARWIN_UNIX03
  	if (*file_name == 0) {
  		errno = ENOENT;
  		return (NULL);
  	}
  #endif

  // create a buffer to store our result if we weren't passed one
  if (!resolved_name) {
    if ((resolved = malloc(PATH_MAX)) == NULL) return (NULL);
  } else {
    resolved = resolved_name;
  }

  url1 = CFURLCreateFromFileSystemRepresentation(NULL, (const UInt8*)file_name, (CFIndex)strlen(file_name), false);
  if (url1 == NULL) { goto error_return; }

  url2 = CFURLCopyAbsoluteURL(url1);
  CFRelease(url1);
  if (url2 == NULL) { goto error_return; }

  url1 = CFURLCreateFileReferenceURL(NULL, url2, NULL);
  CFRelease(url2);
  if (url1 == NULL) { goto error_return; }

  // if there are multiple hard links to the original path, this may end up
  // being _completely_ different from what was intended
  url2 = CFURLCreateFilePathURL(NULL, url1, NULL);
  CFRelease(url1);
  if (url2 == NULL) { goto error_return; }

  path = CFURLCopyFileSystemPath(url2, kCFURLPOSIXPathStyle);
  CFRelease(url2);
  if (path == NULL) { goto error_return; }

  if (CFStringGetCString(path, resolved, PATH_MAX, kCFStringEncodingUTF8)) {
    CFRelease(path);
  } else {
    // unable to convert? path too long? not a clue.
    goto error_return;
  }

  return resolved;

error_return:
  if (!resolved_name) {
    // we weren't passed in an output buffer and created our own. free it
    int e = errno;
    free(resolved);
    errno = e;
  }
  return (NULL);
}

@ttilley I see we've dragged you into this. :-)
It does rebind in the current scope. Look at FSEventsFix, I've simplified and modified fishhook a bit. Your code is very appreciated. I'd love us to work on the same code base, though — the world at large would benefit from a well-tested fix implementation shared between major players.
As for the errors and edge cases, I propose to call the actual realpath first, and only apply CFURL stuff if it succeeded. If realpath succeeds but CFURL fails, just return what realpath returned.

a new release of rb-fsevent has been pushed to rubygems that includes this workaround (only enabled if the issue is detected). leaving this bug open until things settle down a bit.

I just got an email that Apple Engineering has closed my bug report about realpath() as a duplicate.
I can't put into words how angry I am at Apple Engineering. Either they didn't bother to read the report carefully, or they've known about this issue for a long time and simply don't care about fixing it.
 I'm going to ship the Fishhook fix.  I'm simply going to tear out realpath() before creating a stream and then swap back in Apple's broken implementation right after creating the stream.
And the next Apple Engineer I meet is getting a swift kick in the ass.

@andreyvit Thanks for being more talented and dedicated than the clowns in Cupertino.

@andreyvit @bdkjones @ttilley thanks for having resolved that one! :star2:

@andreyvit did most of the work; I just emailed Apple a lot. You guys were 80% of the way there years ago. It just took a little more info to get a workaround in place.
Sent from my iPhone

On May 21, 2015, at 00:17, Thibaud Guillaume-Gentil<EMAIL_ADDRESS>wrote:
@andreyvit @bdkjones @ttilley thanks for having resolved that one!
—
Reply to this email directly or view it on GitHub.

I'm putting the last touches on FSEventsFix. Meanwhile, we've sat down with @tvasenin and found HFS+ driver source code (which is open-source, after all — the fact that was helpfully discovered by first finding some source code references inside a disassembled Mach kernel, lol). We've spent a fair amount of time digging inside, but didn't come across anything that can possibly covert names into lower case. :-( :-)

FSEventsFix 1.0.0 has just been released using unmodified @ttilley's CFURL_realpath (who's been added to copyrights). I've deleted the repair functions and verified that the fix works on actual broken folders. I've also tried to explain the bug and the workaround in more detail in README.
Thanks to everyone involved; a collaboration like this is a highlight of everything that's good about open-source Cocoa community. It's a lot of pleasure to work with you guys.

:ok_hand:

...and of course that was not the end of the story. I've decided that a radical simplification of the API and internals is in order, so let me present FSEventsFix 2.0. :-)
Important changes:

It's 10x safer ©: realpath is now only replaced for the FSEvents binary, not for every image in the process, and we're no longer using a dyld hook.
Added FSEventsFixIsCorrectPathToWatch function to check the result of FSEventStreamCopyPathsBeingWatched for the ultimate test of whether the fix has worked.
No more logging and debug options; FSEventsFixEnable returns bool and takes an optional char ** argument for an error message if any.
For the orig_realpath called from our CFURL-based reimplementation, we're now using the proper _realpath$DARWIN_EXTSN instead of the strictly-POSIX _realpath.
The functions no longer try to be thread-safe by executing everything on an internal queue (which has been removed), and in fact there's no mutable global state except for the import table of FSEvents. If the caller wants thread safety, they can bring their own dispatch queue or whatever.

We've got a new twist to handle.
The FSEventsFix library works great. However, the one thing it does NOT handle is the case where a path is being watched and events are being reported perfectly normally... and then the "singularity" occurs that breaks the folder. Boom: no more events reported.
In my app, the user can fix this by clicking a "refresh" button in my UI, which scans the project folder and recreates the FSEvents stream. But that's manual work, annoying AND the end result is still the same as before the fix: our apps stop responding to changes in certain folders.
Thus far, the only way I can think to combat this is polling. Every 30 seconds, ask the FSEventsStream what paths it thinks it's watching, then test each one with FSEventsFix and, if any of them are broken, recreate the stream.
Anyone have a better approach?

Using polling to make system events work is kind of a joke, but sadly I can't think of better approach.
Is refreshing fast enough?

@bdkjones While this case is certainly possible, I'd say that when the app stops working, the user will notice and will try restarting it (or clicking a refresh button, if one is available) as the first troubleshooting step. I'm not sure we actually need to do anything.

@bdkjones On the other hand, the set of broken folders under my Dropbox seems to change at least weekly, so I can imagine an unfortunate case where the user has to do manual actions every few days. But I think I'm temporarily out of energy for this bug and would rather not deal with optional complications. :-)

Implementing polling as a solution kinda defeats the purpose a bit. Not
entirely, of course, but if you're on battery power I'd imagine polling of
any kind might make one grumpy. (It'd certainly make me grumpy)
A less invasive measure might be to double check monitored paths after an
event of some kind has already fired. It wouldn't always be effective, but
it's as much as I'd be willing to do since it wouldn't involve additional
wake ups from sleep.
Either way, it'd be somewhat complicated. Save the last event id, stop
monitoring, re-monitor but with a "include since event id " provision...
I'm not putting that much effort in without completely rewriting rb-fsevent
as an actual ruby c extension (to take advantage of the "ignore events
caused by this process" flag).
On Tuesday, May 26, 2015, Andrey Tarantsov<EMAIL_ADDRESS>wrote:

@bdkjones https://github.com/bdkjones On the other hand, the set of
broken folders under my Dropbox seems to change at least weekly, so I can
imagine an unfortunate case where the user has to do manual actions every
few days. But I think I'm temporarily out of energy for this bug and would
rather not deal with optional complications. :-)
—
Reply to this email directly or view it on GitHub
https://github.com/thibaudgg/rb-fsevent/issues/10#issuecomment-105673946
.

--
Sent from Gmail Mobile

I agree with @ttilley; if I were interested in solving the problem, I wouldn't do polling. One option would be to create FSEvents watchers for other potential names (for a normal folder, you can preregister a lowercase name, and for a broken folder, you can register its realpath name). So if the path has N components, you would create 2N watchers instead of one (or you can watch 2N paths with a single watcher, add a trick to FSEventsFix to return different values from each intercepted realpath call).

Sigh. OSX 10.11 might break our workaround without fixing fsevents either.  The policy extends additional protections to components on disk and at run-time, only allowing system binaries to be modified by the system installer and software updates. Code injection and runtime attachments are no longer permitted.

I haven't tested it yet. I'm not even sure I'll still have broken folders after updating to test with (or if i'll regret updating for numerous other reasons).

@ttilley We'll see what it actually means in practice. And maybe Apple will fix the bug in 10.11, who knows. I don't think we should worry about this until after the public beta.

@andreyvit well i took the plunge and previously broken folders have fixed themselves, though probably only temporarily... i noticed installd doing an intensive audit for a good 20 minutes. also, google chrome is now the only google app i have installed that will still run. even the google updater service spews error messages. google drive is a complete no go. instant regret.

You didn't upgrade your main partition, did you? Ouch. OS X previews before DP4 are almost always a disaster. Hope you used a separate partition!

Also: the Apple DTS Engineer got back to me with more information. His speculation was not correct, or at least it was not the whole story.
Apple has known about the issue that causes our problem with FSEvents for years, although this is the first time anyone has shown them that the issue breaks FSEvents. The underlying problem is way down in HFS and, apparently, fixing it involves a monumental effort and there are other side effects.
My interpretation of that is that we aren't likely to see this problem go away until Apple abandons HFS+ as their disk format. So my money is on 10.11 still having the same issue as 10.10 and previous.

@bdkjones Any chance you can share the actual DTS exchange with me privately? (I assume @ttilley would love to read it as well.)

indeed i would.

 I've just reproduced the issue.
Before:
% ~/dev/mini/find-fsevents-bugs/find-fsevents-bugs ~/Dropbox
Result 1:
readdir:         /Users/andreyvit/Dropbox/Budget
realpath:        /Users/andreyvit/Dropbox/Budget
!FSCopyAliasInfo: /Users/andreyvit/Dropbox/budget
Result 2:
readdir:         /Users/andreyvit/Dropbox/Clients/Ascendo
realpath:        /Users/andreyvit/Dropbox/Clients/Ascendo
!FSCopyAliasInfo: /Users/andreyvit/Dropbox/Clients/ascendo
Result 3:
readdir:         /Users/andreyvit/Dropbox/Clients/KEY-TEC
realpath:        /Users/andreyvit/Dropbox/Clients/KEY-TEC
!FSCopyAliasInfo: /Users/andreyvit/Dropbox/Clients/key-tec
Done, 3 result(s).
then (naturally, the folder is actually called Writing):
% ls /Users/andreyvit/Dropbox/writing
After:
% ~/dev/mini/find-fsevents-bugs/find-fsevents-bugs ~/Dropbox
Result 1:
readdir:         /Users/andreyvit/Dropbox/Budget
realpath:        /Users/andreyvit/Dropbox/Budget
!FSCopyAliasInfo: /Users/andreyvit/Dropbox/budget
Result 2:
readdir:         /Users/andreyvit/Dropbox/Clients/Ascendo
realpath:        /Users/andreyvit/Dropbox/Clients/Ascendo
!FSCopyAliasInfo: /Users/andreyvit/Dropbox/Clients/ascendo
Result 3:
readdir:         /Users/andreyvit/Dropbox/Clients/KEY-TEC
realpath:        /Users/andreyvit/Dropbox/Clients/KEY-TEC
!FSCopyAliasInfo: /Users/andreyvit/Dropbox/Clients/key-tec
Result 4:
readdir:         /Users/andreyvit/Dropbox/Writing
realpath:        /Users/andreyvit/Dropbox/Writing
!FSCopyAliasInfo: /Users/andreyvit/Dropbox/writing
Done, 4 result(s).

So for those who's not Bryan or Travis, the bug, as explained by the new favorite pet theory of Apple, is that VFS layer sometimes caches a name provided by a userspace process instead of the correct name, and then uses the cached name to answer some API calls.
Based on the above, ‘sometimes’ is actually ‘pretty much all the time’.
Caching is definitely a trivially reproducible part of the problem, but there's also an unknown ingredient that sometimes causes the wrong name to persist across reboots, presumably by getting written to disk (but maybe it just gets broken on every boot).

@andreyvit Huh? You reproduced it just by calling ls? I was doing that for weeks and did not get that on my end. Can you confirm that this breaks any folder on demand?

@bdkjones I did! But maybe I just lucked out. I cannot do it again myself. :-( There's some other precondition required.

poked a kernel dev staffing a lab at WWDC. lol. https://twitter.com/jauricchio/status/608663324004188160

@ttilley They actually had a dedicated file systems lab there (Tue 9:00). Not sure how to find someone from it, though.

may be fixed in 10.11. maybe. https://twitter.com/jauricchio/status/608713202935844864

I can confirm from personal experience (so far) that this is no longer an issue in El Capitan 10.11. For previous installations the workaround has proven to be effective, even if it's a hell of a workaround...
If this ever proves not to be the case, please reopen this issue. :heart:

additional note: rb-fsevent is likely not breaking the additional security constraints on 10.11 only because we check for the issue before hotpatching realpath, and the check has yet to show a broken path. ¯_(ツ)_/¯

Awesome thanks for the news!
@ttilley in case you were not aware of that Rails PR (listen => rb-fsevent) :heart_eyes:


New composites

This is built upon #14 and should be merged afterwards. The top three commits are this PR
Problem
stake account create + delegate composite transactions require blind signing
Changes

Quells integration test compiler warnings from new Rust version
Adds some missing tests
Adds display support for stake account create and delegate composite transaction

@CriesofCarrots here too please :slightly_smiling_face:

Conflict resolved & commits merged in one go in this PR.


Force IPv4 via the Python API

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.08.02
[debug] Python version 3.7.3 (CPython) - Linux-4.19.0-5-amd64-x86_64-with-debian-10.1
[debug] exe versions: ffmpeg 4.1.4-1, ffprobe 4.1.4-1
[debug] Proxy map: {}
I am trying to use the youtube-dl python API. However according to the options listed here, it is not possible to do so programmatically. A quick glance at the source code didn't help find such option.
There is however the option --force-ipv4 available with the program itself. It should be possible to also do so with python, maybe like this:
yt_dl_opts = {                              'format': 'bestaudio/best'         'forceip':'4'         }
And so this would work as if the option -4 was supplied to youtube-dl. Of course it should also be possible to select IPv6 too.
If the option is indeed available with the API, then it should be listed in the documentation.
Thanks you for your time.

should be listed in the documentation

Have you even read it?
https://github.com/ytdl-org/youtube-dl/blob/824fa51165d92ceee01589bf995ebbf009df328c/youtube_dl/YoutubeDL.py#L264

Okay, but what is the value to set to the parameter, to force youtube-dl to use IPv4 for the download ? Using "4" output the following error:

argument of type 'int' is not iterable

Do you manage reading English at all?

Client-side IP address to bind to.
IP address

How is it relevant to forcing youtube-dl to download only via IPv4 ? What IP am I even supposed to give for that purpose ? Maybe how I worded my message was offensive, but it is not clear at all when reading that this parameter allow any kind of control over which version of IP is used for the download.
It also doesn't work, as youtube-dl still hang, waiting for an IPv6 connection it will never get.

@orion40 Thank you!


[ENHANCEMENT] Spacing

It would be awesome to ADD spacing.
For reference https://github.com/impulse/tailwindcss-figma-kit

Let me try.


_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 51s
master passed

+1 :green_heart:
compile
0m 43s
master passed

+1 :green_heart:
checkstyle
0m 19s
master passed

+1 :green_heart:
spotbugs
0m 33s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 25s
the patch passed

+1 :green_heart:
compile
0m 41s
the patch passed

+1 :green_heart:
javac
0m 41s
the patch passed

-0 :warning:
checkstyle
0m 16s
hbase-it: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
17m 12s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.

+1 :green_heart:
spotbugs
0m 40s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 14s
The patch does not generate ASF License warnings.

35m 56s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2879/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2879

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile

uname
Linux a8bacd1c58b1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / dfefff7e59

Default Java
AdoptOpenJDK-1.8.0_232-b09

checkstyle
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2879/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-it.txt

Max. process+thread count
96 (vs. ulimit of 30000)

modules
C: hbase-it U: hbase-it

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2879/1/console

versions
git=2.17.1 maven=3.6.3 spotbugs=3.1.12

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 32s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 51s
master passed

+1 :green_heart:
compile
0m 43s
master passed

+1 :green_heart:
checkstyle
0m 19s
master passed

+1 :green_heart:
spotbugs
0m 33s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 25s
the patch passed

+1 :green_heart:
compile
0m 41s
the patch passed

+1 :green_heart:
javac
0m 41s
the patch passed

-0 :warning:
checkstyle
0m 16s
hbase-it: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
17m 12s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.

+1 :green_heart:
spotbugs
0m 40s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 14s
The patch does not generate ASF License warnings.

35m 56s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2879/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2879

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile

uname
Linux a8bacd1c58b1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / dfefff7e59

Default Java
AdoptOpenJDK-1.8.0_232-b09

checkstyle
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2879/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-it.txt

Max. process+thread count
96 (vs. ulimit of 30000)

modules
C: hbase-it U: hbase-it

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2879/1/console

versions
git=2.17.1 maven=3.6.3 spotbugs=3.1.12

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.


Halt advancement no values found for linkerd metric request-success-rate probably podinfo.tes t is not receiving traffic: running query failed: no values found

I am following the documentation. When I checked the Automated canary promotion I got this error.
$ kubectl -n test describe canary/podinfo | grep -A 30 Events
Events:
  Type     Reason  Age                   From     Message
  ----     ------  ----                  ----     -------
  Warning  Synced  19m                   flagger  canary deployment podinfo.test not ready: waiting for rollout to finish: 0 of 2 updated replicas are available
  Normal   Synced  5m40s (x2 over 19m)   flagger  New revision detected! Scaling up podinfo.test
  Normal   Synced  5m10s (x2 over 18m)   flagger  Starting canary analysis for podinfo.test
  Normal   Synced  5m9s (x2 over 18m)    flagger  Pre-rollout check acceptance-test passed
  Normal   Synced  5m9s (x2 over 18m)    flagger  Advance podinfo.test canary weight 5
  Warning  Synced  2m39s (x10 over 18m)  flagger  Halt advancement no values found for linkerd metric request-success-rate probably podinfo.test is not receiving traffic: running query failed: no values found
  Warning  Synced  2m10s (x2 over 15m)   flagger  Rolling back podinfo.test failed checks threshold reached 5
  Warning  Synced  2m8s (x2 over 15m)    flagger  Canary failed! Scaling down podinfo.test

I am confused here. I got that the problem might be because "podinfo.test is not receiving traffic". Do I need a valid domain here? It was working before without a domain.

Have you enabled linkerd injection on the test namespace? Can you post here kubectl -n test get pods? IS Flagger installed in the Linkerd namespace can you post here kubectl -n linkerd get pods?

Yes I have. I am just re-doing the same thing one more time. BTW I am doing this in k3d. I will update the output of the command you mentioned if I get the same error again.

Hi,
Here is the output
$ kubectl -n test get pods && kubectl -n linkerd get pods
NAME                                  READY   STATUS    RESTARTS   AGE
flagger-loadtester-566c87c6f8-86pn9   1/1     Running   0          14m
podinfo-primary-689b5f78b9-zpxw9      1/1     Running   0          5m40s
podinfo-primary-689b5f78b9-chbrf      1/1     Running   0          5m40s
NAME                                     READY   STATUS    RESTARTS   AGE
flagger-6448bbb9bb-858b5                 1/1     Running   0          29m
linkerd-identity-599cd7f7fc-4slwg        2/2     Running   0          51m
linkerd-prometheus-7d646d9dcc-h9l54      2/2     Running   0          51m
linkerd-web-69d57f458d-4mmh8             2/2     Running   0          51m
linkerd-proxy-injector-ff55c946d-fnl4h   2/2     Running   0          51m
linkerd-tap-6545854-75x9x                2/2     Running   0          51m
linkerd-sp-validator-77fd4b4fbc-dw4qh    2/2     Running   0          51m
linkerd-controller-cb6584b97-tdg7p       2/2     Running   0          51m
linkerd-destination-7457947f9b-xvz4d     2/2     Running   0          51m
linkerd-grafana-7bcb465757-wj6mx         2/2     Running   1          51m

 kubectl describe ns test
Name:         test
Labels:       <none>
Annotations:  linkerd.io/inject: enabled
Status:       Active

No resource quota.

No LimitRange resource.

Well the test namespace is not injected, there should be 2 containers per pod, maybe linkerd is broken on k3s.

oh, as far as I remembered it worked last time. Not sure what happened now. I will check again.

I tried in GKE, it was not working there too. The pods were not being created. Finally, I increased the resources i.e. CPU and RAM. Then after it worked without any problem. I am not sure what is wrong with k3s. I assume it might be due to the same reason.


Update deploy-storage-spaces-direct.md

The way steps in 1.4 is written, gives the impression that Hyper-V is required for Storage Spaces Direct.
Hyper-V is only required if you are looking to do Virtual Switch Networks with this configuration.

@joemeast : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Initial commit for odh-deployer

The odh-deployer is moving from a subdirectory
in odh-operator-allinone to its own repository

@crobby ptal

just may want to drop appropriate pipeline config in here before merging


Rbexy Fails to Load

Whenever trying to add Rbexy to a project you can no longer actually start up the server or console. The following error shows. This is 2.0.0.rc4. I tried on the following pairings to the same result. Zeitwerk is.... lovely.

Ruby 3.1.4, Rails 7.0.8
Ruby 3.1.4, Rails <IP_ADDRESS>
Ruby 3.2.3, Rails <IP_ADDRESS>
Ruby 3.3.0, Rails <IP_ADDRESS>

The only way around this that I can see is to require: false in the Gemfile and require "rbexy"; require "rbexy/rails/engine" after Bundler loads everything.
Error
/Users/context/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/actionview-<IP_ADDRESS>/lib/action_view/helpers/tag_helper.rb:17:in
`<module:TagHelper>': uninitialized constant ActionView::Helpers::TagHelper::OutputSafetyHelper (NameError)

      include OutputSafetyHelper
              ^^^^^^^^^^^^^^^^^^

We may just need to explicitly require that file from action_view at the top of our file that uses it, like what we do here

@patbenatar Yeah... I spent a good 30 minutes trying a number of different ways of not just doing require 'action_view' but I just continued to get this or other similar errors. In the end the hassle wasn't worth it.


Add different storage formats

I don't consider this high priority, but it might be something people will ask for. Right now the recordings are stored as JSON. YAML and SQLite would be 2 other options.

We should add the VcrStorage abstraction (#22) first before doing this one.


UserManager.FindByName does not load custom properties of UserIdentity

Package versions:
{
    "dependencies": {
        "EntityFramework.Commands": "7.0.0-rc1-final",
        "EntityFramework.SQLite": "7.0.0-rc1-final",
        "EntityFramework.MicrosoftSqlServer": "7.0.0-rc1-final",
        "Microsoft.AspNet.Authentication.Cookies": "1.0.0-rc1-final",
        "Microsoft.AspNet.Authorization": "1.0.0-rc1-final",
        "Microsoft.AspNet.Identity.EntityFramework": "3.0.0-rc1-final",
        "Microsoft.AspNet.StaticFiles": "1.0.0-rc1-final",
        "Microsoft.AspNet.Mvc": "6.0.0-rc1-final",
        "Microsoft.AspNet.Mvc.TagHelpers": "6.0.0-rc1-final",
        "Microsoft.AspNet.Hosting": "1.0.0-rc1-final",
        "Microsoft.AspNet.Server.Kestrel": "1.0.0-rc1-final",
        "Microsoft.AspNet.IISPlatformHandler": "1.0.0-rc1-final",
        "Microsoft.AspNet.Diagnostics": "1.0.0-rc1-final",
        "Microsoft.AspNet.Session": "1.0.0-rc1-final",
        "Microsoft.Extensions.Configuration": "1.0.0-rc1-final",
        "Microsoft.Extensions.Configuration.Json": "1.0.0-rc1-final",
        "Microsoft.Extensions.Configuration.EnvironmentVariables": "1.0.0-rc1-final",
        "Microsoft.Extensions.Configuration.UserSecrets": "1.0.0-rc1-final",
        "Microsoft.Extensions.Logging": "1.0.0-rc1-final",
        "Microsoft.Extensions.Logging.Console": "1.0.0-rc1-final",
        "Microsoft.Extensions.Caching.Memory": "1.0.0-rc1-final"
    },
    "frameworks": {
        "dnx451": {},
        "dnxcore50": {}
    }
}

My Startup services configuration:
public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        var ef = services.AddEntityFramework();
        ef.AddSqlServer().AddDbContext<AuthContext>(options =>
        {
            options.UseSqlServer(dbConnectionString);
        });

        services.AddIdentity<User, IdentityRole>(options =>
        { /* ... */ })
        .AddEntityFrameworkStores<AuthContext>()
        .AddDefaultTokenProviders();
    }
}

My Context:
public class AuthContext : IdentityDbContext<Model.User>
{
    public DbSet<UserProfile> UserProfiles { get; set; }

    protected override void OnModelCreating(ModelBuilder builder)
    {
        base.OnModelCreating(builder);

        builder.Entity<UserProfile>(b =>
        {
            b.HasKey(p => p.UserId);
            b.HasOne(p => p.User)
             .WithOne(u => u.Profile)
             .HasForeignKey<UserProfile>(p => p.UserId);
        });
    }
}

My Model:
public class User : IdentityUser
{
    public UserProfile Profile { get; set; }
}

public class UserProfile
{
    public string FirstName { get; set; }
    public string LastName { get; set; }

    public string UserId { get; set; }
    public User User { get; set; }
}

I am adding a user as follows:
public class Account : Controller
{
    public Account(UserManager<User> userManager, AuthContext authContext)
    {
        UserManager = userManager;
        AuthContext = authContext;
    }

    private UserManager<User> UserManager { get; set; }
    public AuthContext AuthContext { get; set; }

    [HttpPost]
    public async Task<IActionResult> SignUp(SignUpViewModel model)
    {
        if (ModelState.IsValid)
        {
            /* ... */
            var user = new User
            {
                UserName = tempUserName,
                Email = model.Email
            };

            user.Profile = new UserProfile
            {
                FirstName = model.FirstName,
                LastName = model.LastName
            };

            var result = await UserManager.CreateAsync(user, model.Password);

            if (result.Succeeded)
            { /* ... */ }
        }

        return View(model);
    }
}

At some point I need to recover data from a user, and do as follows:
var user = await UserManager.FindByNameAsync(userName));

/* ... */

var firstName = user.Profile.FirstName;

Then I get an error:
An unhandled exception occurred while processing the request.
NullReferenceException: ...

I realized that user.Profile is null.
Out of curiosity, I tried to load the profile alone without even assign to any variable.
var user = await UserManager.FindByNameAsync(userName));

// HACK: ???
AuthContext.UserProfiles.SingleOrDefault(p => p.UserId == user.Id);
/* ... */

var firstName = user.Profile.FirstName;

And suddenly my variable previous user was the value of user.Profile filled out correctly.

I noted that the data are entered correctly in the database.

Because my call of UserManager.FindByName not bring my completely filled object first?

Accessing the child entities through the DbContext is fine.
For the future add a new extensibility point which loads the child entities. (include extra data)

Just to add a bit more detail: as a performance optimization ASP.NET Core Identity currently only loads the entities related to a user or a role as needed to satisfy API calls. I.e. it won't load related entities (not even the built-in ones) eagerly on a method call like such as FindByName() because the find methods are only required to return the root.
At this point issuing queries against the DbContext is the recommended way to load related data. If you want to abstract this from the application code you can extend both the Identity store and manager classes to add methods to retrieve and return your custom related data.

I am experiencing a similar situation where I can see when debugging that sometimes the same method
var userEntity = await _userManager.FindByNameAsync(username);

is returning the user with its navigation properties and sometimes it is not.
Thanks @divega for highlighting that because that might be the reason.
Just to clarify, it's like using lazy loading mode (even when in AspNet Core is not enabled nor supported yet as far as I know) in a way where if the userEntity (the IdentityUser instance) is doing something with the userEntity.Roles it will fetch the Roles, but if there is no explicit interaction with userEntity.Roles in the code then it's value will be null?
This "magic" has had me scratching my head for a while :)
Especially because I am using a javascript front-end that was relying on a JSON object userEntity with all its navigation properties, and sometimes these were null.
I will follow your advice and use my own repositories accessing the dbContext to retrieve the User explicitly including what I need.

This is a huge problem with entity framework as you literally cannot retrieve the virtual properties.

And by the way this is in direct conflict with your implementation of Entity Framework! It's absolutely NOT CLEAR what is happening!!

Why is this issue closed?
I do think it is rather ugly and unsafe (encapsulation etc.) to have to manually include required referenced entities, when they are part of the user identity.
Instead of being able to rely on await userManager.FindByNameAsync(username) we have to wrap that in a separate service and remember that we cannot use UserManager for reading?

I have 4 related entities ( we want to leave aspnetusers table flexible for any future changes in the framework), I'm sure this will be grow more soon, having context to use to include related data always a performance constraint. The aspnet identity framework should support to include the related entities when wanted.


fix(cash-in-success): Handle cash-in-success deeplinks with query parameters

Fixes an issue identified in valora-issue-reports: https://valora-app.slack.com/archives/C018VLA3YAK/p1646073886204589
Tested
Yarn test
How others should test
The unit tests should be sufficient

hey @muckt JFYI: We are skipping this task as in "How others should test" section it is mentioned "Unit test is sufficient"
Cc: @jh2oman
thanks !!