Countdown to the new Images.jl

There have been justifiable questions about "what exactly is happening?" so let me post my TODO list here (including completed items, with an explanation of purpose):

[X] ImagesCore.jl: handles low-level tasks like color separations, raw <--> ufixed conversions, dimension permutations. The new feature is the use of julia's swanky new array technology to provide views for all of these operations, so that we don't sometimes-copy-and-sometimes-return-a-view. Note this also effectively (partly in conjunction with the next package) solves the "Images is really confusing about image orientation") problem, becuase of the dimension permutation views.
[X] ImagesAxes.jl: images with "named" dimensions, most especially temporal dimensions (like movies). Based on AxisArrays.jl, which also allows one to specify a Range (even one with physical units) for each axis, which thus incorporates the pixelspacing. All this information is exposed to the type system (rather than through fields of the properties dict). What that means is that indexing operations become type-predictable: for example, if you slice at a particular spot in time, the new array doesn't have a time axis anymore. In the old Images.jl we had to check for this and set the timedim property to 0 at runtime, and because that's slow we were constantly throwing away the dictionary for performance and therefore losing track of orientation. Now everything should Just Work.
[X] ImagesMeta.jl: for those cases where one still wants to attach metadata to images.
[x] ImagesFiltering.jl: better imfilter implementations. Everything will be type-stable, efficient algorithms will be chosen automatically based on the kernel size, and a platform will be in place for more efficient operations with separable kernels. Also options for performing the computation on the GPU, etc. This package will also likely be the first place to test another experimental direction: the ability (at the user's choice) to dispatch to algorithms in other libraries like OpenCV. (Currently this package is well underway but still needs work, and has been the occasion for me making changes to julia and a half dozen other packages. It won't be runnable by anyone else until julia-0.5-rc3 at the earliest.)
[x] rename the above packages to ImageCore etc.
[x] Get necessary dependencies registered and tagged. In addition to the above-mentioned packages, these include:

[x] AxisArrays
[x] SimpleTraits
[x] CatIndices
[x] FFTViews
[x] ComputationalResources
[x] Ranges (https://github.com/JuliaArrays/Ranges.jl) for Ranges that are safe with Unitful quantities (cf https://github.com/JuliaLang/julia/pull/18777)

[x] Replace the old MapInfo machinery with something based on functions. This will probably go in ImageCore.
[x] Make sure that tests are good, and that as many deprecations are in place as possible.
[ ] Make sure all documentation is in place, and good:

[ ] decide about "local" (detailed docs in each pkg) vs "centralized" (everything at juliaimages.io)
[ ] for each package, make ?pkg give an API summary
[ ] edit the pkg descriptions (https://github.com/timholy/Images.jl/issues/542#issuecomment-254480264)
[ ] add usage examples (community contributions necessary here)

[x] On a branch (let's call it "images-next"), rebase Images.jl on these packages. Will make heavy use of @rexport. Get the old tests passing:

[x] core.jl (src file will be deleted)
[x] map.jl

[x] src/map.jl needs deprecation strategy

[x] overlays.jl
[x] algorithms.jl
[x] exposure.jl
[x] edge.jl
[x] writemime.jl (needed for IJulia)
[x] corner.jl
[x] distances.jl

[ ] Update IO packages:

[x] ImageMagick
[ ] QuartzImageIO
[x] Netpbm
[x] NRRD (will return AxisArray objects for axes with physical units or "meaning" to the axes)

[x] Post an announcement here to say "check out branch images-next" if you want to play with the new Images" and collect comments from enterprising testers for about a week or so.
[ ] Merge to master
[ ] Collect comments for another week or so
[ ] Tag a new release and post an announcement on julia-users

Things that could be done before or after:

[ ] Possibly transfer all the intensity-transformation work to a separate package? Basically my thinking is that largish self-contained units might be a separate module. This would include all of @mronian's great work as well as the src/map.jl functionality. (For now I don't really plan on touching that, so that file could probably be copied verbatim.)
[ ] Resizing, rotation, restriction, etc. This seems like a self-contained chunk that should probably be a standalone package? I am not working on this right now at all, I would welcome contributions (@Evizero you have expressed interested in this area?)
[x] Possibly split Overlay into a separate package

Longer-term future: implement lots of missing functions. Other BSD-licensed projects should be used as important inspiration.

Absolutely! Right now I do the operation chaining for my image augmentation package unnecessary inefficient by creating temporary copies after each operation (input -> cropped image -> rotated image -> distorted image .... etc).
After watching your talk I am really sold on the idea of lazy views, which should - if I am not mistaken - allow for just nesting views instead; which would avoid my predicament.
That said, I am not an image processing or computer vision person, so bare with my newbie mistakes I am bound to make, but I am catching on.
To be a little less vague about my true intentions, I essentially want to create a bridge between julias nice Image functionality and machine learning needs, which diverge in some strange places. For example the "z" or "time" dimension does for us store observations with no intrinsic order/relation. Even stranger: in our usecases the color dimension of an image is actually farther apart in memory than the spatial!. so a 3x20x20x1000 image would often be represented in 20x20x3x1000 (this is just a consequence of how some of our algorithms work)
Anyway, thank you for taking the time to explain this!

With regards to your 20x20x3x1000 array, using ImagesCore you can already do this:
julia> using ImagesCore, Colors

julia> a = rand(UInt8, 20, 20, 3, 1000);

julia> summary(a)
"20×20×3×1000 Array{UInt8,4}"

julia> p = permuteddimsview(a, (3,1,2,4));

julia> c = colorview(RGB, ufixedview(p));

julia> summary(c)
"20×20×1000 ImagesCore.ColorView{ColorTypes.RGB{FixedPointNumbers.UFixed{UInt8,8}},3,MappedArrays.MappedArray{FixedPointNumbers.UFixed{UInt8,8},4,Base.PermutedDimsArrays.PermutedDimsArray{UInt8,4,(3,1,2,4),(2,3,1,4),Array{UInt8,4}},ImagesCore.##15#17{FixedPointNumbers.UFixed{UInt8,8}},ImagesCore.##16#18}}"

julia> col = c[5, 3, 80]
RGB{U8}(0.953,0.914,0.208)

julia> red(col).i
0xf3

julia> a[5,3,1,80]
0xf3

julia> green(col).i
0xe9

julia> a[5,3,2,80]
0xe9

julia> blue(col).i
0x35

julia> a[5,3,3,80]
0x35

julia> a[5,3,2,80] = 0
0

julia> c[5,3,80]
RGB{U8}(0.953,0.0,0.208)

That's the thing about these views...it takes a while to realize how something that seems so "boring" can be so insanely useful. In this case we put a "permuteddimsview" inside of a "ufixedview" inside of a "colorview" and should get darn good performance despite all those shennanigans.
Now the only thing left is to wrap it in an AxisArray (using ImagesAxes.jl) that indicates that the final dimension is time (if that's what you are using it for).

Maybe we could include corners(recently overhauled) and edges(probably needs changes) in ImageFeatures.jl since they come under feature detection.
:+1: for 100% test coverage ! Love to see the bright green coverage badge :smile:
Regarding the documentation, I was thinking of having one juliaimages.github.io which will have the docs for everything. I am still a bit confused how to use Documenter.jl to tap into each repository and get the docstring. I think one of the other GSoCers has done it and I'll take help from him. Does this seem like a good idea or should there be a separate documentation for each one?

Should TestImages.jl be moved to JuliaImages? We could also have some other package (ImageSets.jl ?) which will have the famous datasets in a easily importable format.

Maybe we could have stuff like tutorials etc on the main website and if someone wants to dig into the function reference then they would be pointed to the individual websites.

Before I read any of the above, let me say thanks for filling us in!  I was wondering "what exactly is happening???"

All of the above sounds great, I can't wait to kick the tires.
Here's one thing for my personal wishlist, a pretty printer for image types.  When I see something like:
"20×20×1000 ImagesCore.ColorView{ColorTypes.RGB{FixedPointNumbers.UFixed{UInt8,8}},3,MappedArrays.MappedArray{FixedPointNumbers.UFixed{UInt8,8},4,Base.PermutedDimsArrays.PermutedDimsArray{UInt8,4,(3,1,2,4),(2,3,1,4),Array{UInt8,4}},ImagesCore.##15#17{FixedPointNumbers.UFixed{UInt8,8}},ImagesCore.##16#18}}"

I have to sit there and count curly braces to figure out where I am! A few spaces in there would help as well.

@mronian,

Maybe we could include corners(recently overhauled) and edges(probably needs changes) in ImageFeatures.jl since they come under feature detection.

I was wondering the same thing. I am in the middle of putting the gradient kernels into ImagesFiltering, but the edge-detection stuff doesn't quite fit. One option is to leave them in Images (src/algorithms won't entirely go away), but I tend to agree that ImageFeatures might be a better choice. CC @kmsquire to see if he has a preference.

:+1: for 100% test coverage ! Love to see the bright green coverage badge :smile:

Yes, that's important. The new repos are generally closer to that goal. ImagesAxes is much better than it looks, because some bug (probably in julia) incorrectly misses test coverage of traits-functions. See https://github.com/mauro3/SimpleTraits.jl/pull/6.

Regarding the documentation, I was thinking of having one juliaimages.github.io which will have the docs for everything.

I've had the same thought. I think it will be good to have the docs for Images.jl take much more of a tutorial style, and they can link out to the docs for the individual packages for details. But it would be good to also have one giant function reference. I think that will be fairly straightforward with Documenter, but I haven't tested yet.

Should TestImages.jl be moved to JuliaImages? We could also have some other package (ImageSets.jl ?) which will have the famous datasets in a easily importable format.

Yes, good idea.
@rsrock,

Here's one thing for my personal wishlist, a pretty printer for image types.

Yes, I agree. Ideally that would be something fixed in base julia, but perhaps we could pioneer something here. Jeff has been violently opposed to abbreviating the output of typeof(x), but seems fine with customizing the printing of x itself. So we could conceivably modify Base.summary. We'd have to do it for all array types, or perhaps all array types of Colorants, which is a little scary, because it would affect other packages too that were loaded at the same time as Images.

I think putting edge detection in ImageFeatures would be fine.

When we move an algorithm say A, how do we do so without affecting either of the packages? If I just directly copy, it gives a redefinition of A error. What is the correct way to do this so that A is ported and none of the packages depending on the older A are affected?

I think it will be good to have the docs for Images.jl take much more of a tutorial style

@timholy What do you have in mind here? I wrote a beginner's tutorial to julia's approach to handling image data for a project of mine over here, and I'd happily contribute it if you want it (it is somewhat WIP still). But since it is mainly aimed at ML students (possibly new to Julia), and non-image-experts, I am guessing it might be a little too beginner oriented to be useful to you? (also, feel free to criticize me when I made a mistake or over/understated something)

When we move an algorithm say A, how do we do so without affecting either of the packages?

I think we have to do merges to master/tagging in synchrony. In other words, start a new branch in both packages that make the same change.

I wrote a beginner's tutorial

I haven't read it in detail, but this looks like exactly the kind of thing I was envisioning. Most will probably display images in the docs, too.
I haven't looked in detail, but http://scikit-image.org/docs/dev/user_guide.html seems like a promising model to follow.

where would you want such tutorials to be located at?

I think Images.jl should be the home of all the "major" documentation. (Images.jl will move to JuliaImages at the time of the transition.)

We could have the docs in a separate repository -> https://github.com/JuliaImages/juliaimages.github.io so that juliaimages.github.io becomes the website.

One thing to keep in mind is that if the docs are to contain lots of images, is that they have to be stored somewhere, which can be a hassle depending on where the docs are (since one wants to keep the package size small and the git history clean)

We faced this problem in TestImages.jl for the documentation. Turns out you can use the gh-pages branch to store all website related stuff. :)

All good suggestions. :+1: to putting it in the org.

Is JuliaImages going to be the new home for all this?

We should certainly do a blog post on julialang.org once this is done.

Will the new Images be able to be installed and used with a simple using Images? Because right now it looks like it is broken into a huge number of smaller packages, which could be a nightmare to interface with.

Yes, it will. First, they will all be REQUIREd packages, so they'll be installed automatically. Second, their functions will all be @reexported, so unless you want just a subset of the functionality of Images (and therefore load specific sub-packages) you won't even have to care how the functionality is partitioned at the source-code level. Images will probably still have some "real" code in it, but in many ways it is evolving into a meta-package that just bundles a lot of related functionality together.
This split-up is partly designed to make them more accessible to people who want a subset of the functionality of Images...for example https://github.com/JuliaLang/julia/issues/18384. There is no way to satisfy the desire to "reuse code across different types of applications" unless we split things into relatively focused units of functionality. Reexporting is the key that allows you to have this without sacrificing on other equally-worthy goals, like "provide a comprehensive Images package."

Thanks for the clarification. That sounds promising, I haven't heard of
the @reexport feature yet. Is there a framework for running integration
tests in the meta package?

You can run any tests you want...but it sounds like you have something specific in mind with "framework", what do you have in mind?

So currently most Julia packages have a test/ folder with unit tests
that gets run every time a new commit is pushed to master or a PR is
updated. I guess each subpackage of Images will load all of Images every
time and test all functionality to make sure that there are no
regressions in some other Images subpackage. Now that I said it, it
sounds reasonable.
On Wed, Sep 7, 2016, at 15:06, Tim Holy wrote:

You can run any tests you want...but it sounds like you have something
specific in mind with "framework", what do you have in mind?
— You are receiving this because you commented. Reply to this email
directly, view it on GitHub[1], or mute the thread[2].

Links:

https://github.com/timholy/Images.jl/issues/542#issuecomment-245434736
https://github.com/notifications/unsubscribe-auth/ABlaL5F4fW-XkufoV_Kkc82LQqcRFnpIks5qnzV7gaJpZM4Jjs9f

We could define the runtests.jl in Images.jl in such a way that it provides a few options like running specific tests or the tests of each package.

I see your point. This is a valid issue to be concerned about.
In general I would go the other way: I would have the focused packages test their own functionality, and have the meta-package test everything. You're right, though, that this means it's possible to inadvertently break packages, because if you're updating one of the "focused" packages it won't automatically trigger the Images tests.
One model is to have extra tests that run only on Travis. For example, JLD tests JLDArchives (a repository of "outdated" *.jld files) to ensure backward-compatibility. But this doesn't run as part of Pkg.test("JLD") or include("runtests.jl") for just JLD on your local machine--it's something that is specially triggered in the .travis.yml file.
So using this model, each package could have its own focused tests, but then the .travis.yml script also forces the Images.jl tests to run too, to make sure the PR doesn't break other packages. The one downside is that CI will be slower, but perhaps that's the price we have to pay.

Agreed. I think it would still be nice to add an option to runtests like @mronian said because otherwise it would be difficult to test your changes locally prior to pushing the PR. It shouldn't be that much slower than the running the full test suite of the current Images.jl because it's the same number of tests, but split into smaller containers.

Will Images.jl have any functionality itself apart from being a meta package (Since ImageCore etc are already separated) ?

It's already on @timholy's todo list:

On a branch (let's call it "XYZ"), rebase Images.jl on these packages. Will make heavy use of @rexport.

If it itself doesn't have any functionality, will the tests be similar to those in individual packages? I mean, do we need separate tests again for Images.jl or will the individual package tests suffice?

I mean, do we need separate tests again for Images.jl or will the individual package tests suffice?

It would be nice to have a quick way to run all tests of Images* (ImagesFiltering etc) in one base. I envision something like this: make small tweak in ImagesFiltering and run Pkg.test("Images") and make sure you didn't cause any regressions in the other tightly linked Images* packages.

I hope we find a workaround :sweat_smile: as CI will quickly become very slow as these packages are very heavy (ImageFeatures.jl itself takes around 20-30 min now and with additional functionality like AGAST the tests will only bloat up ) :disappointed:

Here's one thing for my personal wishlist, a pretty printer for image types. When I see something like:

20×20×1000 ImagesCore.ColorView{ColorTypes.RGB{FixedPointNumbers.UFixed{UInt8,8}},3,MappedArrays.MappedArray{FixedPointNumbers.UFixed{UInt8,8},4,Base.PermutedDimsArrays.PermutedDimsArray{UInt8,4,(3,1,2,4),(2,3,1,4),Array{UInt8,4}},ImagesCore.##15#17{FixedPointNumbers.UFixed{UInt8,8}},ImagesCore.##16#18}}"

> I have to sit there and count curly braces to figure out where I am! A few spaces in there would help as well.

I would go further than @rsrock and say that this is basically unreadable as is. I can only imagine what error messages look like with that type printed everywhere.

I also find these extremely difficult to read, and would love to change this. The key question is how to fix it.
Currently we rely on the generic Base.summary for printing that kind of type information. The impenetrably-nested braces comes from that typeof(a). Now, we're advised against changing how types are printed, which is bad news for addressing this:

I can only imagine what error messages look like with that type printed everywhere.

But it's legitimate to provide a specialized show or summary function, since that's used for value-printing. So we can at least fix some aspect of this.
I guess in my mind the key question is, how to specialize it? I see three options:

override the Base definition for every AbstractArray
specialize for arrays that have a Colorant eltype
come up with a list of containers (at least ColorView, ChannelView, MappedArrays, ImageMeta) that deserve specializations of summary (if current plans hold, there will be no "one container to rule them all" when it comes to images).

Pros and cons:

Pro: could perhaps really fix the issue. Current thought would be to compute the nesting depth and then do something immensely clever (just have to figure out what that is :wink:) depending on the answer. Con: this will cause a warning to be printed every time you say using Images. More subversively, it would affect the behavior of any other package that the user had co-loaded with Images---it could mess up printing for their (array) types. That's considered bad form.
Pro: most images will presumably be loaded as Colorant types. In particular, IJulia will only show arrays that have Colorant eltypes as an image; the rest will fall back to text output. Con: leaves out ChannelView and any grayscale images that don't explicitly use Gray.
Pro: probably the cleanest in terms of encapsulating for types we care about. Con: will miss important types like SubArray and PermutedDimsArray, which are also very useful containers for images.

To be honest, my favorite solution is 1. I think we could take something for a test drive here, knowing that we're committing a faux pas, with the excuse that if we all like it we'll submit it as a PR for Base.
This is the moment where somebody reading this chimes in with the brilliant suggestion that gets the best of all possible worlds.

While on the topic of how things are printed, perhaps it's worth pointing out https://github.com/JeffBezanson/FixedPointNumbers.jl/pull/51#issuecomment-247507435, which is a proposal to change how the fixed-point types are printed. A major motivation for this proposal is to make text-display of pixels values (or whole images) more compact.

I've periodically been updating the list above. It feels like it's getting close now.
There will be some delay, unfortunately. I did a fair amount of work updating ImageMagick/writemime, but then my laptop died (motherboard problem), and my old laptop wouldn't read the new ones' hard drive. So some items will likely wait until I get my laptop back. So far I haven't found myself short of other tasks in that list to keep me occupied :persevere: .

Locally I have a candidate solution to the printing-of-types problem mentioned above as being high on @rsrock's wish list, https://github.com/timholy/Images.jl/issues/542#issuecomment-239647238. I figured out a way to do it that will only affect types defined in the images-related packages (I think).
The idea is based on the observation that sometimes the type printing is more confusing than the sequence of calls you'd use to create the object.  Here is a demo:
julia> a = rand(UInt8, 5,5,3)
5×5×3 Array{UInt8,3}:
[:, :, 1] =
 0x63  0x44  0xe1  0x1e  0xba
 0x08  0x07  0x3f  0x4c  0x1f
 0x10  0xaf  0xf7  0xe7  0x87
 0x3c  0x37  0x49  0x6e  0xd4
 0xd9  0x05  0xf2  0x5b  0x0f

[:, :, 2] =
 0x8a  0xeb  0x34  0x6b  0x59
 0x0f  0xd6  0x71  0x8d  0x74
 0xe6  0xfa  0xb1  0xba  0xeb
 0x9b  0xb6  0xa4  0xe2  0x24
 0xe2  0x4f  0x62  0xc5  0x70

[:, :, 3] =
 0xfd  0x31  0x24  0x49  0xf3
 0x9a  0x2d  0x80  0x2c  0x10
 0xeb  0x0e  0x03  0x68  0x68
 0xbd  0x0e  0x39  0xf0  0x0a
 0xc1  0x0c  0x66  0x49  0x78

julia> p = permuteddimsview(a, (3,1,2))
3×5×5 permuteddimsview(::Array{UInt8,3}, (3,1,2)) with element type UInt8:
[:, :, 1] =
 0x63  0x08  0x10  0x3c  0xd9
 0x8a  0x0f  0xe6  0x9b  0xe2
 0xfd  0x9a  0xeb  0xbd  0xc1

[:, :, 2] =
 0x44  0x07  0xaf  0x37  0x05
 0xeb  0xd6  0xfa  0xb6  0x4f
 0x31  0x2d  0x0e  0x0e  0x0c

[:, :, 3] =
 0xe1  0x3f  0xf7  0x49  0xf2
 0x34  0x71  0xb1  0xa4  0x62
 0x24  0x80  0x03  0x39  0x66

[:, :, 4] =
 0x1e  0x4c  0xe7  0x6e  0x5b
 0x6b  0x8d  0xba  0xe2  0xc5
 0x49  0x2c  0x68  0xf0  0x49

[:, :, 5] =
 0xba  0x1f  0x87  0xd4  0x0f
 0x59  0x74  0xeb  0x24  0x70
 0xf3  0x10  0x68  0x0a  0x78

julia> u = ufixedview(p)
3×5×5 ufixedview(N0f8, permuteddimsview(::Array{UInt8,3}, (3,1,2))) with element type FixedPointNumbers.UFixed{UInt8,8}:
[:, :, 1] =
 0.388N0f8  0.031N0f8  0.063N0f8  0.235N0f8  0.851N0f8
 0.541N0f8  0.059N0f8  0.902N0f8  0.608N0f8  0.886N0f8
 0.992N0f8  0.604N0f8  0.922N0f8  0.741N0f8  0.757N0f8

[:, :, 2] =
 0.267N0f8  0.027N0f8  0.686N0f8  0.216N0f8  0.02N0f8
 0.922N0f8  0.839N0f8  0.98N0f8   0.714N0f8  0.31N0f8
 0.192N0f8  0.176N0f8  0.055N0f8  0.055N0f8  0.047N0f8

[:, :, 3] =
 0.882N0f8  0.247N0f8  0.969N0f8  0.286N0f8  0.949N0f8
 0.204N0f8  0.443N0f8  0.694N0f8  0.643N0f8  0.384N0f8
 0.141N0f8  0.502N0f8  0.012N0f8  0.224N0f8  0.4N0f8

[:, :, 4] =
 0.118N0f8  0.298N0f8  0.906N0f8  0.431N0f8  0.357N0f8
 0.42N0f8   0.553N0f8  0.729N0f8  0.886N0f8  0.773N0f8
 0.286N0f8  0.173N0f8  0.408N0f8  0.941N0f8  0.286N0f8

[:, :, 5] =
 0.729N0f8  0.122N0f8  0.529N0f8  0.831N0f8  0.059N0f8
 0.349N0f8  0.455N0f8  0.922N0f8  0.141N0f8  0.439N0f8
 0.953N0f8  0.063N0f8  0.408N0f8  0.039N0f8  0.471N0f8

julia> c = colorview(RGB, u)
5×5 ColorView{RGB}(ufixedview(N0f8, permuteddimsview(::Array{UInt8,3}, (3,1,2)))) with element type ColorTypes.RGB{FixedPointNumbers.UFixed{UInt8,8}}:
 RGB{U8}(0.388,0.541,0.992)  RGB{U8}(0.267,0.922,0.192)  RGB{U8}(0.882,0.204,0.141)  RGB{U8}(0.118,0.42,0.286)   RGB{U8}(0.729,0.349,0.953)
 RGB{U8}(0.031,0.059,0.604)  RGB{U8}(0.027,0.839,0.176)  RGB{U8}(0.247,0.443,0.502)  RGB{U8}(0.298,0.553,0.173)  RGB{U8}(0.122,0.455,0.063)
 RGB{U8}(0.063,0.902,0.922)  RGB{U8}(0.686,0.98,0.055)   RGB{U8}(0.969,0.694,0.012)  RGB{U8}(0.906,0.729,0.408)  RGB{U8}(0.529,0.922,0.408)
 RGB{U8}(0.235,0.608,0.741)  RGB{U8}(0.216,0.714,0.055)  RGB{U8}(0.286,0.643,0.224)  RGB{U8}(0.431,0.886,0.941)  RGB{U8}(0.831,0.141,0.039)
 RGB{U8}(0.851,0.886,0.757)  RGB{U8}(0.02,0.31,0.047)    RGB{U8}(0.949,0.384,0.4)    RGB{U8}(0.357,0.773,0.286)  RGB{U8}(0.059,0.439,0.471)

So this still shows as nested, but I think
ColorView{RGB}(ufixedview(N0f8, permuteddimsview(::Array{UInt8,3}, (3,1,2)))) with element type ColorTypes.RGB{FixedPointNumbers.UFixed{UInt8,8}}

is easier to read than
julia> typeof(c)
ImageCore.ColorView{ColorTypes.RGB{FixedPointNumbers.UFixed{UInt8,8}},2,MappedArrays.MappedArray{FixedPointNumbers.UFixed{UInt8,8},3,Base.PermutedDimsArrays.PermutedDimsArray{UInt8,3,(3,1,2),(2,3,1),Array{UInt8,3}},ImageCore.##29#30{FixedPointNumbers.UFixed{UInt8,8}},Base.#reinterpret}}

Thoughts? Is this a good direction? If so, is it sufficient, or are there adjustments needed?

I like it. certainly easier than the type. I am not quite sure I fully follow the pattern yet. For example why write ColorView{RGB}, but unfixedview(N0f8, when both originated from a similar looking method call

To explain: it's because ColorView{C}(A) always creates a ColorView, whereas colorview(C, A) might create a ColorView. It just returns a plain array when A is amenable to reinterpret. This strategy is an attempt to reduce the amount of nesting in the types themselves.
I agree that's a little confusing, though, so I'm happy to entertain alternatives.

Isn't that kind of dispatch true for ufixedview as well https://github.com/JuliaImages/ImageCore.jl/blob/master/src/ImageCore.jl#L90? I guess I am just unclear when a function-style printing is chosen (such as for ufixedview) vs when a type-style printing is chosen (such as for colorview)
Regardless of my confusions (which are numerous no matter what anyways), I do like the change. 👍

I would call this "productive confusion," in the sense that this is 100% valid. So no apologies needed; to the contrary, thanks for raising these concerns.
In the case of ufixedview, I'm not certain I see a way to resolve the issue without introducing problems elsewhere:

Let's say you always want to show it as a type. But there is no type directly corresponding to ufixedview: both ufixedview and rawview create a MappedArray (when reinterpret wouldn't work). We could print it as MappedArray((x->reinterpret(Nf08, x), reinterpret), <inner array type>) but I worry that's considerably harder to read than ufixedview(Nf08, <inner array type>). (But this is nevertheless a viable option.) Alternatively, we could introduce UFixedView and RawView types, but that's an awful lot of duplication of functionality for no actual benefit when MappedArrays already exists.

Let's say you always want to show it as a function call. Then the problem is that you could create a ColorView like this: ColorView{RGB}(rand(3,5,5)). Then in this scenario that would print as colorview(RGB, ::Array{Float32,3}), but the problem is that colorview(RGB, rand(3,5,5)) would return an Array and not a ColorView, which is the actual type of the object that's being printed.

As there are alternative IO-solutions available now, is it still mandatory to install ImageMagick?
Im asking as IM is blocked at my company due to serious security flaws.
It would be nice to have a query at installation to select one of the IO-packages.

There is a query on OSX because of QuartzImageIO, but there isn't a comprehensive alternative on any other platform. Netpbm is probably closest.
This is a great area for motivated folks to contribute! If you have another library in mind, you can see from ImageMagick.jl that it's not an outrageously large undertaking to write a wrapper. (Not a trivial amount of work, either.) Or more ambitiously one could write a parser in pure julia (like the Netpbm and NRRD parsers). But that would be one format at a time, and a much larger undertaking.

http://www.graphicsmagick.org/index.html

See also previous discussion in https://github.com/timholy/Images.jl/issues/24

Actually, GraphicsMagick is known to have inherited some of the security flaws from ImageMagick. For that reason, it is also blocked at our company.
I will see if i can write a little wrapper for some basic IO library.

Near the end of a long journey
I'm pleased to announce that the "new Images" is ready for testing. It's not that absolutely everything is done (see below), but I think it would be appropriate for people to start playing with this. For a list of some of the main changes, please see https://github.com/timholy/Images.jl/blob/images-next/NEWS.md.
Documentation
I've put a lot of work into documentation so far, although that effort isn't completely done. The new documentation will be hosted mostly at http://juliaimages.github.io/latest/, which is intended to be "organizational-level" documentation for all of JuliaImages. Before working on that documentation, I'd also written documentation for ImageCore, ImageFiltering, ImageAxes, and ImageMetadata---I'm wondering if this should just be moved to JuliaImages?
I am hoping this will become very polished documentation, competitive with other image processing suites like skimage, OpenCV, etc., but it is not there yet. What is there is supposed to be good, so do read it with a critical eye and please fix/report problems. In my mind the "tutorial" sections are mostly complete, but it's missing important components (like API documentation) and most especially a bunch of examples (e.g., see http://scikit-image.org/docs/dev/auto_examples/index.html). I really hope that users will contribute to the documentation especially by providing examples.
Trying it out
Pkg.update()

Then download this gist (or just copy/paste the file), and say
include("imagebranch.jl")
imagebranch("fixed-renaming")

If you have dirty packages or are on branches, this will almost certainly fail. I have verified that it does work with a fresh pkg folder, e.g.,
tim@cannon:~/src/julia-0.5$ JULIA_PKGDIR=/tmp/tim/pkgs2 ./julia -q
julia> Pkg.init()
INFO: Initializing package repository /tmp/tim/pkgs2/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl

julia> Pkg.add("PkgDev")
<some output>

julia> Pkg.add("Images")
<some output>

julia> Pkg.add("ImageMagick")  # plus any others you might need
<some output>

julia> include("/tmp/tim/imagebranch.jl")
INFO: Precompiling module PkgDev.
imagebranch (generic function with 1 method)

julia> imagebranch("fixed-renaming")
<lots of output>

If you find yourself unhappy and want to go back to the original, use
imagebranch("release")

and if you want to try the "intermediate" set (one that doesn't include https://github.com/JuliaMath/FixedPointNumbers.jl/pull/51), use
imagebranch("images-next")

Stuff I'd like feedback on

do you like (or hate) the new FixedPointNumber names?
any bolded comments in the JuliaImages documentation
are any deprecations missing? Some things (e.g., setting the "spatialorder" property) have to be errors, but does the error message give clear instructions about how to fix things?
any problems (or things you especially like) about the API changes

Please report problems as separate issues (don't just add to this one), but mark it clearly as to which "overarching branch" you're using ("release", "images-next", or "fixed-renaming").
Tips while running old scripts with new code
First, I urge you to use a branch when making any changes to your own internal code; who knows how long it will be before this merges to master, so you might want to be able to easily go back and forth.
Second, don't forget that you can launch julia as julia --depwarn=no and suppress a whole sea of deprecation warnings. That said, the point of the deprecation warnings is to teach you how to migrate your code, so you'll want to do at least enough of this to start discovering problem points, things you like or dislike, etc.

Thanks! I was able to update using your gist, although I did have to visit all affected package folders and do git fetch --all (which may be an overkill, I know). Otherwise the command imagebranch("fixed-renaming") did not infer which corresponding remote branch to track and threw an error for each package I did not fetch
INFO: Checking out Colors fixed-renaming...
ERROR: GitError(Code:ERROR, Class:Merge, There is no tracking information for the current branch.)
[...]

Thanks! I updated the gist, now it should fetch automatically.

One thing I forgot to ask: do folks think that all the documentation should be moved to http://juliaimages.github.io/latest? (Meaning, should ImageCore, ImageFiltering, etc. continue to maintain their own documentation?) Either way, http://juliaimages.github.io/latest/conversions_views.html#Conversions-vs.-views-1 is largely redundant with https://juliaimages.github.io/ImageCore.jl/latest/views.html#Views-1. Any thoughts on this would be appreciated. Originally I intended to just link the page from ImageCore in to juliaimages.io, but there are technical difficulties with that, and once I thought about just copying I came to the conclusion that some of the groundwork laid on earlier pages led to me wanting to explain things slightly differently. So if you think we should have only one of them, concrete changes for suggestions about which to keep or how to merge them would be appreciated.

Personally I think the following would make sense:

JuliaImages: Contains a high-level documentation of all the functionality without going into details (such as arithmetic overflow). I think this would also be a good location for this example tutorials that you envision. It should contain only that information that a user needs to work with his/her image data without the "burden" of technical details. I think the maturity of a library can often be inferred by how little a user has to learn in order to work with it, even if it means that the user remains oblivious to how cleverly some issues were addressed under the hood.

The Packages: Detailed information and developer information for the provided functionality, such as colorview vs ColorView, the note on arithmetic overflow etc. All the information that makes the interested and potential contributors understand how the packages work.

The principle seems sound. But sometimes the dividing line between the two is unclear, perhaps especially to me (given that I spent so much time thinking about the technical details). To pick just one example: is http://juliaimages.github.io/latest/conversions_views.html#Sharing-memory:-an-introduction-to-views-1, pointing out the differences between a copy and and view, a "detail" or "necessary"?
My favored approach would be for others to cut/paste the docs and move them appropriately, since others have a better "outsider" perspective, but I'm happy to do it particularly if folks give me sufficiently detailed instructions.

My vote (not that I have one to cast) would be to have to docs all in one
place and dedicate a page to each package to explain how it fits into the
ecosystem and any peculiars one would need to know if they were to use it
in isolation. Then you could link directly to that page from the respective
repo.
On Tuesday, October 18, 2016, Tim Holy<EMAIL_ADDRESS>wrote:

The principle seems sound. But sometimes the dividing line between the two
is unclear, perhaps especially to me (given that I spent so much time
thinking about the technical details). To pick just one example: is
http://juliaimages.github.io/latest/conversions_views.html#
Sharing-memory:-an-introduction-to-views-1, pointing out the differences
between a copy and and view, a "detail" or "necessary"?
My favored approach would be for others to cut/paste the docs and move
them appropriately, since others have a better "outsider" perspective, but
I'm happy to do it particularly if folks give me sufficiently detailed
instructions.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/timholy/Images.jl/issues/542#issuecomment-254451693,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA492jT0L3eXLo4jNzlND2djWtXJGlM7ks5q1I3fgaJpZM4Jjs9f
.

You definitely have a vote to cast!

:)
Also... I could throw together a meta spec (for MetaPkg) for the transition
if you want, to avoid making users copy/paste that long gist to setup. I
want to help, but weary of pushing it on anyone considering the tension
over package management.
On Tuesday, October 18, 2016, Tim Holy<EMAIL_ADDRESS>wrote:

You definitely have a vote to cast!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/timholy/Images.jl/issues/542#issuecomment-254465529,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA492k1vK_YdhOxMga_ls3DHXRsHLT8Mks5q1JvIgaJpZM4Jjs9f
.

I haven't played with MetaPkg yet. Very cool, and seems like a systematic solution to precisely the problem my gist was designed to address. I'd be thrilled!

Exactly. My needs were for managing the many packages of JuliaPlots,
JuliaML, and the GLVisualize universe, many of them not registered.
On Tuesday, October 18, 2016, Tim Holy<EMAIL_ADDRESS>wrote:

I haven't played with MetaPkg yet. Very cool, and seems like a systematic
solution to precisely the problem my gist was designed to address. I'd be
thrilled!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/timholy/Images.jl/issues/542#issuecomment-254469409,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA492hAJTKYDUvZns_P485Lwk-vL5-D6ks5q1KAlgaJpZM4Jjs9f
.

would be nice to have more info in each package's README.  i code in a terminal and it's nice when packages contain at least a list of exports therein so that one can just say ?JLD, for example, for a quick jog of the memory.
it's also very important to make the "description" (eg for ImagesCore it is currently "Julia types for representing images") informative.  because, this and the name of the package are all that's listed at http://pkg.julialang.org.  in such a constrained space, i think the word "Julia" is not useful.

I agree with @tbreloff here. One centralized place for documentation would be better. That way you don't have jump back and forth between different sets of docs. I would imagine that the less users are exposed to separate Image packages the better.

I took the liberty of adding a JuliaImages meta spec in MetaPkg. I tested it on my mac laptop, and everything seemed to work, but you should verify yourself (make sure all your work is pushed up because purge will do Pkg.rm and also delete cache/lib/trash files that are left around).
Instead of using that gist linked to above, you should be able to update with the commands:
using MetaPkg
MetaPkg.purge("JuliaImages")
meta_add("JuliaImages")

Then you should be all set up on the "release" path of the gist.
If you want to do the "fixed-renaming" branch:
meta_checkout("JuliaImages", "fixed-renaming")

and if you want to go back to the "release" afterwards:
meta_free("JuliaImages")

I didn't do the "images-next" path yet.. figured I'd get your input first.  I made a gist with the full expected output from the REPL.

This looks awesome.
I have a bunch of not-pushed branches in quite a few of those packages, and I'm hopping on a plane this morning, so it might take me a bit of time to prepare the groundwork for testing this. But I really like how this looks as a way of managing an interwoven set of packages!

I am a little late on this show, but would like to ask where all the metadata handling has gone (pixelspacing and so on) This was a very prominent feature in the stable branch of Images and in our internal codebase almost anything is an image object for exactly that reason. We also attach quite some additional metadata in image objects.

Tim, would you mind opening an issue over on QuartzImageIO with some guidance on what needs to be updated?  I haven't had a chance to try this out yet, but I hope to do so soon.  Thanks.

@tknopp, it's now called ImageMeta. See ImageMetadata.jl. However, things like pixelspacing are no longer encoded in the dictionary; you should use AxisArrays (with a bit of extra functionality provided by ImageAxes). All these packages are loaded when you say using Images, however. I'm especially interested in transitions like the one you're describing, so if you have more than a little difficulty with the transition please let me know: the whole point of this beta period is to find out the pain points, and I can help smooth the pain.
@rsrock, will do.

Ok, thanks. Not sure when we get to this but we will try. I ping my colleague @hofmannmartin to be aware of the Images transition. Since we also set the pixelspacing using im[:pixelspacing] it seems that we will need some transitioning.

The safer thing has always been to say pixelspacing(im), but perhaps this wasn't emphasized strongly enough.

@timholy I see you that you might move the Overlay stuff out. Would that be the recommended way of handling multichannel images or should something else be used instead?

That certainly has been emphasized. There are these annoying people not carefully reading the documentation... Sorry my workflow is sometimes a little exploratory.

@tlnagy, that's in fact already done, see http://juliaimages.github.io/latest/conversions_views.html#Using-colorview-to-make-color-overlays-1 and redundantly described in https://juliaimages.github.io/ImageCore.jl/latest/views.html#View-types-defined-in-ImageCore-1. That allows you to easily fill in channels for red, green, and blue (and you can do magenta/green with colorview(RGB, m, g, m)). If that's not flexible enough, I suspect that better than the old approach would be to define a custom color type, e.g., the 2-channel color
immutable GFPtdTomato{T} <: Color{T, 2}
    gfp::T
    tdTomato::T
end

along with a function that can convert such "colors" to RGB.

@tknopp, now I also see you said "set the pixelspacing". Yes, it might be a little more work, sorry---if you find it too annoying then let's talk about a convenience function, e.g.
im = imageaxes(buf, pixelspacing=(1mm, 1mm))

or something.
As consolation for any pain, among the benefits of having it be more "structured" now is that it won't be lost when you index like im[5:100, 40:80]; formerly that lost both the pixel spacing and the orientation, which was just evil.

Thanks! The ImageCore documentation is quite thorough. It does mention StackedView on several occasions, but I can't seem to find any docs on that.

The balance of advice above was to centralize the docs at JuliaImages, so the final product may look more like what's there than the ImageCore docs.
StackedView is kinda internal; if you use colorview(RGB, r, g, b) then you don't need to touch it. Briefly, StackedView takes m×n arrays a, b, c and creates a 3×m×n view of them.

Is there some documentation how to combine ImageMeta objects with AxisArrays? I would like to start with an object that is a close to the current Image object and start from there.

Sure, they're completely orthogonal: just use an AxisArray for the array argument of ImageMeta. There's already some special-purpose code in place to look for this case, e.g., https://github.com/JuliaImages/ImageMetadata.jl/blob/b8e5c162b71babe457b56669e4cb8530964407f3/src/ImageMetadata.jl#L71-L73.

So, a typical usage of Images.jl was
https://github.com/simonster/NIfTI.jl/blob/2f217aac10d0e48e77eeb437e82a11c203dc2255/examples/mriview.jl#L9
What is your advice to handle this in the future. I will try playing around with this myself, but maybe it would be good to have this particular case discussed in the documentation somewhere.
And whats about the other image loader, are the already ported to the new Image infrastructure?

In case you haven't seen it, https://juliaimages.github.io/ImageAxes.jl/latest/ might help.
Simple (ignoring pixel spacing): ImageMeta(AxisArray(colorview(Gray, ni.raw), :x, :y, :z, :time))
There isn't quite as simple an approach for handling the pixel spacing (this is exactly the kind of feedback I'm looking for), but it is more flexible:
ImageMeta(AxisArray(colorview(Gray, ni.raw), Axis{:x}(rngx), Axis{:y}(rngy), Axis{:z}(rngz), Axis{:time}(rngt)))

where rng[xyzt] is a range that describes the positions of the different coordinates, e.g., 0:2:284 if the pixels are 2mm and go from 0mm to 284mm. (If you want to attach real units, currently Julia's ranges are kinda problematic, but see Unitful.jl, Ranges.jl and https://github.com/JuliaLang/julia/pull/18777.)
Do you need an easier way to specify pixel spacing? The default AxisArray behavior is very flexible (it lets you specify the origin) but maybe somewhat verbose.

Note you don't really need the ImageMeta here, everything is encoded by the AxisArray...one of the main points of the redesign was to encode everything having to do with orientation in the type system so that A[:, 5, :] can preserve the names of the correct axes without hammering performance.

Hm, this is of course flexible but also pretty complicated.
The pixelspacing is one of the most important things since in our (tomographic) imaging method we have anisotropic voxels. Furthermore I use two custom fields "offset" and "rotation" within the image object that allow to define the precise coordinate system in space. With this I can take two Image objects measured with two different tomographs and overlay them doing the appropriate registration/regridding/image fusion. The rotation field is necessary since MRI scans can be arbitrarily angulated.

So if I see it correctly I will have to but the offset into the ranges, but the rotation will remain an ImageMeta field.

Well, offset is exactly what you can encode in the AxisArray ranges. Might want a simpler approach, how about
A = AxisImage(data, (:x, :y, :z, :time), (dx, dy, dz, dt), [(x0, y0, z0, t0)])

?
As for rotation goes...that would either have to be in the metadata, or possibly you could use a TransformedArray from AffineTransforms.jl. That package will be deprecated by CoordinateTransforms.jl, however, but I can't remember the current status.

Yes, such a syntax would be definitely a nice convenience constructor, so thumbs up.

Oh, and to answer an earlier question, yes, all the IO packages have been updated too if you check things out as described in https://github.com/timholy/Images.jl/issues/542#issuecomment-254059092. That gist has been updated several times, so you may want a fresh copy.

Ok thanks.
Its quite interesting to see how this restructuring will change the workflow. Before, all parameters such as pixelspacing where just metadata and I extracted the underlaying array before giving it to a hot loop. Maybe this won't be necessary anymore if the type stability is preserved.

Yes, the idea is you just pass an AbstractArray to whatever you do. That way you never have to make choices between performance and "self-documentation." It should be just as fast as an Array.
Could you open an issue on ImageAxes with https://github.com/timholy/Images.jl/issues/542#issuecomment-270742133, so I don't forget?

Added to AxisArrays (need to be running master).

Closed by #577


Code on GitHub

It seems that almost every chapter in the repository have missing code, is there any plans to update the repo with some notebooks with all the books's code?
Thank you, I'm loving the book.

It seems that almost every chapter in the repository have missing code, is there any plans to update the repo with some notebooks with all the books's code?
Thank you, I'm loving the book.

Hellos, first of all thanks for the question. This repo isn't meant to include all the code from the book, some examples per early chapters; then data and code for the chapters that include mini-projects. I believe small code segments in early chapters are copy-pasteable :relaxed:
Glad you enjoyed the book, currently I'm making giveaway for a Generative AI book, would you be interested?

Is the one on your linkedin? Already commented on it.
Thank you very much

Is the one on your linkedin? Already commented on it.
Thank you very much

yes :wink:


Admin Interface

Hi all,
I did a build of the latest influxdb git. Works fine sofar but I cannot access the admin interface. In the influxdb.conf the admin port is set 8083 but actually it is only listening to 8086. Do I miss something?

:+1:

EIther your config is wrong, or you are not running the config you think. Please follow up on the mailing list if you can't make any progress, and attach your config.

Were you building master or the 0.8.8 branch? If you're talking about master then this is a known issue. We haven't put the admin interface back into the build. See #1439 to track the progress on that.
The config file from previous versions of Influx won't work with 0.9.0. There is a new config file in etc/config.sample.toml

@pauldix -- thanks.
@Sineos -- I had assumed you were building a release version from the source, 0.8.8 for example, if you are building master the Admin UI is still pending. As @pauldix, master is still in development. If you wish the build the latest released code, check out this tag: https://github.com/influxdb/influxdb/tree/v0.8.8


Translatable/Extendable dot navigation instructions

There does not appear to be a way to extend the sr-only text for dot navigation.
Current markup generated the string "Go to slide' is embedded:

<button type="button" aria-current="true">

      Go to slide 1
    </button>

    <button type="button">

      Go to slide 2
    </button>

There should be something similar to the previous next buttons where a method is provided to override this text at the time the carousel is instantiated.
For example here I am instantiating a carousel within Drupal and utilizing Drupal's translate function to provide alternative language support for the previous/next buttons:
var srPrevious = '' + Drupal.t('Previous') + '';
var srNext = '' + Drupal.t('Next') + '';
$('#accessible-carousel').slick({
        dots: true,
        slidesToShow: 2,
        slidesToScroll: 1,
        prevArrow:'<button class="slick-prev slick-arrow" type="button">' + srPrevious + '</button>',
        nextArrow:'<button class="slick-next slick-arrow" type="button">' + srNext + '</button>',
});

the above works well, I just wish the slide dots had a similar feature.  In addition to that it would be great if the goto slide text was more descriptive (maybe include a title for that slide).  Current workaround is to add a slide description data-element on the initial slide div then use javascript to lift and shift the contents of that to update the slide dot sr-only text.

$('#accessible-carousel').slick({
        ...
	customPaging( _slider, i ) {
		return $( '<button type="button">'
				+ ''
				+ 'Gehe zu Bild ' + ( i+1 ) + ''
			+ '</button>' );
		},
});

See here: https://github.com/Accessible360/accessible-slick/blob/master/slick/slick.js#L61-L66


Improve nix setup + fix minor install script typo

Rewrite default.nix
Modify dev scripts to invoke bash through /usr/bin/env (required for systems that don't put bash in /bin/bash)
add a .envrc that invokes nix to integrate with editors
Fix minor issue in quick-install/install.sh where its banner unintentionally invokes gcloud

This looks great thanks!
Regarding the banner, I guess I never noticed, as I have gcloud installed.
I'll take a closer look later but I don't see any issues atm.

Just a friendly pointer to an extremely worthwhile project in this space: https://github.com/numtide/devshell

@blaggacao thanks for pointing out devshell and your example config, I'll definitely check them both out in depth.
With respect to the current PR it's my understanding that Nix hasn't been used as a development tool within this project for awhile. My aim is to keep things simple and workable for my own purposes while minimizing the overhead on the maintainers (that don't use Nix).

Yes, that's accurate @imacg. I don't use nix and can't really test this, but I'm very happy to include it if it doesn't affect others.
@imacg let me know when you're happy with the PR and I'll merge.

Just wanto to let you know that I'm planning on working on firat class support for hostctl and mkcert to nix's improved devshell today because I need to adopt trow in a "link-local" development setup through those tools.
Not meaning to hold back merging, though. An eventual later refactoring is straight forward.

@amouat I'm good with the current PR.

@imacg This PR qualifies for our Hacktoberfest campaign: https://blog.container-solutions.com/container-solutions-does-hacktoberfest
If you'd like a t-shirt, could you e-mail me at<EMAIL_ADDRESS>with your size and address?

Hi @imacg - I don't think you sent me an e-mail? If you'd like a t-shirt please send me an e-mail with your size and address.
Either way, thanks for the PR :)


Add BufferFile.Size() int64

I would love to use BufferFile to stage downloads I don't know the size of before copying them into SQLite blobs, but currently I still have to keep track of their size manually (while with os.File I'd use Stat).

Are you looking for BufferFile.Len()?
This is equivalent to n, _ := bf.Seek(0, os.SEEK_END). I added it to avoid the underlying syscall on large buffers.

I am, but I don’t see it in the godoc?
I admit I did not think about using Seek.

Because I forgot to push it weeks ago. Oops! Done.


[ Question ] Highlight doughnut segment on render

Hello,
I'm trying to highlight a segment of a doughnut chart after it renders. The idea is to already have some data highlighted.
Been looking everywhere and can't seem to find a way to do this.
Any help would be appreciated!

@stephendandrea what kind of highlight are you looking for? Just changing the colour? Changing the size as well?

Looks like this was also asked and answered on SO already.  Just thought I would add a comment so you can close and so others can benefit from later (if they search here instead of SO).
http://stackoverflow.com/questions/42816185/chart-js-set-active-segment-on-initialize/42817308#42817308


region.tag (t_multi)

im use command ".xregion.tag.forsale = 1" on t_multi of house
.xshow region.tag.forsale return 0
im use command ".region.tag.forsale = 1" in house area
.show region.tag.forsale return 1
.xshow region.uid and .show region.uid return equal
.xshow region.name return "name" of t_multi
.show region.name return "name of region (name of t_multi)" like Mainland (Wizard Tower)
i dont know why this work like that
in sphere ini
AreaFlags=02
and i checked if i do .region.tag.forsale = 1
in taglist on t_multi tag.forsale does not appear
and another wish
when command .region.tag.forsale
sphere server should send a reply message

in my defense, I will say that I study the scripts that come with the sphere X.
and I asked this question in discord

Honestly I don't remember how was it in older version of sphere but as I remember it was always like this, maybe I misremember but it can be script-side issue, or we may set region of the multi as a main region, so region could be used on multi.

I agree with this logic in scripts, but not when I do it manually in the game.

I've already sent a fix for it, it seems like REGION reference missed for multis so with a pr build it should work fine.


uploaded Image preview

After Uploading Image , it does not fit into the canvas even it is not responsive.
What you need to do
You need to make sure the the uploaded image fit to the canvas and it responsive as well.

Hi, is this task available? can you please assign me?

@leonyangela make sure to compare before making PR

@ZiaCodes ok thanks

hello, just asking if I can get the 'hacktoberfest-accepted' label for this?? Thanks

@leonyangela sure


Unable to upload files, error saying invalid 'purpose' value

https://github.com/openai/openai-dotnet/blob/3e12a623b0b4fd3b62abb0d7915422a4291b65c9/src/Custom/Files/Internal/InternalFileUploadOptions.cs#L38
when uploading files, it doesn't matter what upload purpose I choose, I get an "invalid purpose" HTTP 400 error. The OpenAI specs show these values in all lower-case, but the code here simply takes the Enum Value and ToString's it which would result in case like Assistants and not assistants
I'm hypothesizing this is the reason none of my uploads work.

Thank you for reaching out, @bc3tech ! Could you share the code that is failing with this error?

Yes I am merely doing
        var files = _azureOpenAiClient.Value.GetFileClient();
        try
        {
            var gptFile = files.UploadFile(fileName, OpenAI.Files.FileUploadPurpose.Vision);

So the problem isn't in the .ToString() now that I see that FileUploadPurpose is a struct & not an enum as I assumed.
However, there's still something going awry here as I cannot seem to use any file uploading capabilities, despite the MultiPart Form Data entry for purpose having the correct value (vision in my case)

Ah, I think this might be problem: I see your client is called _azureOpenAiClient which makes me think you're using the Azure.AI.OpenAI library with the Azure OpenAI service? If so, the Azure OpenAI service (as of version 2024-06-01) does not yet support "vision" as a file purpose. That's probably what's causing the error that you're getting. Could you try uploading files with other purposes, e.g., "assistants"?

Yes I also tried assistants and got the same error.

What version of the Azure OpenAI service are you targeting?

I assume you're referring to the api_version passed in the querystring. I've no idea; I'm simply using the Azure SDK, which internally uses this library, which is failing. I'm not specifying any version in my calls.

Thank you, @bc3tech . I suspect the issue that you are seeing is specific to the Azure service. Since this about the Azure.AI.OpenAI library, could you file this issue in the Azure repo here: 🔗 https://github.com/Azure/azure-sdk-for-net


State value to set for "yamaha.0.Realtime.MAIN.VOL" has to be type "number" but received type "string"

When changing the volume, I receive these messages in the debug log:

Versions:

Adapter version: 0.4.0
JS-Controller version: 3.3.20
Node version: 12.16.3
Operating system: linux

I just checked the code. State is created as string. Would it be possible, that state were created by an older version of adapter? Try to delete state and restart adapter. Is problem solved or still there?

Hey! I just gave it a try, deleted all objects, restarted the instance of the adapter, and unfortunately still get a ton of error messages:

I think you're still using 0.4.0. Please update to 0.4.1 on git and test again.

You are right. I ran the update to 0.4.1, just one info error when starting the adapter, and the volume one when changing volume:

Ok, I'll hve to check. Please give me some days ...

Can you check definition of state yamaha.0.Realtime.MAIN.VOL.
Here I see the following:
{   "type": "state",   "common": {     "name": "VOL",     "type": "string",     "role": "state"   },   "native": {},   "_id": "yamaha.0.Realtime.MAIN.VOL",   "from": "system.adapter.yamaha.0",   "user": "system.user.admin",   "ts":<PHONE_NUMBER>067,   "acl": {     "object": 1636,     "state": 1636,     "owner": "system.user.admin",     "ownerGroup": "system.group.administrator"   } }
You'll find this in objects panel when clicking at the pencil at the right side of the state.

Looks for me like that: {
"type": "state",
"common": {
"name": "VOL",
"type": "number",
"role": "state"
},
"native": {},
"_id": "yamaha.0.Realtime.MAIN.VOL",
"from": "system.adapter.yamaha.0",
"user": "system.user.admin",
"ts":<PHONE_NUMBER>012,
"acl": {
"object": 1636,
"state": 1636,
"owner": "system.user.admin",
"ownerGroup": "system.group.administrator"
}
}

Sorry, to ask again: you're sure that you deleted the whole state, not only the value of the state?

I don't know why, but I deleted the whole 0 folder, and now no error for the volume any more, and it is now also a string, like yours. Just the error for Pure Direct is left.
info | State value to set for "yamaha.0.pureDirect" has to be type "boolean" but received type "string"
{
"_id": "yamaha.0.pureDirect",
"type": "state",
"common": {
"name": "pureDirect",
"type": "boolean",
"role": "state",
"read": true,
"write": true,
"def": false,
"desc": "Pure Direct"
},
"native": {},
"from": "system.adapter.yamaha.0",
"user": "system.user.admin",
"ts":<PHONE_NUMBER>480,
"acl": {
"object": 1636,
"state": 1636,
"owner": "system.user.admin",
"ownerGroup": "system.group.administrator"
}
}

It seems that indeed, pureDirect will be set to "On" and "Off" and not true and false. Therefore I changed definition of state. Please delete state pureDirect and update to latest version in git.

I did the update, deleted all states, restarted the instance, and no error messages any more - good job, thanks for your work!!

You're welcome

Meanwhile I have updated to the latest Version (0.5.1) to solve the mentioned issue. But also after deleting states and later a complete reinstall of the adapter, the type is always set as "number". What am I doing wrong? Any hints?

Please show a screenshot of the state and of the settings (pencil icon)

{
"type": "state",
"common": {
"name": "VOL",
"type": "number",
"role": "state"
},
"native": {},
"_id": "yamaha.0.Realtime.MAIN.VOL",
"from": "system.adapter.yamaha.0",
"user": "system.user.admin",
"ts":<PHONE_NUMBER>080,
"acl": {
"object": 1636,
"state": 1636,
"owner": "system.user.admin",
"ownerGroup": "system.group.administrator"
}
}

What are the values stored in this state? Numbers or strings? Do you get warning messages about wrong types like the original poster?

yes, it's the same massage:
State value to set for "yamaha.0.Realtime.MAIN.VOL" has to be type "number" but received type "string"
That's why I can't explain it and asked.
After installing the adapter, the state does not yet exist. Only after using functions - e.g. the volume buttons on the remote control - the state is (wrong) created again.

Maybe I found a workaround: I manually changed the state type of xxx.MAIN.VOL to "string". After that, the message was gone. (Let's see how long)
In this context, however, I have noticed three "new" similar messages:
State value to set for "yamaha.0.Realtime.NETRADIO.CURRLINE" has to be type "number" but received type "string"
State value to set for "yamaha.0.Realtime.NETRADIO.MAXLINE" has to be type "number" but received type "string"
State value to set for "yamaha.0.Realtime.NETRADIO.LISTLAYER" has to be type "number" but received type "string"

Reopen because? Please post new logs and such ith a current version

Reopen because he said, that thre are still messages about wrong type. I closed it after the first fix, but then additional issues were found. So I opened it again.

I would love to get a new issue with a current version and a new log :-)) because else we mix toooo much together


[IRSA] Needs runAsUser: 1001

v.1.15.3 / k8s 1.21 on EKS (w/Fargate pods)
Disclaimer: Very new to all things k8s and cert-manager, any of these assertions could be wrong.
I followed the instructions to use IAM roles for my ServiceAccount, but I was always getting this:
Error presenting challenge: error instantiating route53 challenge solver: unable to assume role: AccessDenied: User: arn:aws:sts::xxx is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::yyy

Now I'm not completely sure that this fixed it, but I redeployed with the following values and it works now:
#cert-manager-values.yml
installCRDs: true
serviceAccount:
  create: false
  name: cert-manager
# the securityContext is required, so the pod can access files required to assume the IAM role
securityContext:
  fsGroup: 1001
  runAsUser: 1001
# this part is only required on Fargate; there's a port conflict there
webhook:
  securePort: 10251

Differences vs. the docs (and what I was trying initially) are

I used eksctl create iamserviceaccount to make the ServiceAccount, and told the chart not to make its own one. So C+Ping the ARN is not needed.
I added runAsUser: 1001

I think the chart-created SA should have been fine, but this feels less error prone to me. But I guess the runAsUser was what fixed it? If so, could that be added to the examples?
p.s. a minimal ClusterIssuer looks like
apiVersion: v1
kind: Namespace
metadata:
  name: letsencrypt-staging
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging-issuer
  namespace: letsencrypt-staging
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email<EMAIL_ADDRESS>    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - selector:
        dnsZones: ['your.domain.here']
      dns01:
        route53:
          region: your-region-here

maybe something like that could be added to the docs? Originally I had the role ARN in here but that's not needed in the IRSA case it seems.

@iainlane I've added a note about this to the documentation, in:

https://github.com/cert-manager/website/pull/1555

I'm not convinced that it is any longer necessary to change the cert-manager user and group.
Read @TBBle 's remarks in https://github.com/cert-manager/website/pull/1555#discussion_r1793968110 .
If you or others report that it is no longer necessary to alter the securityContext, then we shall remove that step from the documentation.


How to remove the lines above and below a post header?

There are lines above and below each post header. I'd like to remove the line above and remove the line below?

Can you be more explicit, or maybe share a screenshot or code sample? It's been a long time, and I don't know exactly what you are talking about... 😕

Of course. Here you go: https://www.dropbox.com/s/ps3d1lc4913q784/Screenshot 2016-04-12 15.52.35.png?dl=0

I would probably just hide them, with CSS...look at https://github.com/jipiboily/monologue/wiki/Customizations on how to do that.

Correct. My real issue is that I didn't know which CSS selector to override. After looking through the monologue source, specifically the monologue.css file, I added the CSS below to my project's custom.css file  to get what I wanted:
.content header {
border-top: none;
}


Request

Thank you for the plugin.
I have request.
Because iOS 13 is combining se sensor together can you make humidity and temperature as separate devices?

Reading this
It is fixed in the 13.2 beta.
https://homekithero.com/2019/10/02/whats-new-in-the-home-app-in-ios-ipados-13-2-developer-beta-1
Sent from my iPad

On Oct 2, 2019, at 8:25 AM, toshibochan<EMAIL_ADDRESS>wrote:
Thank you for the plugin.
I have request.
Because iOS 13 is combining se sensor together can you make humidity and temperature as separate devices?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I glad Apple will add this option. Thanks.


clean up redundant caches. domain global though

On several projects, where I've used sw-toolbox, I'm using versioning numbers as well to invalidate all my files at once - when it's needed of course.
Only problem is, that I ended up having multiple redundant caches, that basically just filled up space.
The challenge in removing redundant caches is the fact, that it's removing it form the whole domain.
Meaning if the website has a subsite under /serviceworker/is/the/best using a scoped serviceworker, their cache might all of a sudden not be available.
I'm not sure how to solve this yet, but I really badly needed the clean up and thought I might as well just implement it my self :)
How should we go around this?

Signed, sealed, delivered
On Tue, 3 May 2016 at 15:19 googlebot<EMAIL_ADDRESS>wrote:

Thanks for your pull request. It looks like this may be your first
contribution to a Google open source project. Before we can look at your
pull request, you'll need to sign a Contributor License Agreement (CLA).
[image: :memo:] Please visit https://cla.developers.google.com/
https://cla.developers.google.com/ to sign.
Once you've signed, please reply here (e.g. I signed it!) and we'll
verify. Thanks.

If you've already signed a CLA, it's possible we don't have your
GitHub username or you're using a different email address. Check your
existing CLA data https://cla.developers.google.com/clas and verify
that your email is set on your git commits
https://help.github.com/articles/setting-your-email-in-git/.
If you signed the CLA as a corporation, please let us know the
company's name.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/GoogleChrome/sw-toolbox/pull/130#issuecomment-216524550

Thanks for this @donlion, could you add some tests to make sure this works and doesn't regress.
I'm not sure if the renameCache method needs to be exposed, I'd expect the renameCacheAndCleanUp to be the only methods OR have two methods renameCache() cleanUpCaches().
@wibblymat and @jeffposnick what are your thoughts on this? I hit this problem last night on my own site.

Yeah. Reason why I kept renameCache and renameCahceAndCleanUp is for the domain-global "danger".
Say I've got domain.com as my main website and want to launch a web-app under the directory domain.com/web-app. If the users landed on domain.com and it cleaned up the cache, the cache for my web-app would be gone as well.
This is obviously a set example, but it could happen.
How should we go around it?
We could "namespace" the cache name to the service-workers scope and only remove caches within the current scope. What do you think?

I'd expect the renameCacheAndCleanUp to be the only methods OR have two methods renameCache() cleanUpCaches().

I concur.

@wibblymat and @jeffposnick what are your thoughts on this? I hit this problem last night on my own site.

Friendly ping Mat and Jeff for your take on this before we ask @donlion for tests.

The default cache name already contains the current service worker's scope:
name: '$$$toolbox-cache$$$' + scope + '$$$'

I may be missing something subtle here, but it doesn't sound like multiple service workers' usage of sw-toolbox would introduce a conflict, since they each should have their own scope.

Thanks for the PR!
I am currently -1 on this. If you use the default cache name then toolbox already manages it for you, and you don't have to clean anything up.
Maybe we could come up with some sort of optional helper, though, that makes tidying up custom caches  easier. Maybe something like expireCaches(prefix, suffix) that will, during activate, delete any cache that starts with prefix but does not end with suffix. This should help with versioned caches where the prefix will contain the scope and the suffix will be the current version. So if you have a cache called scope + '-images-' + version, then when you upgrade from version 4 to version 5 the version 4 cache will be deleted for you.
Does that make sense? Would it actually solve your problem?

Thanks for the initial review of the idea, @wibblymat and @jeffposnick. My two cents are that this PR would introduce sufficiently dangerous behavior that it's not worth pursuing in the current form. We do however greatly appreciate the contribution.

If a user is either managing a cache outside of sw-toolbox (e.g. via sw-precache or just vanilla Cache Storage API calls) or using sw-toolbox but with explicitly named caches (e.g. in this example), then those caches would end up being deleted by this code.

This is my concern too thus a -1 I'm afraid.

Maybe we could come up with some sort of optional helper, though, that makes tidying up custom caches easier. Maybe something like expireCaches(prefix, suffix) that will, during activate, delete any cache that starts with prefix but does not end with suffix. This should help with versioned caches where the prefix will contain the scope and the suffix will be the current version. So if you have a cache called scope + '-images-' + version, then when you upgrade from version 4 to version 5 the version 4 cache will be deleted for you.

I think this might be worth moving into a new issue. I'll file one for us to explore whether there's more that can be done to ease the process of tidying up caches.


Nice Article

Nice Article. You can check this article which is very short but nice -
https://indianjackofalltrade.com/what-is-new-in-sharepoint-2019/

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9f0c473c-9953-3f98-5ba5-7fda35879614
Version Independent ID: 79194e89-ce43-eefc-6439-cf2bd5bdac57
Content: New and improved features in SharePoint Server 2019 Public Preview
Content Source: SharePoint/SharePointServer/what-s-new/new-and-improved-features-in-SharePoint-Server-2019.md
Product: sharepoint-server-itpro
GitHub Login: @bentoncity
Microsoft Alias: stevhord

@Krishnangshu17 Thank you for the link and feedback


how to request permission for READ_PHONE_STATE

what did you use to request permission for READ_PHONE_STAE

Hi @darixsamani, to request access to the "READ_PHONE_STATE" permissions you need to use the "permission" package which allows you to request most of the available permissions.
Example:
Future<bool> requestPermission() async {
    var status = await Permission.phone.request();

    switch (status) {
      case PermissionStatus.denied:
      case PermissionStatus.restricted:
      case PermissionStatus.limited:
      case PermissionStatus.permanentlyDenied:
        return false;
      case PermissionStatus.granted:
        return true;
    }
  }


publish<EMAIL_ADDRESS>

Requested by: @bitsandfoxes
Quick links:

View changes
View check runs

Assign the accepted label to this issue to approve the release.
Targets

[ ] symbol-collector
[ ] upm
[ ] github
[ ] registry

Targets marked with a checkbox have already been executed.  Administrators can manually tick a checkbox to force craft to skip it.

We'll do some tests on the package before releasing so please don't slap approved on this one yet


Convert 'Bugzilla' references to the standard format

In virt team, contributors use
Bugzilla: <bzid OR bzlink>

in the MR message to reference bugs in Red Hat's Bugzilla.
This format, though, is different from the one required by BZ checks in
dist-git commit messages, which is (among others):
Resolves: bz#<bzid>

Do a multiline substitution, to convert the former to the later, and use
this string as the message for the dist-git MR and the commit message.
Signed-off-by: Hunor Csomortáni<EMAIL_ADDRESS>

@LaneWolf, please take a look on the test if this covers all the use cases you need.


Treat escaped newline as valid sequence

Summary
We weren't treating the escaped newline as a valid condition to trigger the safer fix (add an extra backslash before each invalid escape sequence).
Closes https://github.com/astral-sh/ruff/issues/11461.

Nice!


Misleading JSGI Promise Docs

James Cook reported a doc issue:

In the Promise docs, there is an example that I don't think works under any circumstances.
https://github.com/ringo/ringojs/blob/master/modules/ringo/promise.js#L26-L34
exports.asyncAction = function(request) {
var response = new Deferred();
setTimeout(function() {
response.resolve({
status: 200, headers: {}, body: ["Delayed"]
});
}, 1000);
return response.promise;
}
I don't see any code that would allow a promise to be returned to any of the JSGI methods and be able to handle it. In fact, I would really like this to exist. I was hoping I could just 'yield asyncAction()' to get the result, but that doesn't seem to work with this version of Promise.
It seems like this behavior bugged me quite a bit 3.5 years ago also.

The promise response has been replaced by AsyncRepsonse.


GPIO Interface

Write an Interface and a Implementation of it for accessing the GPIO Pins

Interface written, will need to check up on it on monday

Reopened because still needs work


Could not bind properties to CuratorConfiguration.ZookeeperProperties

你好 请问这个是不是没有上传ZookeeperProperties 这个文件呢？

Consider going to Kafdrop 3. As of version 3.10, the requirement to use ZooKeeper has been dropped completely. Cluster information is sourced entirely from the Kafka admin API.
There is no real development taking place on Kafdrop 2.x. Consider updating to the next major version – Kafdrop 3.


Poor visual orientation in folders of the workspace

In left EXPLORER pane of VSCode, when I add some folders to the workspace, the labels of these folders are of the same font, case and color as their subfolders and thus are poorly recognizable. There is only small indentation which is not enough. For example, if I expand several folders and subfolders in workspace, It is harder to find the particular project and decide which files and subfolders belong to which project. E.g. visual orientation in workspace is harder. For example in Visual Studio there is a bigger indentation and also there is (different) icon in front of project label.

This sounds like general feedback about VS Code's file explorer. In that case you should contribute over at the VS Code issue tracker.
Or is there something about this that is Live Share specific?

Thanks for your feedback. Yes I am closing this issue here and moving to vs code issue tracker.


Critical vulnerability in ltijs-sequelize

Any way you could patch this critical vulnerability?
No fix available
node_modules/mysql2
ltijs-sequelize  *
Depends on vulnerable versions of mysql2
node_modules/ltijs-sequelize

Yes can this be fixed?  I wont be allowed to deploy my code if there are critical issues.


Compact "divide and conquer" paginator

Always shows pages around first, last and current. Also shows the "..." ranges that can be expanded in "divide and conquer" manner to reach any desired page in 1-3 clicks.
Demo:

Closing in favor of a simpler approach (TBD).


Sort lines sorts quoted values inconsistently

Issue Type: Bug
Sort lines sorts quoted values inconsistently with the same values without double-quotes. I searched for similar bugs but didn't find much. Although, there's some complaint about sorting with mixed case, but I specifically avoid that in my examples.

Sort the following lines (asc)
a
abc-def
ab
abc

You should get
a
ab
abc
abc-def

Now sort the same lines, but wrapped in double-quotes (like items in a JSON literal often are)
"a"
"abc-def"
"ab"
"abc"

I expect to get
"a"
"ab"
"abc"
"abc-def"

But I actually get this, which must be incorrect, right?
"a"
"ab"
"abc-def"
"abc"

VS Code version: Code 1.28.0 (431ef9da3cf88a7e164f9d33bf62695e07c6c2a9, 2018-10-05T14:58:53.203Z)
OS version: Windows_NT x64 10.0.17134

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz (4 x 2808)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: disabled_softwarerasterization: enabledvideo_decode: enabledvideo_encode: enabledwebgl: enabledwebgl2: enabled

Memory (System)
15.89GB (2.21GB free)

Process Argv
.

Screen Reader
no

VM
0%

Extensions (7)

Extension
Author (truncated)
Version

tslint
eg2
1.0.40

vscode-gitk
how
1.3.2

csharp
ms-
1.16.2

Go
ms-
0.6.91

PowerShell
ms-
1.9.0

vsliveshare
ms-
0.3.790

vscode-docker
Pet
0.3.1

(1 theme extensions excluded)

Hi, I have looked into this issue and I believe I found why the sorting is not consistent. If you look at https://github.com/Microsoft/vscode/blob/43823dae97f07760958ffbb5e16c7f219307159c/src/vs/editor/contrib/linesOperations/sortLinesCommand.ts#L75  it uses localeCompare. localeCompare does not regonize "a" and a as the same.  This can be demonstrated with this line of code:
var a = '\"A\"';
var b = 'A';
console.log(a.localeCompare(b));

I think one way of solving the problem would be to strip all special characters from the string array and then sorting.

IMO the root cause is we try to compare - and " and - wins in this case.

TLDR; impact of unicode collation explored
This particular example does look very strange! Here are the code points for the characters in these strings:
let unquoted_strings = [
    [97],  // a
    [97, 98],  // ab
    [97, 98, 99],  // abc
    [97, 98, 99, 45, 100, 101, 102],  // abc-def
]

let quoted_strings = [
    [34, 97, 34],  // "a"
    [34, 97, 98, 34],  // "ab"
    [34, 97, 98, 99, 45, 100, 101, 102, 34],  // "abc-def"
    [34, 97, 98, 99, 34],  // "abc"
]

...It makes no apparent sense that 34 sorts after 45 in those last two strings... and there's this:
'a' < 'b' // true
// in compare functions: -1 means a<b; 0 means a==b; 1 means a>b
'a'.localeCompare('b') // -1
'"' < '-' // true
'"'.localeCompare('-') // 1 <-- ??? not -1 ??? (╯°□°) ╯ ︵┻━┻

Why? Because Javascript's locale-aware (unicode) string comparison function (used by vscode sort functions) is not primarily based on comparing code point values, instead it is done "by assigning multi-level weights to characters or sequences of characters and then comparing those weights on each level" aka "collation."

the standard Unicode Collation Algorithm (UCA) comes with a default weight table for all assigned characters

I believe that refers to the latest UCA allkeys file -- the format of the allkeys file is described here.
Here are some lines from the current allkeys file (in no particular order) :
0022  ; [*031D.0020.0002] # QUOTATION MARK
002D  ; [*020D.0020.0002] # HYPHEN-MINUS
0061  ; [.1FA2.0020.0002] # LATIN SMALL LETTER A
0062  ; [.1FBC.0020.0002] # LATIN SMALL LETTER B
0063  ; [.1FD6.0020.0002] # LATIN SMALL LETTER C

Expressing the above hexadecimal numbers in base 10:
34  ; [*797.32.2] # QUOTATION MARK
45  ; [*525.32.2] # HYPHEN-MINUS
97  ; [.8098.32.2] # LATIN SMALL LETTER A
98  ; [.8124.32.2] # LATIN SMALL LETTER B
99  ; [.8150.32.2] # LATIN SMALL LETTER C

So for the codepoints 34 (quotation mark) and 45 (hyphen/minus) their respective weights boil down to 797 and 525 respectively. They are kinda "flipped" at least with respect to ASCII/their codepoint values. (╯°□°) ╯ ︵ᴉᴉɔsɐ
What could be done to make vscode sort in the way you might expect/want? localeCompare supports some options like ignorePunctuation (and also sensitivity)
let quoted_strings = [ '"a"', '"ab"', '"abc-def"',  '"abc"' ];
quoted_strings.sort(
  (a,b)=> a.localeCompare(b, undefined, {ignorePunctuation: true})
);
// quoted_strings == [ '"a"', '"ab"', '"abc"', '"abc-def"' ]

Setting ignorePunctuation: true would help this case, but would be very unhelpful for others. I would love to just have more sort commands in the menu with different values for ignorePunctuation and sensitivity which give access to the sorting we need for a particular purpose. It could also be useful to have a sort command that does actually sort by codepoint values.


multi-signer prep

There are a number of things needed for tdnsd to be a viable "signer" in a multi-signer context. Among them:

[x] A more sophisticated model for a zone's DNSKEYs. Must include a distinction between internal keys (that may be rolled) and external keys that belong to another signer and should just be kept in the RRset.
[ ] Same thing for the NS RRset: internal and external NS.
[ ] Either add support for TSIG or add SIG(0) support to MUSIC or add API support to both so that they communicate. API support probably best alternative.
[ ] Add support for sending NOTIFY(DNSKEY) sideways (to either a central MUSIC or a sidecar MUSIC) when the contents of the DNSKEY RRset changes.

Initially we will not add any key rollover support to TDNSD.

How should a MUSIC sidecar and TDNSD shake hands? In the API case we can do whatever we want, but in the tdns-agent + DDNS updates the communication is more limited. The base line things that need to be established are:

[ ] TDNSD/tdns-agent should notify MUSIC about DNSKEY events for the following zones ...
[ ] TDNSD/tdns-agent should notify MUSIC about changes to delegation data (i.e. same trigger as will initiate a sync with the parent) for the following zones ...

For DNSKEYs it would seem suitable to send a NOTIFY(DNSKEY) when changing ZSKs, but perhaps a NOTIFY(CDS) when changing KSKs. For changes to the delegation data, why not send a NOTIFY(CSYNC) to MUSIC (in addition to sending it to the parent).
How to know where to send these notifications? TDNSD/tdns-agent could look for the DSYNC RRset in the zone, but for a new scheme, perhaps "MULTI-SIGNER NOTIFY", aka scheme=4.


Bump senzing from 3.3.0 to 3.3.1

Pull requests:

https://github.com/Senzing/senzingapi-runtime/pull/34


Don't write value to model when change comes from model.

If a changed value comes from the model, then we shouldn't be writing it back to the model. Usually this would be a no-op but when a StringFormat is applied it causes the formatted value to be sent back to the underlying data model, and if the StringFormat is not round-trippable causes an exception.

Tested and given 👍 by customer, so going to merge.


Recipe to install super agent limited preview

Added recipe for super agent preview for Linux(ubuntu, debian, rhel, suse, aws linux)

Do we wish to exclude super-agent recipes during guided installation similar to OTEL?
Infra + Log is installed by default by CLI unless specified. For OTEL, do we want to skip these default installations?
Many of the tests are failing, probably due to the package not being ready yet, When do we expect it to be ready?

I reviewed the recipe, it looks fine. I'm just not sure how {{.NR_CLI_INFRA_AGENT}} variable works. Also, it's mentioned we want to allow user to able to choose between infra and OTEL. I'm not seeing anything related to that in the recipe.

Do we wish to exclude super-agent recipes during guided installation similar to OTEL?
Infra + Log is installed by default by CLI unless specified. For OTEL, do we want to skip these default installations?
Many of the tests are failing, probably due to the package not being ready yet, When do we expect it to be ready?

I reviewed the recipe, it looks fine. I'm just not sure how {{.NR_CLI_INFRA_AGENT}} variable works. Also, it's mentioned we want to allow user to able to choose between infra and OTEL. I'm not seeing anything related to that in the recipe.

Hi @NRhzhao:

Yes we want to do similar to OTEL.
Yes, we want to skip those installations, but, we would like to tell the users to install the logging recipe after the super agent (-n logs-integration), but we have seen there is the dependency of the old infra-agent so if they execute the recipe it will also install the old infra-agent. Is there a way to disable the dependency?
Regarding the tests, the packages are there, I don't know why most tests fail but just a few of them run. Also note that one test from infra-agent is being executed (https://github.com/newrelic/open-install-library/actions/runs/5736264513/job/15712399760?pr=949#logs). Can you help us to understand why this test is executed and why the tests fail?

Do we wish to exclude super-agent recipes during guided installation similar to OTEL?
Infra + Log is installed by default by CLI unless specified. For OTEL, do we want to skip these default installations?
Many of the tests are failing, probably due to the package not being ready yet, When do we expect it to be ready?

I reviewed the recipe, it looks fine. I'm just not sure how {{.NR_CLI_INFRA_AGENT}} variable works. Also, it's mentioned we want to allow user to able to choose between infra and OTEL. I'm not seeing anything related to that in the recipe.

Hi @NRhzhao:

Yes we want to do similar to OTEL.
Yes, we want to skip those installations, but, we would like to tell the users to install the logging recipe after the super agent (-n logs-integration), but we have seen there is the dependency of the old infra-agent so if they execute the recipe it will also install the old infra-agent. Is there a way to disable the dependency?
Regarding the tests, the packages are there, I don't know why most tests fail but just a few of them run. Also note that one test from infra-agent is being executed (https://github.com/newrelic/open-install-library/actions/runs/5736264513/job/15712399760?pr=949#logs). Can you help us to understand why this test is executed and why the other tests fail?

Hello @alvarocabanas,

We hardcoded for not installing OTEL during guided installation due to time restrictions. This time around, the team would like to add a new schema in the recipe to signal to skip Guided Install. I can create a ticket and get that started.
The logs recipe depends on infra recipe to install, and they are bundled together today.  Other OHI also depends on Infra to install. We don't have a way right now to install logs recipes without infra (Without hacks). How I understand is super-agent will handle all other installations? Maybe this is something we can discuss?  @Julien4218 @svandamme
Only super-agent tests are running now after I merged from main branch. I also tweaked github action settings, we been having some issue with our tests running. The tests are failing for some of the distros due to the recipe, can you help take a look? I included the result, finding below.

2. ogs recipe depends on infra recipe to install, and they are bundled together today.  Other OHI also depends on Infra to install. We don't have a way right now to install logs recipes without infra (Without hacks). How I understand is super-agent will handle all other installations? Maybe this is something we can discuss?  @Julien4218 @svandamme

Hi @NRhzhao:
Thanks for fixing the tests with the merge, regarding the few machines that failed, I tested one of them and it's because of an issue with the build super-agent. It's an error we are fixing and it's regarding really old distros having old versions of GLIBC. When we have a new release I will relaunch those tests.

ogs recipe depends on infra recipe to install, and they are bundled together today.  Other OHI also depends on Infra to install. We don't have a way right now to install logs recipes without infra (Without hacks). How I understand is super-agent will handle all other installations? Maybe this is something we can discuss?  @Julien4218 @svandamme

Hi @NRhzhao: Thanks for fixing the tests with the merge, regarding the few machines that failed, I tested one of them and it's because of an issue with the build super-agent. It's an error we are fixing and it's regarding really old distros having old versions of GLIBC. When we have a new release I will relaunch those tests.

Can you check for Amazon Linux 2, and Amazon Linux 2023 tests? I think there might be some incorrect logic since we combined all the rhel recipes (I notice there are difference in assertPreReq and install sections) These two distro are our most popular, want to make sure they are working as intended.

Did a quick test:

Running guided install does not discover super-agent, or log-super-agent
Target install with -n super-agent,logs-integration-super-agent works the first time. But sequential times show log unsupported @alvarocabanas
Need to update the tests since you updated the name to super-agent
Might be nice to add couple test or update couple test to install logs + super-agent

✅  Running guided install does not install/offer super-agent
✅ Able to run super agent install and infra-install
Update the super agent log tests, make sure you have NEW_RELIC_CLI_SKIP_CORE=1, and -n super-agent,log-integration-super-agent otherwise regular log is installed
Everything looks good, are we ready to merge the PR? One last thing is maybe to consider moving all the tests under manual folder. Where it's now, it's going to be part of regression tests, I'm not sure we need to include them at this point.

If I install infra first, then install super-agent, I get two host entities in UI.
If I install super agent first then infra, I only get one host entity.


Allow closing brackets on the same line

Pre-Submission Checklist

[x] Your pull request targets the staging branch of freeCodeCamp.
[x] Branch starts with either fix/, feature/, or translate/ (e.g. fix/signin-issue)
[x] You have only one commit (if not, squash them into one commit).
[x] All new and existing tests pass the command npm test. Use git commit --amend to amend any fixes.

Type of Change

[x] Small bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds new functionality)
[x] Breaking change (fix or feature that would change existing functionality)
[ ] Add new translation (feature adding new translations)

Checklist:

[x] Tested changes locally.
[x] Closes currently open issue (replace XXXX with an issue no): Closes #13078

Description

@Greenheart @ajain17 I was going to go in favor of changing the first test to be: assert(code.match(/\\{\\s*\\}\\);/g) to ensure no text is there. Right now the first test checks only that the last few characters of each function are removed.

@dhcodes Any updates on this? I think it would be nice against @ajain17 if we could come to a conclusion :wink:
Do we request further changes or can we merge this?

Hi, sorry for the delay. was traveling. will do it today.

Kind of indifferent. Sort of depends on if you feel the part in the between the { } should be blank

@Greenheart @dhcodes bump for a review.

I'm not sure this ever got updated? @ajain17

@dhcodes  I updated it about 16 days back

Okay. Camperbot was down so it was hard to tell. My bad, will test today. Thanks!

Tested locally. LGTM Thanks for making this fix! 👍  💯  🌮

سوف امسح الايميل
بتاريخ ٢٠١٧/٠٢/٢٤ ٧:٢٦ م، كتب "Dylan"<EMAIL_ADDRESS>
Tested locally. LGTM Thanks for making this fix! 👍 💯 🌮
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/freeCodeCamp/freeCodeCamp/pull/13080#issuecomment-282335196,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHhvsdO4qyjmZPEYf67WPBCQ6Ip0ERWjks5rfwTGgaJpZM4L1XjI
.


document cross support

fix readme to not recommend non-cross-friendly constructs that use
deprecrated buildRustCrate argument to Cargo.nix
improve warning
fix this warning in default.nix
Source: https://github.com/kolloch/crate2nix/pull/160/files#diff-381a73e3c364568cfbd7c56e32e1db595235606948c51c06f04ff99d75aec589
cc @Ericson2314
Motivation: this warning mentions buildRustCrate but in nix-du's nix files there is no occurence of buildRustCrate in manually written files. It took me really long to understand why the warning was fired.

The ci is flakey, I give up.

ping @kolloch

Is getting rid of callPackage related?

this should be squashed at merge.
@Ericson2314 yes.

Thank you very much :)
Feel free to add yourself to the changelog.

It would be nice if callPackage was still OK though.

@Ericson2314 I agree. Basically, I strive to prevent breaking anything that worked before, except it is clearly marked as internal. Is it broken now?
I missed the change that breaks it.

you can callPackage { buildRustCrate = null; } but that's verbose imo


SSD1306Wire: Option to use other speeds than 700kHz

Hi Daniel.
Thank you for your great library.
I have both an OLED and a PCF8574 I/O expander on the same I2C-bus.
My problem is that the PCF8574 is a 100kHz device, and SSD1306Wire::connect() sets the bus-speed to 700kHz.
I think it should be possible for the user of the library to control the clock frequency.
One possible solution is to provide an extra constructor: SSD1306Wire(uint8_t _address, TwoWire* wire). Then I can pass &Wire after setting it up like I want.
Another soultion is to add the wanted clock frequency to the existing constructor, and let 700kHz be the default: SSD1306Wire(uint8_t _address, uint8_t _sda, uint8_t _scl, unsigned int freq = 700000)
I will gladly provide a pull request, but I would like to get your input first on which solution you prefer.
Best regards,
Søren Thing, Denmark

any progress on this?

I am still waiting for an indication regarding Daniel Eichhorn preferred solution.


Adds NavList.GroupHeading component

We needed linked group headings when developing the new Primer docs. Instead of faking it by using ActionList.GroupHeading, we discussed updating the NavList API to allow rendering NavList.Group headings using NavList.GroupHeading instead of the more restrictive title prop: <NavList.Group title="Group title">
This was discussed in a weekly Primer patterns/API working sesh on Sept 24 2024 (notes)
The NavList group headings would be styled exactly the same except when they're hovered. The first group heading in this image is being hovered.

Changelog
Adds NavList.GroupHeading component
New
NavList.GroupHeading component
Changed

Removed

Rollout strategy

[ ] Patch release
[x] Minor release
[ ] Major release; if selected, include a written rollout or migration plan
[ ] None; if selected, include a brief description as to why

Testing & Reviewing

Merge checklist

[ ] Added/updated tests
[ ] Added/updated documentation
[x] Added/updated previews (Storybook)
[ ] Changes are SSR compatible
[x] Tested in Chrome
[ ] Tested in Firefox
[ ] Tested in Safari
[ ] Tested in Edge
[ ] (GitHub staff only) Integration tests pass at github/github (Learn more about how to run integration tests)

I'm seeing some different styles in the heading between the "With Group" and "With Group Heading Links" stories (mainly the color), wondering if that's intended

This is an intentional style change we decided to make last week.

Also wondering what would happen if a user used this in alongside the title prop (and if we even care to cover/prevent for that use case)

Great question! I'll look into this and add some guards to prevent undesired results.