support ES6 features

Several valuable ES6 features appear to be stable (arrow functions, classes (modulo private members), block scoping) and this month's (May 2013) tc39 meeting seems to be the deadline for a wiki-complete module system spec. Transpilers and engines are already prototyping support, and it is important that tern is not left behind.
If I recall correctly, the esprima and spidermonkey folks have started to agree on the syntax/AST extensions, but tern also needs to take the scoping, typing, and semantics into account (eg desugaring class syntax to ES5 features while keeping ES6 locations).

Is there any roadmap to support ES6 features?
I'm especially interested in classes and module imports.

Is there any roadmap to support ES6 features?

Tern starts supporting ES6  with ecma6.json https://github.com/marijnh/tern/blob/master/defs/ecma6.json
@marijnh has managed Promise, but ecma6.json must be improved again like Iterator support.
But I think Acorn should be customized to support ES6 inside tern.

@angelozerr

But I think Acorn should be customized to support ES6 inside tern.
I don't really understand why it should be customized to support ES6 differently inside Tern? How is that different from already implemented ES6 support?

@RReverser I don't know acorn, but with acorn you can set ecmaVersion http://marijnhaverbeke.nl/acorn/#section-4 although inside tern it is never done.
I suppose setting ecmaVersion gives features according the version (perhaps acorn is able to support iterator protocal?
But perhaps I'm wrong?

Yes, we are currently not enabling ES6 parsing when calling acorn.parse because Tern does not understand the ES6 node types. The way to start supporting this would be to extend the AST walkers in infer.js so that they properly handle the new syntactic constructs.
I will to work on this eventually, but I'll need time and funding. The second can probably be addressed with another crowd-funder, which I intend to start at some point when I am less busy.

@marijnh I was also just thinking about getting to work on this as soon as finish better loose support in JSX, as I've got some free time, but same problem with 2nd part of your statement :)

Here's what you can do: Open a bounty on Bountysource for the specific ES6 features you are most interested in. Link it from this issue, and either put money into it right away, or ask me and/or Ingvar what we'd charge to implement it. There's a quite a lot of new stuff in ES6, but if we split it up into smaller features, estimating the cost and raising the funds shouldn't be too hard.

:+1: I've been using sublime code intel which doesn't support ES6. If ternjs supported ES6 I would definitely switch, as it will make coding much more pleasant. I see class support as being the most important here, as it will have a large impact on code completion.

Hi all,
I've spent the past weeks implementing ES6 support for Tern, including module support. You can see a demo at http://ternjs.net/es6/doc/demo/#ES6
Since giving work away and hoping for the best isn't turning out to be a good way to earn a living, I'm holding the code hostage (not open sourcing it) until the Bountysource item at https://www.bountysource.com/issues/1141202-support-es6-features reaches 2000€. If you want to donate towards that sum in another way, contact me.
Some notes on the module loader system:

The node module loader was split out of the node plugin and into modules (basic module tracking) and commonjs (require-style module loading) plugins.
There's a new es_modules plugin that manages import/export module connections, also using the infrastructure in the modules plugin
The modules plugin only defines a very primitive module resolution strategy (resolve relative paths). The node plugin adds node's module resolution rules. So if you are using import for node-style modules (under node_modules/...), you have to load the node plugin to find them.
Plugins dependencies will be mostly automatically loaded, but if you use Tern in a way where plugins can't be automatically loaded (raw script tags, for example), you'll have to include them manually.

Other ES6 stuff implemented in this branch:

Block scopes and let/const bindings
Destructuring and default values in assignments, parameters, and bindings
Tuples/heterogenously typed arrays to make patterns like [a, b] = [foo, bar] work
Rest arguments
Generators
Symbols and symbol properties
For/of iteration
Classes and super calls
Extended object literal notation (computed properties, succinct methods, name-only properties)
Arrow functions
Template strings
All methods and types added by the ES6 spec
Improved existing support for Promises

Update: Since BountySource doesn't seem to do Euros, I've switched the goal's currency to dollars. (I.e. you just got a 10% off deal).
Also, everybody who contributes $30 or more has my permission to start using this code by git cloning http://marijnhaverbeke.nl/git/tern-es6/. Proper packaging will happen when the goal amount is reached.

@marijnh the linked tern-es6 version does not yield any results for anything when I try it with the tern vim plugin. I always only get the emptry string as return value. Have you tested it with the vim plugin ? Are there any kind of breaking changes that would need to be adapted on the vim plugin side ?

Hi @gilligan, did you npm install after getting/upgrading the code? I just tried with Vim, and it works on my machine. (The HTTP protocol, which the Vim plugin uses, stayed the same.)

after I append export default to any file, tern-es6 seems to stop working anymore

@marijnh sometimes it works, apparently rarely though. I will try to work out some minimal test cases

@L42y @gilligan The export default issue is fixed now (was a typo in a property name)

@marijnh Where I can get the patch? Couldn't find it on GitHub or http://marijnhaverbeke.nl/git/tern-es6/

Ah, right, I'm in the middle of a server migration and marijnhaverbeke.nl isn't switched over yet. I updated the repo on that server as well now.

@marijnh asking for bounties is great check,
but what will be after $2000 mark?
well, I don't think I would cover your time spent, and what for next work to do?
Should it be actually some monthly check to motivate (but not cover all expenses)
What can be next?
I guess http://ternjs.net/ should get "Donate" button or this bounty will stay open forever.
Bountysource looks better as more transparent.
P.S. it is midnight now.

I would have to agree. I have already contributed 30USD because that was what I was supposed to pay to for allowance to use the es6 branch. That already felt somewhat "off" to be honest.
As you can see I have no issues with contributing something to an open source project, but this should happen organized in a transparent fashion in my opinion. Currently this is the exact opposite.
ES6 support is already there and people have to pay to use it. That is just weird. At this point you might as well make it a closed source project and ask me to just buy it. Which I would in fact do (or already did).

Please don't take this the wrong way, I am merely trying to point out how this is perceived by me and maybe others. It remains your project with which you can do as you please of course.
Best,
Tobi
Sent from my iPhone

On 30 Aug 2015, at 18:01, Paul Verest<EMAIL_ADDRESS>wrote:
@marijnh asking for bounties is great check,
but what will be after $2000 mark?
well, I don't think I would cover your time spent, and what for next work to do?
Should it be actually some monthly check to motivate (but not cover all expenses)
What can be next?
I guess http://ternjs.net/ should get "Donate" button or this bounty will stay open forever.
Bountysource looks better as more transparent.
P.S. it is midnight now.
—
Reply to this email directly or view it on GitHub.

Just for the record, I donated and I don't perceive it that way; I would definitely not donate for a closed source project, because I believe that the benefit an open source Tern would bring to the JS community now that we are getting ES6 can be truly great (ES6 as a language is a lot more amenable to static analysis, but as its new few editors have decent support for it which makes people reluctant to switch)

We've reached $ 1,772 at bountysource and I've recently received another Tern-related contribution outside of that platform that I'm going to count too, putting us over the goal. I've just merged my es6 branch into master, and the ES6 work is now under the MIT license.
@gilligan Could you elaborate on your concerns about lack of transparency? I stated exactly what I did, and what happens when my financial goal is reached. You state that you'd prefer for this to be a closed-source commercial project. How would not being able to read, modify, or fork the source benefit you?

(Please open new issues for shortcomings and bugs found in the ES6 implementation.)

And I forgot to mention -- there'll be a new release with this code tomorrow.

Thank you for your hard work and for making tern “open source”.
But I just can’t hold back my opinion that people should be weary of non-copyleft licenses and CLAs, because they allow such “un-sportsmanlike” behavior.

@Swatinem Do elaborate what exactly you are perceiving as un-sportmanlike behavior. (And how a copyleft license would have helped.)

@marijnh It may be in everybody's best interest to lock this thread as it's now a discussion of licenses instead of ES6 support in tern.

Sorry, I obviously failed at expressing myself. I would certainly not prefer a closed source approach. I have been travelling the whole day and I am deas tired. Will reply properly soon, sorry & g'nite :)
Sent from my iPhone

On 30 Aug 2015, at 22:04, Marijn Haverbeke<EMAIL_ADDRESS>wrote:
We've reached $ 1,772 at bountysource and I've recently received another Tern-related contribution outside of that platform that I'm going to count too, putting us over the goal. I've just merged my es6 branch into master, and the ES6 work is now under the MIT license.
@gilligan Could you elaborate on your concerns about lack of transparency? I stated exactly what I did, and what happens when my financial goal is reached. You state that you'd prefer for this to be a closed-source commercial project. How would not being able to read, modify, or fork the source benefit you?
—
Reply to this email directly or view it on GitHub.

I think Marin should be a slave to all of us and have no rights, only
obligations to JavaScript programmers who don't have the time or skill to
do what he can.
Seriously, thanks Marin. I think getting the community to sponsor the work
is an incredibly reasonable approach. The money hopefully enables you to
have some time to really work on this project and then it's shared with the
community in true open source fashion. Honestly, I almost like the hostage
approach more - less risk as a donater.
On Sun, Aug 30, 2015, 3:19 PM Tobias Pflug<EMAIL_ADDRESS>wrote:

Sorry, I obviously failed at expressing myself. I would certainly not
prefer a closed source approach. I have been travelling the whole day and I
am deas tired. Will reply properly soon, sorry & g'nite :)
Sent from my iPhone

On 30 Aug 2015, at 22:04, Marijn Haverbeke<EMAIL_ADDRESS>wrote:
We've reached $ 1,772 at bountysource and I've recently received another
Tern-related contribution outside of that platform that I'm going to count
too, putting us over the goal. I've just merged my es6 branch into master,
and the ES6 work is now under the MIT license.
@gilligan Could you elaborate on your concerns about lack of
transparency? I stated exactly what I did, and what happens when my
financial goal is reached. You state that you'd prefer for this to be a
closed-source commercial project. How would not being able to read, modify,
or fork the source benefit you?
—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
https://github.com/marijnh/tern/issues/116#issuecomment-136208056.

Since the trolling is now commencing I will move this elsewhere. You should think about how you communicate. About exaggerations and accusations that abort discussions instead of resolving them by actually having them.

On 31 Aug 2015, at 04:17, Ben Joldersma<EMAIL_ADDRESS>wrote:
I think Marin should be a slave to all of us and have no rights, only
obligations to JavaScript programmers who don't have the time or skill to
do what he can.
Seriously, thanks Marin. I think getting the community to sponsor the work
is an incredibly reasonable approach. The money hopefully enables you to
have some time to really work on this project and then it's shared with the
community in true open source fashion. Honestly, I almost like the hostage
approach more - less risk as a donater.
On Sun, Aug 30, 2015, 3:19 PM Tobias Pflug<EMAIL_ADDRESS>wrote:

Sorry, I obviously failed at expressing myself. I would certainly not
prefer a closed source approach. I have been travelling the whole day and I
am deas tired. Will reply properly soon, sorry & g'nite :)
Sent from my iPhone

On 30 Aug 2015, at 22:04, Marijn Haverbeke<EMAIL_ADDRESS>wrote:
We've reached $ 1,772 at bountysource and I've recently received another
Tern-related contribution outside of that platform that I'm going to count
too, putting us over the goal. I've just merged my es6 branch into master,
and the ES6 work is now under the MIT license.
@gilligan Could you elaborate on your concerns about lack of
transparency? I stated exactly what I did, and what happens when my
financial goal is reached. You state that you'd prefer for this to be a
closed-source commercial project. How would not being able to read, modify,
or fork the source benefit you?
—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
https://github.com/marijnh/tern/issues/116#issuecomment-136208056.

—
Reply to this email directly or view it on GitHub.

@gilligan I was trying to have a discussion, by asking you to clarify your issue. I see no exaggeration or accusation in any of my messages here. I am, in fact, rather pissed off about your tone earlier in the thread, but I am trying to figure out what you mean before jumping to any conclusions.

Oh i did not mean you at all. And if I have offended you in any way I am sorry since this was not my intention.
What I was referring to were messages by people that jumped in drawing black/white pictures while apparently not understanding what I was on about.
"OMG This looks to me like someone said something about F/OSS that I don't like. Lets jump in !"
I have used open source software all my life, make anything l do open and try to open source things at work.
What I was trying to say was...

i thought it as unfortunate that there is some donation button with no clear goal/timeline/topic. You clarified it in comments later on

i found it odd that you would ask for donations on something that already exists in a private repo. I practically bought my es6 tern copy.

That being said I did donate and started talking to my employer about supporting the project. It boils down to me (and some colleagues) finding the process a bit odd.
I pointed out that it is your project and you can do with it as you wish. Yet I think it is only fair that I'd be allowed to share some concerns.
Lets not draw conclusions too quickly and cool down.

On 31 Aug 2015, at 09:54, Marijn Haverbeke<EMAIL_ADDRESS>wrote:
@gilligan I was trying to have a discussion, by asking you to clarify your issue. I see no exaggeration or accusation in any of my messages here. I am, in fact, rather pissed off about your tone earlier in the thread, but I am trying to figure out what you mean before jumping to any conclusions.
—
Reply to this email directly or view it on GitHub.

i found it odd that you would ask for donations on something that already exists in a private repo. I practically bought my es6 tern copy.

The strategy of asking people to pay for something they can get for free isn't usually terribly effective (though I'm definitely working on that front too, see my funding page, also linked from the project page). I've found that having some kind of 'carrot' that makes donating attractive can make a huge difference. I do not see how implementing first, then asking for money, is problematic. At least that way you're able to play with the implementation and verify that it works before deciding on whether to shell out for it.

The difference is exactly the difference between donating and purchasing.
By "donating" 30 USD I purchased a license (by shape of your expressed allowance) to use the es6 featured branch in your private repo.
I did do this because I am really fond of tern - it is a productivity boost to me - but I find this quite odd.
That is the reason why I suggest that you might want to think about a different model if money is a key factor. I have no problem in actually purchasing it. You could still have it on github and/or make it free for non commercial use.
So nobody pushed the donate button? Yes, that is what the open source community is like a lot. But your carrot is not just a carrot it hints at the fact that maybe you want/need a different model.
So that is what I referred to with the cake analogy. A github repo won't make an income.

On 31 Aug 2015, at 12:38, Marijn Haverbeke<EMAIL_ADDRESS>wrote:

i found it odd that you would ask for donations on something that already exists in a private repo. I practically bought my es6 tern copy.

The strategy of asking people to pay for something they can get for free isn't usually terribly effective (though I'm definitely working on that front too, see my funding page, also linked from the project page). I've found that having some kind of 'carrot' that makes donating attractive can make a huge difference. I do not see how implementing first, then asking for money, is problematic. At least that way you're able to play with the implementation and verify that it works before deciding on whether to shell out for it.
—
Reply to this email directly or view it on GitHub.

it hints at the fact that maybe you want/need a different model.

No, I definitely don't. See also this blog post

I really don’t want to make this into a flame war of any kind.
Also note that I’m not a lawyer so I probably have no idea what I’m talking about.
To reply to @gilligan

I pointed out that it is your project and you can do with it as you wish.

I don’ think that statement is true. According to github its the shared work of 59 different contributors (although clearly @marijnh has done most of it)
But the statement is also true because of the license. The MIT License basically gives anyone the ability to do whatever they wish with the software, including making a closed source fork and sell it for money.
Doing so is just fine by the license, but it sure offends some of the other 58 contributors. It sure would have pissed me off a lot. But alas, I am not one of the contributors.
I actually did check out the code some time ago and started implementing a feature without much success and never got around to actually finish it and make it work. But now at least I know not to contribute anything in the future. (Unless I sub-license my fork as GPL, which the MIT license allows me to do)
I admire your goal to actually make a living writing open source software and I wish you all the best for it. As in, doing the things you love as opposed to doing the things someone else tells you to. But the point of open source is that it invites others to contribute as well. And holding [their] code hostage as you said is simply not a good way to invite further contributions.
Again, I’m very sorry if this might have started a flame war.  The master branch was entirely unaffected by the crowd-funding, and the only new work on the (briefly) non-free branch was my own work.


Multiple Windows

It occurred to me - why isn't there some sort of level in the UI tree above Window?  This would make synchronizing multiple windows a lot less trouble.  Something like
App [
    Window [ ]
    Window [ ]
    ...
]

Alternately, the App level would have a single child which would be some collection of Windows.  It might also make sense to instead of App call the root Context, since maybe there's a reason why the developer would want more than one of them.
My use case is that I want a UI that's friendly with multiple monitors.  Something like Visual Studio's dock system seems like a good place to start, and it would be nice to be able to easily break toolbars and such off into separate windows without having to mess with juggling state between multiple models and update paths.
Are there limitations in Avalonia itself which would make this annoying to implement?  Maybe things to do with threading?

Hey @keldor314 ,
thanks for the patience. That's a great question, let me get into the details a bit.

It occurred to me - why isn't there some sort of level in the UI tree above Window? This would make synchronizing multiple windows a lot less trouble. Something like
App [
    Window [ ]
    Window [ ]
    ...
]

There are a few thing worth considering that lead to my decision of rooting the MVU App on the window level.
App [
    Window [ ]
    if x then Window [ ]
    if y then Window [ ]
    Window [ ]
]

The example above could lead to serious performance issues as adding a window in between would effectively recreate and render all windows below.
It also means we do a lot more work on every view update as we end calculating the diff for all windows.
We would need some concept of a keyed list (interesting for other cases as well) to make this more efficient.

Alternately, the App level would have a single child which would be some collection of Windows. It might also make sense to instead of App call the root Context, since maybe there's a reason why the developer would want more than one of them.

It's a bit more complicated even as windows only exist on the desktop and avalonia is heading towards mobile support.

My use case is that I want a UI that's friendly with multiple monitors. Something like Visual Studio's dock system seems like a good place to start, and it would be nice to be able to easily break toolbars and such off into separate windows without having to mess with juggling state between multiple models and update paths.

Understood. I think for your scenario you might want to split the app into smaller components that then communicate with a central dispatch loop. FuncUI does not have good support for components yet, but it should be possible to create that yourself.

Are there limitations in Avalonia itself which would make this annoying to implement?
There might - It has been a while since I looked at it. Not sure if a window inherits from control - that's currently required for diffing. (But we could change that as it's an artificial constraint - there even is a draft PR for that)

Maybe things to do with threading?
Hmm, I don't think threading is an issue here.

Hrmm...  Let's see if I can clarify my requirements.
I'm actually coding a fractal editor, where the toolbars control parameters of the fractal.  These are flame fractals, and have a huge number of various parameters controlling different aspects of the rendering, so there are multiple tool components, to edit the coloration, the parameters for the IFS, animation, etc.  However, everything ultimately is bound to the same model, or at the very least, subsets of this model.
So I guess what I want is for the view to be virtualized, where the root can be as broad or narrow as needed, whether we're talking about a high level view spanning multiple windows or confined to the content of a panel somewhere.
Another option is to allow a view to be bound to multiple models.  For instance, it might have an internal state, but also drive a live preview through a shared external model, and ultimately commit changes to the global document model when the user hits "apply".  This also requires that a model be bound to multiple views.  And of course, the bindings need to be dynamic, where an entirely new view might be created when the user opens a dialog, etc.
Concerning mobile, I don't really see a problem.  The highest level context is simply restricted to having a single "window" control.  It's not a good idea to try to reuse a GUI between desktop and mobile anyway since the form factor is so different.

I can't speak for the internals but for the time being, I'll share my experience with that
I experimented with multiple windows a few months ago, but it's kind of clunky what I did was the following

Create a class which has a CLR Event
Subscribe to that event from my multiple windows (the windows are created from the main one)
Provide the an interface to your update function that can call a service

let update  (msg : Msg) (state: State) (windowService: IMyWindowService) =
       match msg with
       |  OpenWindow ->
            state, Cmd.ofAsync.perform dialogService.OpenWindow ()

each window will have it's own Elmish loop so you can subscribe to your service's events
not sure if that helps

the windows would look like this on the constructor
https://github.com/AvaloniaCommunity/Avalonia.FuncUI/blob/master/src/Examples/Examples.MusicPlayer/Shell.fs#L201-L212

@keldor314 I have a sample for you that might work
https://github.com/AngelMunoz/MultiWindow
in that example I open each window as a Dialog but you can new the windows and show them/hide them based on the commands/subscriptions you may have based on that sample
I don't think (given the nature of Avalonia itself, Avalonia doesn't have a window tree either AFAIK) this is something possible for the moment
Please check the sample if you're interested in this and let me know if there are some questions to that approach I'll close this in the meanwhile

@keldor314 I have a sample for you that might work
https://github.com/AngelMunoz/MultiWindow
in that example I open each window as a Dialog but you can new the windows and show them/hide them based on the commands/subscriptions you may have based on that sample
I don't think (given the nature of Avalonia itself, Avalonia doesn't have a window tree either AFAIK) this is something possible for the moment
Please check the sample if you're interested in this and let me know if there are some questions to that approach I'll close this in the meanwhile


Typos

wich -> which

Coverage remained the same at 92.065% when pulling f60e85058a17b3158a15cff47f3e1afa15310964 on led:patch-2 into 1f34cd7c82b6f5e7d0a08353df801e5686984cdb on jorgebastida:0.9.


Docker build is broken

Following the changes in https://github.com/docker-library/mongo/commit/791d400b9e84a298356b78db01956d98db973b9d, building the Docker image for MongoDB 4.4 doesn't work because Error response from daemon: Dockerfile parse error line 83: ARG requires exactly one argument.
This should be corrected, at least in https://github.com/docker-library/mongo/commit/791d400b9e84a298356b78db01956d98db973b9d#diff-64bdf01b91b9801069c15f8fd273c15306316bc74320059836a839d62307cfae.

Can you provide some more detail?  I can't reproduce: :confused:
$ docker build --pull https://github.com/docker-library/mongo.git#:4.4
Sending build context to Docker daemon  29.18kB
Step 1/20 : FROM ubuntu:bionic
bionic: Pulling from library/ubuntu
Digest: sha256:538529c9d229fb55f50e6746b119e899775205d62c0fc1b7e679b30d02ecb6e8
Status: Image is up to date for ubuntu:bionic
 ---> 4eb8f7c43909
...
Step 9/20 : ARG MONGO_PACKAGE=mongodb-org MONGO_REPO=repo.mongodb.org
 ---> Running in 8331bc5611ab
Removing intermediate container 8331bc5611ab
 ---> 897879f53032
Step 10/20 : ENV MONGO_PACKAGE=${MONGO_PACKAGE} MONGO_REPO=${MONGO_REPO}
 ---> Running in 3bcec62aa4ba
Removing intermediate container 3bcec62aa4ba
 ---> 6789f9ce0727
Step 11/20 : ENV MONGO_MAJOR 4.4
 ---> Running in 841b2982cdda
Removing intermediate container 841b2982cdda
 ---> 9c3041177f12
Step 12/20 : RUN echo "deb http://$MONGO_REPO/apt/ubuntu bionic/${MONGO_PACKAGE%-unstable}/$MONGO_MAJOR multiverse" | tee "/etc/apt/sources.list.d/${MONGO_PACKAGE%-unstable}.list"
 ---> Running in e5687ba0221b
deb http://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.4 multiverse
Removing intermediate container e5687ba0221b
 ---> 7b31d6744be8
...
 ---> 66ede2bb496e
Step 20/20 : CMD ["mongod"]
 ---> Running in c92f0c962b78
Removing intermediate container c92f0c962b78
 ---> d6341af03a5c
Successfully built d6341af03a5c

Well, it's weird that it works for you because clearly from https://github.com/docker-library/mongo/blob/6377a52b25e9b7790c7082fbdc3f587069255d27/4.4/Dockerfile#L83, there are 2 ARG on the same line and that doesn't work on my side. From the Docker documentation https://docs.docker.com/engine/reference/builder/#arg, it seems that ARG is only one value at a time.

What's your Docker version?

What's your Docker version?

19.03.8, not the newest I agree. But not that old :smiley:

This seems to only be a limitation when using older buildkit for docker build; regular docker build works fine and a newer buildkit also works fine (https://github.com/moby/buildkit/pull/1692).


[NETBEANS-54] Module Review #136

Update license to apache to file libs.jaxb/src/org/netbeans/libs/jaxb/Bundle.properties

Based on your valuable comments on PR#257, I reviewed the license for white spaces and it is ok but I changed the commit message to be "[NETBEANS-54] Module Review libs.jaxb"

Please let me know if it accepted so I can resolve more issues same way :-)

Did slight change to match exactly the existing! with the same single commit and agreed comment format.


It's possible to select zero encodings

I can Cmd+click the selected other encoding and then there are no encodings selected (but it's still using the one you last selected)

Are you talking about the Choose Encoding window below?

If so, to confirm, you are unselecting the currently selected option, leaving nothing selected? Is the expected outcome, to not allow such de-selection?

If so, to confirm, you are unselecting the currently selected option, leaving nothing selected?
Yes.

Is the expected outcome, to not allow such de-selection?
Either that, or if none are selected default back to ASCII?

OK, I'm going to get this done. @ridiculousfish any preference on the expected outcome on deselection? or should I just not allow it to happen?

OK, there's a fix at Pull Request #126
Defaults to MacRoman encoding, just like HexFiend does on startup.

Since the search can change the listing, there needs to be support for no selection. However I don't know if deselecting should really change anything in this situation. I feel like only a new selection should make a change.
Another idea is the window could have a field that shows the current encoding, and a button to Apply, but that's Windows behavior :)

My main issue was that it wasted my time trying to figure out the state of the app. So anything that plugs holes in the UI gets my vote.

Maybe we just copy what TextEdit does where the encodings window is essentially a "favorites" window:

I'm going to close this. I don't think it's a significant issue. If it becomes an issue for others in the future we can revisit, or if someone wants to implement the suggested change above feel free.

I respectfully disagree it's not an issue, given that I wasted a good amount of time figuring out the current hidden behaviour. IMHO


In build_browser_hub, give different colours to distinguish seq types, instead of creating separated tracks

A good option might be to add to build_browser_hub the color_column parameter from maf_to_custom_track. In this way, the user can specify what they want to distinguish.

Here are all the scenarios regarding bedToBigBed_path setup.
local_web_host_dir = "~/repos/LLMPP"
hub_dir = "hubs/ashm_test"

my_meta = get_gambl_metadata(seq_type_filter = c("genome", "capture")) %>%
  filter(pathology %in% c("BL"))

# bedToBigBed_path is missing and there is a config.yml file
setwd("~/repos/gamblr_repos/GAMBLR.results")
build_browser_hub(
  these_samples_metadata = my_meta,
  these_seq_types = c("genome", "capture"),
  projection = "hg38",
  local_web_host_dir = local_web_host_dir,
  hub_dir = paste0(hub_dir, "_hg38"),
  splitColumnName = "pathology",
  longLabel = "Public aSHM mutations separated by pathologies",
  contact_email =<EMAIL_ADDRESS>)
# it works fine

# bedToBigBed_path is missing and there is NO config.yml file
setwd("~/repos/gamblr_repos/GAMBLR.utils")
build_browser_hub(
  these_samples_metadata = my_meta,
  these_seq_types = c("genome", "capture"),
  projection = "hg38",
  local_web_host_dir = local_web_host_dir,
  hub_dir = paste0(hub_dir, "_hg38"),
  splitColumnName = "pathology",
  longLabel = "Public aSHM mutations separated by pathologies",
  contact_email =<EMAIL_ADDRESS>)
# Error: Since a `bedToBigBed_path` wasn't provided, `build_browser_hub` tries to use a config.yml file to get the bedToBigBed path. However...
# Error in config::get("dependencies"): Config file config.yml not found in current working directory or parent directories

# bedToBigBed_path is provided and it's valid
setwd("~/repos/gamblr_repos/GAMBLR.utils")
build_browser_hub(
  these_samples_metadata = my_meta,
  these_seq_types = c("genome", "capture"),
  projection = "hg38",
  local_web_host_dir = local_web_host_dir,
  hub_dir = paste0(hub_dir, "_hg38"),
  splitColumnName = "pathology",
  longLabel = "Public aSHM mutations separated by pathologies",
  contact_email =<EMAIL_ADDRESS>  bedToBigBed_path = "/projects/rmorin_scratch/conda_environments/ucsc_utils/bin/bedToBigBed"
)
# it works fine

# bedToBigBed_path is provided and it's valid
setwd("~/repos/gamblr_repos/GAMBLR.utils")
build_browser_hub(
  these_samples_metadata = my_meta,
  these_seq_types = c("genome", "capture"),
  projection = "hg38",
  local_web_host_dir = local_web_host_dir,
  hub_dir = paste0(hub_dir, "_hg38"),
  splitColumnName = "pathology",
  longLabel = "Public aSHM mutations separated by pathologies",
  contact_email =<EMAIL_ADDRESS>  bedToBigBed_path = "/an/invalid/path"
)
# Error in build_browser_hub(these_samples_metadata = my_meta, these_seq_types = c("genome",  :
#  `bedToBigBed_path` points to a non-existent file.


Discern 'Dave_scores' better

Maybe add a column for whether or not the restaurant has a 'dave_score'

Could also be done in Tableau but it might make more sense to do it in the csv


Ability to create .psd files

Hey @Gofven , the PhotoshopAPI fully supports both psd and psb file formats. It will usually just deduce the file format based on what extension you pass to the LayeredFile.write method. Are you using python or c++?

Waah i accidentally pressed enter or an uncomplete issue, my bad! But i do indeed have a certain issue with it (o also thanks for the blazing fast response!!)
I'm attempting to create an ImageLayer_8bit layer and add it to a new LayeredFile_8bit instance, then write the data to a directory. When attempting this operation, i do end up with a successful export, however the psd file is completely unreadable when i'm attempting to use Krita to read it.
Do you have any suggestions to why this may occur c:? I do have a rather messy file that i've been attempting to fix by trial and error (before i start cleaning it up), but to no avail, here it is if you'd like to see more on the issue: https://www.toptal.com/developers/hastebin

No worries, its definitely happened to me too ;). Im not too familiar with krita but id be happy to look into it. I've so far not run into any issues with psds/psbs written by the PhotoshopAPI (except for files with only groups) but the format is a fiddly one and there is many exceptions.
I think you accidentally sent an empty pastebin but I'll take a look once you've sent it! Since krita is open source I can also then just hook up a debugger to it and tell exactly whats wrong

my bad, here's the URL!
https://hastebin.com/share/huvuroveyo.kotlin

Thanks for the link! I'll try running it once back home and then seeing if photoshop can open the imagefile and if its krita specific or psapi but from a first glance it looks like it should work.
Fyi youre probably better off using numpy slicing operations for getting the image data in the right format rather than looping over the images pixel by pixel :)

No problem c:
Also yes going pixel by pixel wasn't the greatest solution, given how slow it is. I read on the numpy docs and decided to use swapaxes, just the line image_data.swapaxes(0, 2) and it's good to go c:
Although the same issue persist so i'm not quite sure whether the swap actually works or if i'm going to be presented by a glorious collection of mismatched pixels, either the case it runs!

Yeah swapaxes should work! Keep in mind that if youre running the release version of psapi you need to copy the array as swapaxes only changes the internal stride etc. not the actual data. See the discussion below on how to do that
https://github.com/EmilDohne/PhotoshopAPI/discussions/71
In the next version you should be able to see this behaviour change by the psapi making it contiguous internally :)

I've figured it out thanks to #71 you referenced c:
It's due to me not declaring the Color Mode, height, and width when creating a LayeredFile_8bit instance!
Thanks again for the help you provided!!

I'm very glad you were able to figure this out! I'll go ahead and close out this ticket but this actually made me figure out a part of the issue mentioned her https://github.com/EmilDohne/PhotoshopAPI/issues/65. Photoshop doesnt actually allow files smaller than 1 pixel which I've failed to catch. The color mode should default to RGB anyways so that shouldnt have been an issue :).
Making the size default initialize to 1 actually also allows one to visually debug the file easily rather than getting hit with some obscure error


Add cargo test in CI workflow, fix failing Rustdoc examples

Current CI workflow does not perform any testing on the codebase - which in fact does not include any unit or integration tests.
The lack of testing lowers the confidence users might have in the library, and makes upgrading dependencies complex as impact of such upgrades (ex: breaking changes) can't be identified easily.
This PR aims to try improving this, by having cargo test in the CI workflow, and fixing existing Rustdoc examples that are currently failing after such addition.

Adding cargo test to CI workflow and fixing existing Rustdoc examples have been performed in two commits not only as best practices to ease review, but also to highlight why fixing the Rustdoc examples was needed.
However, as workflow execution following PR requires approval, please use ppiotr3k/ybc in order to see the evolution of the CI status for each of those commits.


tt-rss-plugin-auth-ldap: fix compat with latest tt-rss

Motivation for this change
PHP8 broke the support...
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Relase notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 132329 at 88a326a9 run on x86_64-linux 1

1 package built successfully:

tt-rss-plugin-auth-ldap

2 suggestions:

warning: missing-patch-comment
Consider adding a comment explaining the purpose of this patch on the line preceeding.
Near pkgs/servers/tt-rss/plugin-auth-ldap/default.nix:15:5:
   |
15 |     (fetchpatch {
   |     ^

warning: missing-phase-hooks
installPhase should probably contain runHook preInstall and runHook postInstall.
Near pkgs/servers/tt-rss/plugin-auth-ldap/default.nix:26:3:
   |
26 |   installPhase = ''
   |   ^


Use UUID instead of name

Hello,
Could you use UUID instead of name for the blacklist command?
And if possible, just add the players to the users list only if needed. This would be helpful for servers with a lot of players.
Best regards,

Yeah i think it would be better with UUID, gonna switch asap. I'll see what i can do to optimize the users management system! Thank you for these advices!


Keys restore error - Invalid argument(s): Invalid or corrupted pad block

Steps to reproduce:

Install the app
Create keys and login
Reinstall the app (or clean app data)
Restore the keys created

The app loads the key from backup server but during key decryption in TikiKeysModel.decrypt(), it throws this error.

not a bug. this means the passphrase was wrong


feat(layers): Added the possibility to manually refresh a layer

Refresh the layer using the doRefresh property.
Commented by @pieterjandesmedt here:
https://github.com/tombatossals/angular-leaflet-directive/issues/730#issuecomment-95597177

LGTM

Hey guys, why do you set .doRefresh to false?
if (newBaseLayers[name].doRefresh) {
    newBaseLayers[name].doRefresh = false;
}

I thinks this disables continuous updates after the first update right?
Also, it breaks unit test layers.watch.markers should add and remove layers in watch in layersDirective.watch.markersSpec.js

Yess @pieterjandesmedt, that's because I understood that the "doRefresh" property is a one-time-use action, so from the controller you set this property and the directive processes it in the life-cycle, after processed it returns to the false value.
I opted for this approach because if you don't have an imperative switch to refresh,  when does the layer has to be refreshed? Every X seconds? If not set an interval, every digest cycle the layer would be deleted and re-created.
You're right about the tests, it needed an extra check, last commit solved it.
Maybe we could improve this. What do you think about broadcasting a message from controller to directive announcing the need to refresh the layer?

Aha, I see what you mean. I don't think broadcasting a message is necessary. The layers watch should pick it up when a controller or service sets doRefresh to true, right?

Yess, this would be the example, based on the one you provide:
http://tombatossals.github.io/angular-leaflet-directive/examples/0000-viewer.html#/layers/refresh-overlay-every-minute-example
The idea is that you set the layer property "doRefresh" to true, and immediatly this is processed in the directive watch, which re-creates the layer and sets the doRefresh property to false.

Ok, fine by me!


 or FirebaseRulesTokenType.) expected, got '['

Hello!
Thanks for the awesome plugin!
By the way I ran into a bug. I wish I could fix it myself, but...
In the meantime, I'll report back to you.
Environment
Version: 1.1.1-SNAPSHOT
Result
 or FirebaseRulesTokenType.) expected, got '['

rules_version = '2';

service cloud.firestore {
    match /databases/{database}/documents {
        function documentRef(paths) {
            return path([['databases', database, 'documents'].join('/'), paths.join('/')].join('/'));
        }
    }
}

Thanks for testing the plugin, the changes gonna be available for version: 1.1.2

The fix is too soon! Thank you!


feat(trace)!: persist the correct size of the txs

Currently, we use the length of the tx hash as the size. This means that all txs have the size 32. It would be more helpful that the tracing data captures the actual size of the transactions passing back and forth

I'd suggest changing the title of the PR to feat instead of fix, also as it updates the signature of WriteMempoolTx, we may want to indicate that it is a breaking change i.e., feat!

I'd suggest changing the title of the PR to feat instead of fix, also as it updates the signature of WriteMempoolTx, we may want to indicate that it is a breaking change i.e., feat!

I think it can be argued both ways as a fix or a feat. It's a fix because we were previously saving the size, it's just we were using the wrong size (the hash instead of the tx itself).


Snapshot testing and arm64 builds unavailable

Due to a sudden increase in renewal costs for our dedicated server, we’ve temporarily lost our private build agents. We’re exploring alternatives. If you’re interested in sponsoring compute, a single ARM64 machine with about 16GB RAM and 80 GB disk space would meet our needs. Thank you for your support.

arm64 is back.


Update rtc_io_periph.h (IDFGH-4374)

solves undefined GPIO_PIN_COUNT error when compiling ulp component.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

also components/soc/esp32/rtc_io_periph.c line 85 needs same:
const rtc_gpio_desc_t rtc_gpio_desc[SOC_GPIO_PIN_COUNT]
using: ESP-IDF v4.3-dev-1901-g178b122-dirty

Thanks for your contribution.

Duplicate of #6029, which still hasn't been merged in nearly 2 months.
I also found this, and fixed it a while back, but realized the duplicate... #6131

thanks, didn't see that!
I'm closing this one


Fix interaction between collapsed and muted messages

At present, muting and collapsing a message do not interact correctly. The desired behavior is:

If a message is muted, not expanded, and collapsed, it should just appear as muted (no visual changes in how it's displayed due to the collapsed state).
If a message is muted, expanded, and collapsed, it should appear as collapsed (it currently does not).

@zulipbot claim

@alya could you please be more specific about muting a message.I am not getting what does muting a message means.

@aparna-bhatt I’m working on this issue.

@zulipbot claim

@zulipbot abandon

@alya To make sure I understand the problem correctly, could you please provide a visual.
That would be helpful.
Thanks!

I recommend playing with muting and expand/collapse UI to understand the issue.

https://user-images.githubusercontent.com/86338542/223362744-d7fe42f2-0327-475d-9b2a-7381ddc8291f.mp4
@alya are you suggesting that the topic in the screen-cast after muted should collapse from the view ?
If it is so then what should appear in the UI?

Are you using the "collapse message" feature?

https://user-images.githubusercontent.com/86338542/223373727-a5d311ae-e56e-4d0a-ba0b-c20c088507f5.mp4
Are you suggesting that the message should not be expanded it should remain collapsed?

If a message is muted, expanded, and collapsed, it should appear as collapsed (it currently does not).

I think you're looking at a muted topic. You should be looking at messages from a muted user.

@zulipbot claim

@alya could you please confirm if the following GIF is the problem (1st point). We need to maintain the This message was hidden because you have muted the sender state, right?

@zulipbot claim

If a message is muted, expanded, and collapsed, it should appear as collapsed (it currently does not).

I think you're looking at a muted topic. You should be looking at messages from a muted user.

Sorry for late reply.
I will try to re-create as suggested.
Thank you.

@zulipbot claim


Configure hosts on project install

This is rather a minor issue, but maybe you find it useful.
I would like to suggest to add a certain setup process for each new project. For example, when you just cloned the development stack and run npm install, guide the developer through a similar process as when creating a new package.json (a list of entries awaiting for your input). This way we can allow to set a certain host alias for the project:
Enter a desired host alias (default: localhost): my-metro.dev
...
Your project is available under "http://my-metro.dev".

We may take use of shell commands and this (or similar) packages: Hostile. Script will run the following when any custom input is received:
hostile set localhost:7000 my-metro.dev

What do you think about it?

I would rather follow the KSS rule. We wan't to have this project easily maintainable. We will not cover all issues, we want to cover the most problematic parts.
What we definitely need to have is better documentation with "install/usage" section, with point for changing project’s :port number.

@janpanschab a good point. Working on the documentation right now. I will cover this aspect.

This question is no longer discussable, so I will close the issue.
Required information about changing the default port will be added in the documentation.


Fixfix some unfixed-answer test

Proposed changes
Issue Number: close #xxx
Problem summary
delete regression-test/suites/query_p0/sql_functions/math_functions/test_running_difference.sql
modify regression-test/suites/query_p0/sql_functions/table_function/explode_json_array.groovy
Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[x] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


Multithreaded rendering

Bungie has made a presentation on multithreaded rendering.
In their engine, the game simulation and rendering are done in parallel with a step of synchronization where a specific selection of data is copied, transformed and prepared for the next frame to be rendered. This step of synchronization is where no game logic nor rendering is running to avoid data race condition.

This would involve a lot of change in how the engine currently works (ECS, Renderer, maybe the asset system etc).

This is how I do multithreaded Bepu physics in Focus Engine, but haven't messed with updating game logic in parallel. What you can do, and can provide similar performance bonuses, is to write your game simulation logic to be multithreaded (e.g. update lots of in-game objects at once).

Interestingly, a lot of our rendering architecture is inspired from this exact talk.
We also have the Extract/Prepare/Render pipeline, with lot of inner parts happening in multithread.
We don't do the render frame - 1 thing yet (it's quite a lot of work to generalize it for a game engine -- possibly a lot of it would have to be done by the user/game anyway).

It does make sense now,

(it's quite a lot of work to generalize it for a game engine -- possibly a lot of it would have to be done by the user/game anyway)

I'm not sure i understand what you mean by that, what would make it complicated to be generalized ?

it would be nice if we can push game logic to second thread, so game won't freeze while you dragging window


ci: Update Commit used for the Upgrade tests for release-v0.34.x

Fixes #N/A
Description

Update commit for Upgrade suite

How was this change tested?
Does this change impact docs?

[ ] Yes, PR includes docs updates
[ ] Yes, issue opened: #
[x] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>0
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 77.922%

Totals

Change from base Build<PHONE_NUMBER>3:
0.0%

Covered Lines:
5446

Relevant Lines:
6989

💛  - Coveralls


Validate URLs for file transfer

Make sure the user's URL input is a URL that points to a file
Alert the user if it's not

Potentially make sure that the file it points to is not a static html file


Issue 883

The current unit tests plays the receive-bitcoin sound

I agree that the pull request code to be released under the MIT licence and does not contain any copyrighted material that I do not own.

The client plays the receive-bitcoin sound when Play sound button is clicked (Preferences->Sounds)

Thanks, Jonny!

You are welcome


SoftAP vs BLE provisioning (MEGH-3697)

What to consider when choosing?
Is any of the methods better? Or depends from something?
Using ESP32C3

@jacek12345 , generally we recommend using BLE so that the phone does not have to disconnect from its existing Wi-Fi network just for provisioning. Moreover, we have noticed that some phones automatically disconnect from the SoftAP interface since that network does not have Internet.

Thank You for response.


Switch to .Net Global Tool for JetBrains tools

As per docs here:
https://www.jetbrains.com/help/resharper/ReSharper_Command_Line_Tools.html#install-and-use-resharper-command-line-tools-as-net-core-tools

This may require upstream changes in Cake to support.
If I remember correctly, DupFinder and InspectCode is provided from Cake itself and not an addin.
Alternatively, an addin may perhaps be created to consolidate the calling since it will need to call the same executable every time (not completely sure about this one).

Just to follow up with the above...
In the .Net Global Tool version of the JetBrains tools, there is a new command jb which will then invoke either jb dupfinder or jb inspectcode.  So it isn't a straight call to dupfinder or inspectcode, those actually need to be provided as an argument to the jb command.
Off the top of my head, I am not sure how best to handle this...

Re-opening this issue for now, since there are problem running the global tool on Linux, and also the suggestions is to create a new Cake.JetBrains addin, rather than trying to force the existing DupFinder and InspectCode aliases to use the new global tool.

For reference: here is the issue regarding DupFinder
Warning: While cataloging the assembly “JetBrains.Platform.MsBuildAutoDetectTask, Version=7<IP_ADDRESS>, Culture=neutral, PublicKeyToken=1010a0d8d6380325”, could not load the assembly reference named “Microsoft.Build.Framework, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”.
Warning: While cataloging the assembly “JetBrains.Platform.MsBuildAutoDetectTask, Version=7<IP_ADDRESS>, Culture=neutral, PublicKeyToken=1010a0d8d6380325”, could not load the assembly reference named “Microsoft.Build, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”.
Warning: While cataloging the assembly “JetBrains.Platform.MsBuildTask, Version=7<IP_ADDRESS>, Culture=neutral, PublicKeyToken=1010a0d8d6380325”, could not load the assembly reference named “Microsoft.Build, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”.
Warning: While cataloging the assembly “JetBrains.Platform.MsBuildTask, Version=7<IP_ADDRESS>, Culture=neutral, PublicKeyToken=1010a0d8d6380325”, could not load the assembly reference named “Microsoft.Build.Framework, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”.
Warning: While cataloging the assembly “JetBrains.ReSharper.Features.Internal, Version=7<IP_ADDRESS>, Culture=neutral, PublicKeyToken=1010a0d8d6380325”, could not load the assembly reference named “Microsoft.Build, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”.
Warning: While cataloging the assembly “JetBrains.CommandLine.Products, Version=7<IP_ADDRESS>, Culture=neutral, PublicKeyToken=1010a0d8d6380325”, could not load the assembly reference named “Microsoft.Build, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”.
Warning: While cataloging the assembly “JetBrains.CommandLine.InspectCode.MsBuild, Version=7<IP_ADDRESS>, Culture=neutral, PublicKeyToken=1010a0d8d6380325”, could not load the assembly reference named “Microsoft.Build.Framework, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”.
JetBrains Duplicates Finder 2020.2.2
Running in 64-bit mode, .NET runtime 3.1.8 under Unix <IP_ADDRESS>5
Using toolset version 16.0 from /usr/share/dotnet/sdk/3.1.402
Collecting files to analyze:
  from project '/home/runner/work/Cake.Transifex/Cake.Transifex/Cake.Transifex.sln'.
23 files found to analyze.
Duplicates report was written to /tmp/tmpGcXl2k.tmp
An error occurred when executing task 'DupFinder'.

https://github.com/cake-contrib/Cake.Transifex/runs/1159059895?check_suite_focus=true#step:6:1113
There are a few possible problems that cause the above error

There is a mistake from the developers, and they have marked the dependencies as private
The executable must be executed with dotnet jb instead of dotnet-jb. This means it needs to be installed as a proper tool, and not to the tools directory (which causes cake to not find it :( )
There is a hard requirement on the .NET Core runtime being used (unlikely)


Rare ERR_BUFFER_OUT_OF_BOUNDS

Describe the bug
During chaos testing on latest master was able to trigger
debug: {"timestamp":"2020-04-30T00:19:02.499Z","logger":"kafkajs","message":"consumer has stopped, disconnecting","groupId":"connector-common-integration-play-v2"}
debug: {"timestamp":"2020-04-30T00:19:03.049Z","logger":"kafkajs","message":"Fetching from 12 partitions for 2 out of 2 topics","topics":["some-topic","some-topic-2"],"activeTopicPartitions":[{"topic":"some-topic","partitions":[0,10,11,2,4,7]},{"topic":"some-topic-2","partitions":[0,10,11,4,7,8]}],"pausedTopicPartitions":[]}
debug: {"timestamp":"2020-04-30T00:19:03.049Z","logger":"kafkajs","message":"Connecting","broker":"xyz:9092","clientId":"kafkajs-50270","ssl":false,"sasl":false}
debug: {"timestamp":"2020-04-30T00:19:03.049Z","logger":"kafkajs","message":"Connecting","broker":"xyz:9092","clientId":"kafkajs-50270","ssl":false,"sasl":false}
debug: {"timestamp":"2020-04-30T00:19:03.049Z","logger":"kafkajs","message":"Connecting","broker":"xyz:9092","clientId":"kafkajs-50270","ssl":false,"sasl":false}
debug: {"timestamp":"2020-04-30T00:19:05.795Z","logger":"kafkajs","message":"Request Fetch(key: 1, version: 7)","broker":"xyz:9092","clientId":"kafkajs-50270","correlationId":12,"expectResponse":true,"size":187}
warn: {"timestamp":"2020-04-30T00:19:06.151Z","logger":"kafkajs","message":"Response without match","clientId":"kafkajs-50270","broker":"xyz:9092","correlationId":7}
debug: {"timestamp":"2020-04-30T00:19:06.193Z","logger":"kafkajs","message":"Request Fetch(key: 1, version: 7)","broker":"xyz:9092","clientId":"kafkajs-50270","correlationId":10,"expectResponse":true,"size":235}
debug: {"timestamp":"2020-04-30T00:19:06.193Z","logger":"kafkajs","message":"Request Fetch(key: 1, version: 7)","broker":"xyz:9092","clientId":"kafkajs-50270","correlationId":83,"expectResponse":true,"size":211}
Consumed 0
warn: {"timestamp":"2020-04-30T00:19:06.800Z","logger":"kafkajs","message":"Response without match","clientId":"kafkajs-50270","broker":"xyz:9092","correlationId":5}
RangeError [ERR_BUFFER_OUT_OF_BOUNDS]: Attempt to write outside buffer bounds
    at boundsError (internal/buffer.js:47:11)
    at Buffer.readInt32BE (internal/buffer.js:361:5)
    at Decoder.readInt32 (/Users/dmitrii/Desktop/Projects/at_connector_common/node_modules/kafkajs/src/protocol/decoder.js:50:31)
    at Connection.processData (/Users/dmitrii/Desktop/Projects/at_connector_common/node_modules/kafkajs/src/network/connection.js:410:38)
    at Socket.onData (/Users/dmitrii/Desktop/Projects/at_connector_common/node_modules/kafkajs/src/network/connection.js:121:14)
    at Socket.emit (events.js:198:13)
    at Socket.EventEmitter.emit (domain.js:448:20)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at Socket.Readable.push (_stream_readable.js:224:10)

Node process just died after that error
To Reproduce
Unfortunately i have encounter this particular error only once and it is hard to trigger again, running more investigations at the moment...
Expected behavior
Do not have this error
Environment:

OS:  Mac OS 10.15.3
KafkaJS version: latest master branch
Kafka version: 2.3.1
NodeJS version: 10.16.3

I have also come across this error multiple times and have had a hard time pinning it down
Environment:
Centos: 7.6.x
Kafka Versions: 2.3.1, 2.4.0, 2.4.1, 2.5
Node JS Version:  v10.18.0
KafkaJS version: 1.12.0
Different scenarios
I have run kafka servers in a cluster, 3 zookeepers and 3 kafka instances, all on different servers with replication factor of 3
I have run one single zookeeper and kafka server.
I have run consumer groups with 1 topic, 3 partitions and processed almost 3 Billion records over 2 weeks with no issue. 3 separate instances of the program
The latest run, I had 3 consumer groups, consuming 3 separate topics which all have 1 partition each. One of the programs hit the error below after about 5.5million messages consumed. The other 2 are still running with no issue
I thought maybe it was related to load on the kafka server, which caused a hiccup. But running 3 instances on the same groupId should produce the same load of total messages in/out.
My suggestion would be to change the bufferId to 64bit instead of 32 since that is what kafka is trying to give it back at boundsError (internal/buffer.js:47:11)\n    at Buffer.readInt32BE (internal/buffer.js:361:5)\n
I am going to try upgrading to the latest version of node and seeing if that helps
Specific Error message
{"level":"ERROR","timestamp":"2020-05-10T19:13:00.059Z","logger":"kafkajs","message":"[Connection] Response Fetch(key: 1, version: 7)","broker":"ip:9092","clientId":"kafkajs","error":"Attempt to write outside buffer bounds","correlationId":8,"size":1048655}
{"level":"ERROR","timestamp":"2020-05-10T19:13:00.060Z","logger":"kafkajs","message":"[Consumer] Crash: KafkaJSNumberOfRetriesExceeded: Attempt to write outside buffer bounds","groupId":"groupId","retryCount":3,"stack":"KafkaJSNonRetriableError\n  Caused by: RangeError [ERR_BUFFER_OUT_OF_BOUNDS]: Attempt to write outside buffer bounds\n    at boundsError (internal/buffer.js:47:11)\n    at Buffer.readInt32BE (internal/buffer.js:361:5)\n    at Decoder.readInt32 (/node_modules/kafkajs/src/protocol/decoder.js:50:31)\n    at module.exports (/node_modules/kafkajs/src/protocol/recordBatch/v0/decoder.js:41:40)\n    at decodeMessages (/node_modules/kafkajs/src/protocol/requests/fetch/v4/decodeMessages.js:27:35)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"}
{"level":"ERROR","timestamp":"2020-05-10T19:13:00.067Z","logger":"kafkajs","message":"[Consumer] Restarting the consumer in 730ms","retryCount":3,"retryTime":730,"groupId":"groupId"}
The kafka server produced this log
[2020-05-10 17:42:42,842] INFO [ProducerStateManager partition=groupId-0] Writing producer snapshot at offset 4922471 (kafka.log.ProducerStateManager)
[2020-05-10 17:42:42,843] INFO [Log partition=groupId-0, dir=/opt/kafka/datafiles/kafka-logs-1] Rolled new log segment at offset 4922471 in 10 ms. (kafka.log.Log)
[2020-05-10 17:48:40,019] INFO [GroupCoordinator 1]: Member[group.instance.id None, member.id kafkajs-6202ff7b-8d95-4dc3-9828-2ba0dd802078] in group groupId has left, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2020-05-10 17:48:40,019] INFO [GroupCoordinator 1]: Preparing to rebalance group groupId in state PreparingRebalance with old generation 1 (__consumer_offsets-36) (reason: removing member kafkajs-6202ff7b-8d95-4dc3-9828-2ba0dd802078 on LeaveGroup) (kafka.coordinator.group.GroupCoordinator)
[2020-05-10 17:48:40,019] INFO [GroupCoordinator 1]: Group groupId with generation 2 is now empty (__consumer_offsets-36) (kafka.coordinator.group.GroupCoordinator)
[2020-05-10 17:48:40,877] INFO [GroupCoordinator 1]: Dynamic Member with unknown member id joins group groupid in Empty state. Created a new member id kafkajs-6d50c1c3-d0ac-4061-a30f-769f8fac3ef4 for this member and add to the group. (kafka.coordinator.group.GroupCoordinator)

It looks like your two issues are actually unrelated. The first one is coming from the connection (before we even get into parsing the actual message protocol), when we try to read the correlation id from the response. The second issue is when trying to decode a RecordBatch.
For the second issue, there's some really tricky situations involving partial RecordBatches, or batches that contain both the older MessageSet and the newer RecordBatch (or even partials of them both). My unsubstantiated guess at this point would be that this is happening due to a version upgrade at some point. We encountered that a few years ago, and fixed the issue, but perhaps there's still some permutation of it that happens in some rare circumstance. Without having access to the actual buffer that couldn't be parsed, it's really hard to know.
What we did when this happened was to log the current offset of the consumer (and topic and partition) and then run a separate consumer that only consumed that particular offset and logged the fetch buffer by running with extreme debug logging turned on, KAFKAJS_LOG_LEVEL=debug KAFKAJS_DEBUG_PROTOCOL_BUFFERS=1 KAFKAJS_DEBUG_EXTENDED_PROTOCOL_BUFFERS=1. That way we could get the exact buffer that couldn't be parsed and try to debug the issue using that.
Of course, the fetch buffer will contain whatever data it is you're consuming, so you may or may not be able to share that, depending on how sensitive the data is. If you can't share it, even privately, there's unfortunately not much we can do.

Thanks @Nevon for the follow up .  A few follow up questions or comments

I was able to replicate this fairly consistently by putting my kafka servers into a cluster but on separate servers.  I ended up breaking my cluster, re-installing everything, and just running on a single node which seemed to stabilize the environment for now.

When you say record batches, I normally send 100 to 1000 records in a send call. Do you think changing that to 1 could prevent this issue? I am ok with a bit lesser performance for more reliability and would like to get back to a cluster prior to production

I will follow your suggestion about turning logging up once I hit the issue. my question is, do I need to start a new consumer from that offset? Or should I be able to restart my existing one since it should pick up where it left off with more logging. I should be able to share this data as it isn't super sensitive but will do it over DM

I received the failing fetch response in a private message and was debugging it during the weekend. Documenting it here for posterity.
The request is a Fetch V7, which uses the V4 decodeMessages, as the recordbatch format hasn't changed. V4 decodeMessages in turn uses the RecordBatchDecoder v0. The recordbatch decoding is pretty straight forward, following the protocol specification, where it first reads the base offset followed by a byte length for the batch, and then creates a new decoder by taking the length value it read.
In this case, the fetch response consists of two record sets. The first one is decoded just fine, and initially the second one is also decoded without any problem. However, at a certain point, when we read the length of the record batch we get back 0, so we end up creating a buffer of length 0 that we then try to read from. It's not a partial record set, which we've seen in the past, because the actual length of the record set buffer is correct according to the length read from the record set protocol, it's just this particular record batch in the record set that according to the protocol has a length of 0.
I considered if maybe it's possible that the protocol allows for empty record batches, although it doesn't make sense, so I tried treating it as a partial record batch and bailing out if that's the case, but it doesn't feel right since we only got through like half of the record set:
console.log src/protocol/decoder.js:94
    Determining if we can read another record batch {
    bufferByteLength: 1048576,
    offset: 485305,
    length: 49,
    canRead: true
    }
console.log src/protocol/requests/fetch/v4/decodeMessages.js:30
    Delegating to RecordBatchDecoder { recordsSoFar: 1981, magicByte: 2 }
console.log src/protocol/recordBatch/v0/decoder.js:33
    Starting to decode record batch {
    firstOffset: '2756082',
    length: 8761,
    decoderLength: 8761,
    fetchDecoderLength: 554498
    }
console.log src/protocol/recordBatch/v0/decoder.js:104
    Reading 36 records from RecordBatch
console.log src/protocol/decoder.js:94
    Determining if we can read another record batch {
    bufferByteLength: 1048576,
    offset: 494078,
    length: 49,
    canRead: true
    }
console.log src/protocol/requests/fetch/v4/decodeMessages.js:30
    Delegating to RecordBatchDecoder { recordsSoFar: 2017, magicByte: 2 }
console.log src/protocol/recordBatch/v0/decoder.js:33
    Starting to decode record batch {
    firstOffset: '0',
    length: 0,
    decoderLength: 0,
    fetchDecoderLength: 554486
    }
console.log src/protocol/recordBatch/v0/decoder.js:43
    Received RecordBatch with 0 length

A next step might be to try to manually inspect the response from the offending offset and see if it makes any kind of sense. It would also be interesting to try to decode it using some other client and see what comes out the other end.

@Nevon It had been a few months. Was just checking in to see if this issue had been looked at or was on the list some where. Overall, the library is great, so this is the last remaining item that was on the list since I am running a single Kafka server as a stand alone.
Thanks!

Hey @Nevon, I was able to reproduce the issue, but I'm new to Kafka and I'm not sure if that's just me doing something wrong. In a nutshell, whenever I try to consume from a topic with some unprocessed messages (current offset < log-end offset) - consumer throws RangeError [ERR_BUFFER_OUT_OF_BOUNDS]: Attempt to access memory outside buffer bounds.
Steps to reproduce:

brew install kafka, which got me 3.2.1
zookeeper-server-start /opt/homebrew/etc/zookeeper/zoo.cfg
kafka-server-start /opt/homebrew/etc/kafka/server.properties
kafka-topics --boostrap-server localhost:9092 --create --topic nutopic
node producer.js
node consumer.js

It works fine, however, when consumer is already running before I produce a message. I can also use seek to go to the last message offset and consume new messages in the topic, but I kinda need the old ones too 😅
Please, let me know if you need any more details 🙏
Code
producer.js
const { Kafka } = require('kafkajs');

const kafka = new Kafka({ brokers: ['localhost:9092'] });

const producer = kafka.producer();

async function produce() {
  await producer.connect();

  await producer.send({
    topic: 'nutopic',
    messages: [{ value: '123' }]
  });

  process.exit();
}

produce();

consumer.js
const { Kafka } = require('kafkajs');

const kafka = new Kafka({ brokers: ['localhost:9092'] });

const consumer = kafka.consumer({
  groupId: 'js-consumer',
  sessionTimeout: 30000
});

async function consume() {
  await consumer.connect();
  await consumer.subscribe({ topic: 'nutopic' });

  await consumer.run({
    eachBatch: async payload => {
      console.log(payload.batch.messages.length);
    }
  });
}

consume();

I randomly started getting this error again after a few years. But it was after rolling onto a new kafka server and server where the consumer runs. Both these servers were VMs like the rest, but these were completely under powered (think $40 dollar servers) and using network attached storage. My gut feeling is that there is some kind of race condition at high transaction volume and the network latency was too high.
Upgrading both servers to more dedicated boxes with "direct" attached storage seems to have resolved the issue.


Add note about the official upstream wrapper

@fasterthanlime, I've only just noticed that there is now an official wrapper.
As far as I can see, it doesn't provide the custom dictionary support as contributed by @millisecond, but it seems we are otherwise duplicating effort.
What are your thoughts - does this project still add value for statically linking the brotli implementation?

What are your thoughts - does this project still add value for statically linking the brotli implementation?

That's a lot of value to me (and what I would expect go packages to do), but I obviously can't force you to keep maintaining this package if you don't want to :)

I don't mind maintaining it if people will use it :) I also feel that it doesn't quite fit the Go way of doing things to dynamically link.

I also feel that it doesn't quite fit the Go way of doing things to dynamically link.

Yup I feel the same way, I'm kinda disappointing that that's what the "official bindings" do. We've been using kothar/brotli-go in production for a while and have no plans to change that, so I very much appreciate your continued work on it :)


SyntaxError: Unexpected end of JSON input

Hi
I want to remove all the properties from my geojson file so for that I first try to make a njson file like that:
/root/npm/bin/ndjson-split 'd.features'   < test.json   > test.ndjson
but I get this error:
stdin:1
{"type":"FeatureCollection","features":[
^
SyntaxError: Unexpected end of JSON input

My geojson is a valid one, it's just here: https://gist.github.com/2803media/b52c9078c6c7932396ce6b42d7f0073e
Thanks

After digging, it seems that if the geojson had some space or line break it will failed the ndjson-split function.
The walkthrough to debug was:
-to convert the geojson to topojson
-convert topojson to geojson
-then do ndjson-split
I don't know if it's possible to integrate this point in ndjson-split

ndjson-split expects NDJSON input, not JSON. You can use ndjson-cat to convert JSON into NDJSON (i.e., to remove newlines).
ndjson-cat test.json | ndjson-split 'd.features' > test.ndjson


feat(x86_64::paging): Implement paging & kernel remap

This PR adds a working implementation of paging and remapping the kernel into the high address space on x86_64! 🍾
Note that interrupts had to be re-disabled temporarily because of issues from #78 that I'm still looking into.
This PR also includes a number of changes to the elf and alloc crates that were necessary to get paging working correctly. It doesn't finish the whole "allocator rewrite" project, but it brings us further along.
Paging issues:

Closes #18
Closes #19
Closes #20

Allocator issues:

Closes #64
Closes #65
Closes #66
Closes #72

Also makes some progress on issues #31, #73, #76, and #84.

This should probably increment the version numbers on alloc, paging, memory, elf, and the main kernel. Possibly also cpu -- I should see if any API changes in there are permanent.

Note that this PR adds the paging and params crates, which will add more crates to the task lists on #86  and #102


[SPARK-36424][SQL] Support eliminate limits in AQE Optimizer

What changes were proposed in this pull request?

override the maxRows method in LogicalQueryStage
add rule EliminateLimits in AQEOptimizer

Why are the changes needed?

In Ad-hoc scenario, we always add limit for the query if user have no special limit value, but not all limit is nesessary.
With the power of AQE, we can eliminate limits using running statistics.
Does this PR introduce any user-facing change?

no
How was this patch tested?

add test

Test build #142082 has started for PR 33651 at commit 4299af5.

cc @maropu  @cloud-fan @viirya @HyukjinKwon

Test build #142084 has started for PR 33651 at commit c076d76.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46592/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46595/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46592/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46592/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46595/

Test build #142082 has finished for PR 33651 at commit 4299af5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait EliminateLimitsBase extends Rule[LogicalPlan]

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46595/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142082/

Test build #142084 has finished for PR 33651 at commit c076d76.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait EliminateLimitsBase extends Rule[LogicalPlan]

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142084/

Test build #142101 has started for PR 33651 at commit 8aae958.

Test build #142102 has started for PR 33651 at commit 4510816.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46613/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46613/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46614/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46614/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46614/

Test build #142101 has finished for PR 33651 at commit 8aae958.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142101/

Test build #142102 has finished for PR 33651 at commit 4510816.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142102/

Test build #142117 has started for PR 33651 at commit ba7c42a.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46629/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46629/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46629/

Test build #142119 has started for PR 33651 at commit 180ef97.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46632/

Test build #142130 has started for PR 33651 at commit e8f17a4.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46632/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46632/

Test build #142117 has finished for PR 33651 at commit ba7c42a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46642/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142117/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46642/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46642/

retest this please

Test build #142152 has started for PR 33651 at commit e8f17a4.

Test build #142152 has finished for PR 33651 at commit e8f17a4.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142152/

Test build #142119 has finished for PR 33651 at commit 180ef97.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142119/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142130/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46664/

Test build #142162 has started for PR 33651 at commit 6f28a37.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46664/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46664/

Test build #142162 has finished for PR 33651 at commit 6f28a37.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142162/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46675/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46675/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46675/

retest this please

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142178/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46690/

retest this please

Test build #142187 has started for PR 33651 at commit 6f28a37.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46699/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46699/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46699/

Test build #142187 has finished for PR 33651 at commit 6f28a37.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142187/

thanks, merging to master!


NPM packaging guidance for converting comilper d.ts files to external modules

TypeScript Version:
1.8.2
Code
On this page the statement is made that the modules should "be written as external modules" when using with the typings entry.
I am using declarations option with outFile to produce a single *.d.ts file which I want to include in my NPM package. This produces ambient module declarations, which is not compatible with using the typings entry in package.json.
Does guidance exist to convert these compiler produced *.d.ts files to an acceptable format for use with the typings entry in package.json?

The problem here is that by using --outFile, you're creating modules that can be consumed only in an AMD/SystemJS setting, and in turn, you can't consume it in vanilla Node.js. The same thing applies to your .d.ts file if you're using the Node strategy. At that point, you've basically made it problematic for your users to consume your library. So I'm wondering how you're currently intending for users to consume the library over npm.

Working on that, and looking for the best way. I am only using outFile to build the single d.ts file in my build flow. When I create the .js file I compile into separate files and then combine using browserify. It is that .js file I would like to distribute with the d.ts file along side so that case would be:
npm install blah
thing = require(blah)
and thing has intellisense in my new typescript project thanks to included d.ts file.
Perhaps I am going about this entirely wrongly, which would be good to know and I can go in another direction.

I think I see what you mean - but my feeling is that you should have non-bundled .js/.d.ts files in your package for regular npm users (sometimes in a folder called lib), and Browserify should be doing the custom concatenation steps for anyone who wants to bundle (often into a folder called dist).

Not sure if I am closer or not. Rebuilt the NPM package and included a lib folder that includes my raw compiled js files with the d.ts files along side. So file.js and file.d.ts site side by side. I updated the main to point to file.js and typings to point to both the entire lib folder as well as the specific file.d.ts but now get the error "Module '' resolves to a non-module entity and cannot be imported using this construct."
Honestly not sure if that is the js or d.ts file that is causing the issue. I am using UMD for build type for the js files, the d.ts files are just included as they are output. I am using the "import { name } from "npm-module-name";" syntax. I'll keep plugging away and see what I can find.

but now get the error "Module '' resolves to a non-module entity and cannot be imported using this construct."

Where exactly are you getting this error?

I get that error when I hover over the Import statement in the TypeScript file in the test project.
Now, interestingly (at least to me), if I change the file extension of my test file from .ts to .js and I change the include style to: var thing = require("npm-module-name");
Everything works as expected. I get intellisense support and all the various components resolve as expected off my root entity "thing".
So I guess my question has become how to mimic this behavior in a .ts file? I am using vs code, so perhaps the limitation is there. Seems to be really close to what I want to end up with.

And solved. I changed the import to: import thing= require("npm-module-name"); in the .ts file and everything works as expected. So the final solution was:

Include an uncompressed build in the npm package with the d.ts files along side the js files
Use the main and typings values in the package.json to point to the root js and d.ts files.
Use the import thing= require("npm-module-name"); include syntax (maybe unique to how my code is structured)

Thanks Daniel for the help on this, going to go ahead and close the issue.


Wasm: spec tests + unreachable state + bug fixes

Update wasm testsuite files from spec repo.
Removed unreachable type and use Unreachable state instead. Simplify how we handle unreachable to be more spec compliant.
Trap if we want to access the memory without a memory available

@MikeHolman @Krovatkin @arunetm @commonlisp Can you guys review this please ?

ping

:shipit:


Changing the template compilation result to use the new syntax

This pull request changes template class generators to generate code with the new syntax (with require) instead of the old one (with $dependencies).
A specific use case (tested by test.aria.templates.reloadResources.ReloadResourcesTestCase) requires some changes in noder-js (cf ariatemplates/noder-js#31). It is handled in a separate commit.
Before integrating this PR:

[ ] wait for ariatemplates/noder-js#31 to be integrated in noder-js
[ ] use the new version of noder-js

Fine for me.


adds required_keys for most Secrets

🚀 The docs preview is ready! Check it out here:     https://modal-labs-examples--frontend-preview-248cbc5.modal.run


CC1101 support?

This might help?
https://github.com/h-RAT/EvilCrowRF_Custom_Firmware_CC1101_FlipperZero

JustCallMeKoko is working on adding this to the marauder codebase, you can fork in the meantime if you'd like


error 500 on search by unzr

in case there are more then one person with the same unzr it is an error on search person with unzr.
On search person if there is an unzr do not search only by unzr, but AND unzr=$unzr. Error is only for REST search.
request dev
curl -X GET \
  'https://api.dev.asclepius.com.ua/api/persons?first_name=%D0%9E%D0%BA%D1%81%D0%B0%D0%BD%D0%B0&last_name=%D0%9A%D0%BE%D0%BD%D0%BE%D0%B2%D0%B0%D0%BB%D0%B5%D0%BD%D0%BA%D0%B0&birth_date=1986-11-16&unzr=19861116-01480' \
  -H 'Authorization: Bearer VURCRDhneDNZTWJaWXM5TnVnRkMvdz09' \
  -H 'Postman-Token: 188aec91-d299-48c1-87de-090d6ad284ad' \
  -H 'api-key: ejJ1VWFqZlNUUlpmMWNIMmc0YTB1UT09' \
  -H 'cache-control: no-cache'

response
{
    "error": {
        "errors": {
            "detail": "Internal server error"
        }
    },
    "meta": {
        "code": 500,
        "request_id": "a39037e0-7829-4acb-8634-cf02f1b31793#2553",
        "type": "object",
        "url": "http://api-svc.il/api/persons"
    }
}

After person search was moved on RPC in #477 the issue is not reproducing anymore.


[TACHYON-709] Fix deadlock because different blocks may share the same lock on worker side

https://tachyon.atlassian.net/browse/TACHYON-709

+1 LGTM, thanks for the fix!

+1 LGTM!  thanks for the patch


[Examples] fix remark link in blog-starter's README

Looks like the correct link is https://github.com/remarkjs/remark
Nice example by the way!

Thanks!


Test

Functionality checked to run consistently.

Codecov Report

Merging #4 into master will increase coverage by 1.99%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master       #4      +/-   ##
==========================================
+ Coverage   17.17%   19.16%   +1.99%
==========================================
  Files          10       11       +1
  Lines         594      407     -187
==========================================
- Hits          102       78      -24
+ Misses        492      329     -163

Impacted Files
Coverage Δ

hn2016_falwa/lwa_ncforce.py
0% <0%> (ø)

hn2016_falwa/qgpv_lwa_ncforce.py
0% <0%> (ø)

hn2016_falwa/oopinterface.py

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 230a556...1d78862. Read the comment docs.


Egg Hatching logs do not truncate IV

Expected Behavior
Egg hatching log output should match the rest of the log output.
Actual Behavior
Log output does not match the rest of the bot's output.
[23:27:12] Egg hatched with a Magmar (CP 1244 - IV 10/14/11 0.777777777778), 1000 exp, 2780 stardust and 27 candies.
compared to the release output:
[23:27:14] Exchanged Magmar [CP 1244] [IV 0.78] for candy.
I'd expect the same MonName [CP ###] [IV #.##] output.
It might be worth adding this sort of string output formatting to the utils class.

Looks like it just needs to be changed over yet to using the inventory class(unless there is a reason this should be in utils instead of inventory?). Looks like transferring was just moved over to it:
https://github.com/PokemonGoF/PokemonGo-Bot/commit/8b5af8d766ac161be926932b05e3fc785a33661d

Didn't realize the Inventory class was now in. That sounds perfect for this.

Seems resolved.


Better error when split libraries doesn't yield any sequences.

The current error is:
Status: Job 42cadf02-5ad8-4104-a211-4d4be7c6b866: failed - Error executing Split libraries:
['Traceback (most recent call last):\n', ' File "/home/qiita/qiita_main/qiita_plugins/target_gene/tgp/plugin.py", line 56, in execute_job\n output_dir)\n', ' File "/home/qiita/qiita_main/qiita_plugins/target_gene/tgp/split_libraries/split_libraries.py", line 285, in split_libraries\n generate_demux_file(output_dir)\n', ' File "/home/qiita/qiita_main/qiita_plugins/target_gene/tgp/split_libraries/util.py", line 140, in generate_demux_file\n to_hdf5(fastq_fp, f)\n', ' File "/home/qiita/qiita_main/qiita_ware/demux.py", line 355, in to_hdf5\n sample_stats, full_stats = _summarize_lengths(_per_sample_lengths(fp))\n', ' File "/home/qiita/qiita_main/qiita_ware/demux.py", line 243, in _summarize_lengths\n full_stats = stat(n=all_lengths.size, max=all_lengths.max(),\n', ' File "/opt/numpy/1.7.2/lib/python2.7/site-packages/numpy/core/_methods.py", line 10, in _amax\n out=out, keepdims=keepdims)\n', 'ValueError: zero-size array to reduction operation maximum which has no identity\n']

Just to build a bit of intuition on how to fix this:  the error is raised from the generation of the demux file. Before doing such call, we should check if the output FASTQ files is empty or not.


Start doing testing of compiler components

Currently there are only end-to-end tests. We should have property and unit testing of smaller components in the compiler. An example is the simplifier which should ideally be idempotent (it's currently not, but testing that is the first step to solve that problem).

Is it this function that you're talking about?
Also, are you thinking about having an examples-based test or a property-based test for this? Any particular testing framework(s) to be used, such as tasty? Afaict, all the current tests are purely in Sixten and nothing in Haskell, so the testing script will also need to be modified/replaced.

Yeah, that's the one!
I was thinking of a property based test, but thinking about it some more that would be fairly involved to get going, since we'll want to generate arbitrary well-typed terms. And an additional complication is that the function doesn't currently satisfy the property.
The right way to approach this is probably to get a trivial test, maybe for an Util function, running, and we can make a separate issue for testing simplification.
I like tasty since it allows both unit tests and property tests. If that's also your preference I say we should go for that. And you're right about the current testing situation.

By Util function, do you mean the ones in the Util module or in its submodules? I understood the simplifier part, but I'm not sure what property needs to be tested here ... could you give a concrete example?
I'm new to this but tasty's documentation seems nice, so I'm happy to use it. Would it be better to convert the existing tests first to run with tasty and then add new tests?

So I already started porting over the tests using Tasty.Golden :P. I've got the success tests working. It seems that about ~25 of those didn't have corresponding -expected files so tasty ended up automatically creating empty files. I will clean up the code and push soon-ish, so you can at least have a look and point out any glaring errors. Should I commit the newly generated empty files too?
After the first review's done, I'll write the rest of the code to incorporate the other existing tests; that part shouldn't take much effort.

What I meant was that the important thing is to get some test up and running (and not try to do the more complicated simplification test at once) and was suggesting to look in the Util module for something trivial. Since you're porting the other tests there will already be some tests though, so it's not necessary. Sorry if I'm being unclear. :)
Running the existing tests with Tasty too sounds like a good idea. Probably easier than hacking shellscripts! Yeah, commit the empty files too.
Looking forward to seeing this!

Quick question: what is the "$@" supposed to do (ex. here)? I've seen it before in the context of makefiles, but I don't think it does anything in bash ...

$@ is all the arguments given to test.sh, which means that it can be run e.g. with -v8 -O2 to set Sixten's verbosity level to 8 and optimisation to 2. It's occasionally useful so if it can be done with Tasty it would be nice, though it's not a must. :)

Okay, so here's the cleaned up version with the success tests working. https://github.com/theindigamer/sixten/commit/97aa8fad41143b680bbd4166b2d3d9b270e63661
I'll do the success multi tests next. For args, it seems like tasty always parses the command line args which is problematic...

Looks great so far! Skip the arguments for now if it's problematic.

So I did figure out how to get the options working, the command to run is a bit verbose though:
stack test --ta='--sixten-args='-v8''

This is because Tasty tries to capture/use all arguments, so I have to create a dummy argument there, and then forward it to sixten.
Also, that makes the tests fail because of the verbosity 🙁. So I will replace the 'run' command with the 'test' command and just check the exit code instead of running golden tests (which compare stdin with the golden file, oops...).

Lastly, I noticed something odd: supplying some flags to sixten run makes it crash.
$ sixten run -O2 tests/success/00-basics/Int.vix
sixten: opt: callProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
 $ sixten run -o a.out tests/success/00-basics/Int.vix
sixten: a.out: callProcess: runInteractiveProcess: exec: does not exist (No such file or directory)

The compile command seems to be working fine though, so I don't get what's wrong ... you should be able to reproduce this on b4968e81d.

That way of passing flags seems alright to me (though the quirks seem a bit weird). Having it work with -v might not be strictly necessary, but sometimes testing at different -O levels can reveal bugs so that would be good to have.
The last thing is probably that the LLVM binary opt, used when you supply -O2 to sixten, has a different name on your machine like the other LLVM binaries.

Ooh, you're right! I didn't realize that opt was a binary. 😅  That said, don't you think the -o flag behaviour with run is incorrect? It does end up creating the correct binary, but emits an error saying it couldn't find the binary...

Ah, good catch! Should be fixed by https://github.com/ollef/sixten/commit/aa3b6b8df6167282a8bbebe738c4ff1df45da8cf. The problem was that it was using a relative path to the file, but I guess callProcess would then look for a system binary and not in the current working directory.

We have the setup to do this now thanks to @theindigamer, so it's just a matter of starting to do it when it makes sense. :tada:


IS 6.1.0 docs contain attributeSyncMethod config docs without the feature.

Is your suggestion related to a missing or misleading document? Please describe.
IS 6.1.0 docs mentions attributeSyncMethod config at the IDP rest api docs [1].

But this feature was added to the product after the 6.1.0 release. Therefore 6.1.0 pack does not support this config.
Describe the improvement
The 6.1.0 docs should not mention this config.
[1] https://is.docs.wso2.com/en/latest/apis/idp-rest-api/#/Provisioning/getJITConfig

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!