First-Class Typescript Support

Problem Statement
Typings for Ramda are sometimes non accurate (examples), working with Typescript and Ramda can sometimes be frustrating and somehow badly impact adoption.
This appears to be one of the main pain points in my Ramda lessons.
Proposed Solution
As this library makes her way to the version 1.0.0 it would be great if we could consider building it in Typescript directly.
Benefits

Native support for ts definitions (no-need to install @types/ramda or @types/npm-ramda).
Drive adoption in the typescript world
Overall better developer experience for Javascript developers

I'd love to propose a PR! 🥇

If other people share my concern about not writing javascript libraries in javascript, I've come up with a way to solve many of the problems inherent in maintaining type definitions for iter-tools. Typedefs are maintained in the library, and I use a little special sauce to write my tests in such a way that the type checker can verify that typing constraints are not violated. The cool thing is that the approach works even if you intend to have first-class support for both Typescript and flow types, which I do.

@hitmands:
Another well-written proposal.
While I don't have any major objection to including such typings, I do have several concerns.

Chiefly, the current team has little knowledge of, and for the most part I think, little interest in Typescript.  So this would mean adding to the team someone who wants to commit to keeping these in sync on every release.  Are you interested in doing that?

Should this also include Flow?  Would we need a separate person to handle that?  What about working with other type systems as they come along?  How do we decide when it's become important enough to offer support, or when it's become unimportant enough to drop it?  It wasn't that long ago that we'd have been talking about CoffeeScript instead.

I don't know if it's true at the moment, but there have been times when Typescript was not expressive enough to capture certain Ramda functions.  If we own the Typescript, do we also have to forgo more sophisticated functions because we cannot express them in Typescript?  Or how do we document that certain functions are out-of-bounds in Typescript?

While I agree that it would be worth having control over these typings, I'm simply not sure that the team is ready for the responsibility.
But I would love to see a PR if you're interested in creating one.  Perhaps that would show us that it's not as difficult as I imagine.

Chiefly, the current team has little knowledge of, and for the most part I think, little interest in Typescript. So this would mean adding to the team someone who wants to commit to keeping these in sync on every release. Are you interested in doing that?

I'd love to be part of this project, and yes I'd be interested on supporting it. I use ramda every day in every project, I teach ramda and also very active on s.o. ramda topics.

Should this also include Flow? Would we need a separate person to handle that? What about working with other type systems as they come along? How do we decide when it's become important enough to offer support, or when it's become unimportant enough to drop it? It wasn't that long ago that we'd have been talking about CoffeeScript instead.

My proposal is only about Typescript as I think it is the widest used js superset atm. We could easily generate flow definitions via https://github.com/joarwilk/flowgen if the team would also want to support flow.

I don't know if it's true at the moment, but there have been times when Typescript was not expressive enough to capture certain Ramda functions. If we own the Typescript, do we also have to forgo more sophisticated functions because we cannot express them in Typescript? Or how do we document that certain functions are out-of-bounds in Typescript?

Libraries such Ramda are definitely difficult to write in typescript as most of their constructs (e.g. partial application) don't play very nicely with ts. However, as showed by @types/ramda and @types/npm-ramda a lot can be done... It is likely that not everything could be typed safely, and for those cases, we would find a way to properly communicate it. Those are also problems that already exist with current typings... My proposal is about to solve/mitigate them.

@CrossEye Are you sure you're on the same page as @hitmands? You talk about having needing to have someone to keep the types in sync. I think the proposal is to actually to change all the js files in the library to ts files. All contributors would then need to know typescript, not just a few to help with libdefs.

@conartist6: you're right, I did misread.  I read "we could consider building in Typescript directly," which I took to mean bringing the typings files in-house.  What was actually there was "we could consider building it in Typescript directly," which is a very different notion.
@hitmands: We'll have to check with the other contributors, but this is suddenly much less interesting to me.  I'm not even slightly interested in writing in TS myself.  I suppose there's room for parallel code-bases, but that sounds a bit daunting.

If I were to build a TS native library it might look quite different, and I think a number of people have been working on it.

I've enjoyed watching fp-ts and have taken a number of ideas from it for a separate project I'm working on.

Apologies for my confusing wording, yes I actually meant to rewrite ramda in typescript.
I understand if main contributors are not really interested in this,
I just thought that would be a great deal for driving adoption.
Libraries that are type safe are also genereally easier to adopt.

Ramda doesn't have the correct ergonomics to really be a solid player in TS IMO.

I think it'd be difficult to port ramda to TS, but not impossible.

and last I checked compose couldn't be typed correctly

As any function that takes rest-like arguments, also compose is difficult to type in TS, and therefore needs a lot of manual typing... but that's okay I guess!

Anyway,
I appreciate the authors point of view, and happy to not proceed with this.
However, I'd like to also move the focus on the user's standpoint. I really think that for typescript developers, as well as javascript developers, having ramda typed safely would represent a big game changing choice.

That's why I came up with options 3. Write native javascript code with typedefs alongside. Write tests which are allowed to used a (very) limited subset of type definition syntax that is common to Typescript and flow (e.g. const value: Type = ...;). Then you can run your test files with javascript (use @babel/plugin-transform-flow-strip-types), and type check them with typescript (and/or flow). Thus if your typedefs don't permit a common usage, you'll know.

@conartist6: That's interesting.  Might we also be able to simply keep a collection of TS test files designed only to check that the function types are right?  That is, could we move test to test/js and add test/ts and test/flow that are processed with whatever @babell configuration is necessary?  I don't know enough about TS or Flow testing to know if this makes sense.  But it would be a cleaner breakdown: Update source and test/js for functionality changes.  Update the TS typing file and test/ts for TS typing changes, and something similar for Flow.

@hitmands:

Apologies for my confusing wording,

Not at all.  I simply saw what I wanted to see, overlooking an important word.  :smile:

Ramda doesn't have the correct ergonomics to really be a solid player in TS IMO.

I think it'd be difficult to port ramda to TS, but not impossible.

I'm not sure what it would entail.  I don't really know TS well.  But do you think that it could be written in such a way that it would break absolutely nothing?  What would happen to the types for functions that don't map into the TS world -- as it seems compose might not?

However, I'd like to also move the focus on the user's standpoint. I really think that for typescript developers, as well as javascript developers, having ramda typed safely would represent a big game changing choice.

That's why I wouldn't mind moving the typing files in-house, if we had someone who wanted to maintain them.  But this so far has been a library written for JS developers by JS developers.  We didn't try hard to ensure CoffeeScript compatibility, we don't spend any effort worrying about how it might work with ClojureScript, PureScript, or Elm, and we haven't put in significant effort toward TypeScript or Flow.  Part of this is just a worry about the Flavor of the Month.  But it's more than that; it's that this was designed to be a fairly low-level JS library.  Anything else has to come second.
This exchange was posted in the javascript channel of https://functionalprogramming.slack.com today, and seems quite relevant:

Enrico Polanski
may I ask why many people do fp in js rather when a frendlier version in ts might be available?
(honest question, not criticism, I just like being assisted by a compiler) (edited)
masaeedu
Depends on what you consider to be friendlier I guess. You need a bunch of hacks to implement features that have been outstanding issues for years
well, "implement" I should say, you can't really implement these type system features inside the type system properly
Enrico Polanski
such as?
ts does suffer with some polymorphic functions
but generally it's expressive and quite good at infering if you know how to get around it
masaeedu
higher kinded types, polymorphism in covariant positions, equirecursive types, inference of function parameter types from function body, metaprogramming

solomon
does typescript have iso-recursive types? why would you want equi-recursive over iso-recursive?
masaeedu
equirecursive is a lot more convenient than iso-recursive, and without compiler-solved typeclasses it's doubly inconvenient to use an iso-recursive encoding (edited)

masaeedu
you can go look at the average typescript fp library and compare the quantity of ugly type boilerplate to actual code that's doing something

James Sinclair
I find the same thing with Flow.

masaeedu
if you know how to get around it
I got tired of getting around it
Enrico Polanski
well, it takes lots of time to get used to its quirks
masaeedu
well, I used it from ~2014 to ~2018 before I gave up
maybe it takes more time
Serhii Kyrychenko
What's wrong with @types/ramda ? Isn't it well typed?
Enrico Polanski
plenty of things in ramda are untypeable in ts
e.g. type safe curry
try to have your ts compiler infer correctly the curry of this function: f: <A,B> (a: A, b: B) => [A,B]

Serhii Kyrychenko
Do we want to support named (not only ordered) currying? b => a => [a, b]? What if A and B types have not empty intersection - how do we know if it was b or a in f(x)?

I'm not sure what it would entail. I don't really know TS well. But do you think that it could be written in such a way that it would break absolutely nothing?

Yes, of course, typescript is just a js superset... a valid javascript file is also a valid typescript file.

What would happen to the types for functions that don't map into the TS world -- as it seems compose might not?

As said above, everything from js maps to ts. Some typings can result verbose, as the compose case where you would need to manually type every possible function call... You can look at how compose has been typed in redux, or any other library.

That's why I wouldn't mind moving the typing files in-house, if we had someone who wanted to maintain them. But this so far has been a library written for JS developers by JS developers. We didn't try hard to ensure CoffeeScript compatibility, we don't spend any effort worrying about how it might work with ClojureScript, PureScript, or Elm, and we haven't put in significant effort toward TypeScript or Flow. Part of this is just a worry about the Flavor of the Month. But it's more than that; it's that this was designed to be a fairly low-level JS library. Anything else has to come second.

This doesn't only help ts developers, any fairly modern IDE (wstorm, vscode) would automatically infer typings if present. This works in js as well as typescript (vs code doc). This really improves the overall user experience.
and I also agree with most of the masaeedu's points, but still think that Ramda is doable in typescript.
This is how it would look like (pseudo-code):
interface Map {
	<A, B>(fn: (a: A) => B, list: A[]): B[]
	<A, B>(fn: (a: A) => B): (list: A[]) => B[]
}

const map: Map = (fn, list) => list.map(fn);

currying (etc.) would require a bit of manual typing, but nothing undoable imho!

Yeah it's not too bad to type compose that way. iter-tools does as well. But it's also not a typedef you could ever infer from native typescript.
Typedefs for the flat method  actually illustrate several of the problems. First, overloads like those with shouldFlat are wholly beyond typescript's expressiveness. Second, any additional parameters to the function would now double the number of lines needed. This is a serious maintenance cost to go through all the lines and make sure nothing is missed, and there's also no way to test all those definitions except by writing a test case for each line of the typedef. To keep things spiraling completely out of control we only allow a single partial application in our typescript types. That is to say, never (A) => (B) => (S) => R, only (A, B) => (S) => R and (A, B, S) => R.
The biggest problem you have with the as-native typescript design though is that it doesn't work at all unless you write a correct and usable type for the curry method. As far as I can tell this is impossible.
I think all this just drives home that you can't design a library like this for (or in) typescript. You can make best effort typedefs which are quite useful, but you are still designing for JS.

I think all this just drives home that you can't design a library like this for (or in) typescript. You can make best effort typedefs which are quite useful, but you are still designing for JS.

That concisely says what I've often tried to convey about Ramda's relationship to TS.

I'm curious if those who know TS well could tell me whether -- if we clean up our H-M typings to avoid all the ellipses -- we could generate everything needed for TS from them.  Is that just a pipe-dream?

This is how it would look like (pseudo-code):

interface Map {
  <A, B>(fn: (a: A) => B, list: A[]): B[]
  <A, B>(fn: (a: A) => B): (list: A[]) => B[]
}

const map: Map = (fn, list) => list.map(fn);

I think this illustrates the issue pretty well. Let's take this initial interface as a starting point and try to get at maps actual type here.
The first thing I see is that this interface only deals with Array where in actuality map does so much more than that. Say we have an Object{A} or a Maybe(A) or a SomeOtherFunctor(A) so now we have a generic container type (let's call it Functor) and this generic container also has generic contents (A). map needs to be able to take an F of A's and a function from A to B and return an F (the same type, not an arbitrary Functor) of B's regardless of what the concrete type of F or A are, so long as F is a Functor.
From my understanding, TS can't do this without a lot of effort / workarounds. Neither of the current Ramda type definitions get it right from what I can see, though not for lack of trying.

But this so far has been a library written for JS developers by JS developers.

I think it'd be difficult to port ramda to TS, but not impossible.

I think it would be impossible for an exact port, but I do think we could get relatively close. I just don't know why we would. As @CrossEye has mentioned, this is a library that was designed for JS. I agree that a type file would be a boon for Typescript developers, but I don't see changing the implementation language as being beneficial to anyone. It would limit the amount of developers comfortable contributing, especially as it has already been acknowledged that typing the sorts of functions Ramda offers is hard and often the results are difficult to read and difficult to get right.

The first thing I see is that this interface only deals with Array where in actuality map does so much more than that. Say we have an Object{A} or a Maybe(A) or a SomeOtherFunctor(A) so now we have a generic container type (let's call it Functor) and this generic container also has generic contents (A). map needs to be able to take an F of A's and a function from A to B and return an F (the same type, not an arbitrary Functor) of B's regardless of what the concrete type of F or A are, so long as F is a Functor.

type Functor<T> = Array<T> | { [key: string]: T };

interface MAP {
	<A, B>(fn: (a: A) => B): (functor: Functor<A>) => Functor<B>;
	<A, B>(fn: (a: A) => B, functor: Functor<A>): Functor<B>;
}

declare var foo: MAP;

const list = foo(() => 'hi', [1, 2, 3]);
const obj = foo(() => 'hi')({ a: 1, b: 2, c: 3 });

I am not saying that porting ramda in typescript would be easy,
I am just saying that is doable...
Currying would need to be done manually, as the MAP example above...
Algebraic Data Structures should be created manually (ts supports unions and intersections)

this is a library that was designed for JS

I was saying that also JS developers would benefit from it since any modern IDE supports ts declarations in js too.

If a full porting is not to be considered,
then what about what @CrossEye said about in-housing ts declarations?

@hitmands: But note that there are plenty of other Functors out there.  Function is a functor parameterized by the return type.  Most implementations of Maybe, Either, Future/Task and many more are functors and users can define their own.  So we would need a much more robust definition of Functor, presumably matching that of FantasyLand.

If a full porting is not to be considered, then what about what @CrossEye said about in-housing ts declarations?

That strikes me as much more reasonable.  But that means finding someone who can do this interested in a longish-term commitment to maintaining it as Ramda progresses.  The current core team simply does not have the requisite skills.  Ideally there should be more than one, but if we found one person, we could get going and see if others would join.
Are you that person?  I've seen your answers on StackOverflow and your reasonable discussions here.  I would certainly be willing to put your name before the group to be another committer if you are interested in working on this.

I'm also still really curious if we could automate a conversion from better-disciplined versions of our current Hindly-Milner style typings.  That would be the best of both worlds.

note that there are plenty of other Functors out there

Hey @CrossEye, of course there are... mine was an example, and yes we would Functor to ideally match FantasyLand...
__

Are you that person? I've seen your answers on StackOverflow and your reasonable discussions here. I would certainly be willing to put your name before the group to be another committer if you are interested in working on this.

I'd love that,
I use Ramda, I love it and I'd certainly like to help to make it more robust.

I'm also still really curious if we could automate a conversion from better-disciplined versions of our current Hindley-Milner style typings. That would be the best of both worlds.

I am not aware of any converter and I'd find difficult to imagine one :)

@hitmands:

I use Ramda, I love it and I'd certainly like to help to make it more robust.

Great.  I'll discuss it with the group and get back to you soon.

I'm also still really curious if we could automate a conversion from better-disciplined versions of our current Hindley-Milner style typings. That would be the best of both worlds.

I am not aware of any converter and I'd find difficult to imagine one :)

Well, if I find the time, once you have some typings completed, I'll take a look to see what we might manage.  Because that would be a great win!

I think this went stale and can be probably closed

Hmmm... stale? It would still be a cool project to convert ramda to TypeScript, I think. The typings on DefinitelyTyped are still limited in some ways, e.g. not all functions can work with __ there. And in the meantime TypeScript has some cool features for functional programming, there's things like Parameters<T>, which constructs a tuple type of the types of the parameters of a function type T. It would still be a project with adding alll the types, setting up the tooling, and would change how the code looks like. But it would also be super awesome to have accurate types for ramda 🎉

@bxt:
I'm thinking that would be more of a fork than a change.  I'm quite sure that I'm not alone among the Ramda core team who's not interested in maintaining Ramda in TS.
I haven't paid a great deal of attention lately; has Typescript advanced far enough that it can support all the features that Ramda includes?  It hasn't always done so.

I see, in this case I think I'd rather have a well-maintained ramda then one in TypeScript. I have to admit, I also don't know wether TypeScript would 100% support all of ramda's features atm. I guess the only way to find out is indeed to create a fork and try it 🙂  At the moment I don't have the time to really do that either, so I guess feel free to close the issue.

It isn't just used in the codebase. type is a top level function on the public API.
How is this being handled with the existing typings for Ramda that are out there?

... and how do we make the call about when to do such changes?  How widespread does another library / language / tool have to be for Ramda to make accommodations to work with it?  Does it matter how close it is in spirit to Ramda?  Does it depend only on how hard it is to make those accommodations?
Hard questions.

I think in order to move forward this we would need to hammer out some practical concerns around this proposal.

Is there a way to ensure Javascript developers can still get PRs through without Typescript knowledge?

While we tend to have a bit of bikeshedding around the PRs themselves, it's fairly straightforward to get a PR up.  Ramda was the first OS project I contributed to. It's important to me personally that the codebase remains accessible - much of my professional life has been working with junior and self taught developers and I like that our process is straightforward enough for anybody to follow. I don't know if this is a priority for other maintainers.
For reference, here is our current contributing page: https://github.com/ramda/ramda/blob/master/CONTRIBUTING.md.
A first step would be to work out how we would need to change that document. If the change involves needing to write up typings for the function in Typescript, I fear I may no longer be qualified to contribute to a library I've been maintaining for the last 5 or so years.

Type complexity: My limited understanding is that to even get close to valid types for some of our functions requires some deep Typescript knowledge that is currently beyond my ken (I have worked with Typescript extensively but not exclusively in a professional environment for about a year total in multiple places - including my current position). My understanding that certain type concepts that are relevant to Ramda are impossible to express safely in Typescript. This is mainly the Fantasyland functions which rely on Higher Kinded Types (https://www.stephanboyer.com/post/115/higher-rank-and-higher-kinded-types). It could also apply to functions like curry but I don't know enough to say. The type concerns for a library like this are significantly different than the concerns with writing application level code in Typescript.

I think this point has been fairly well covered, but I just want to be sure we have a strategy for when we can't safely type a function. How good is good enough? How do we communicate the differences?
My bottom line is that while I think that adding types would benefit Typescript developers, this isn't just a library for Typescript developers. I am a solid no on migrating to Typescript as an implementation language, and ambivalent on adding the typings in house separate from the implementation.

My thought as to process is that updating the typing file would be a requirement of a release process, not a more regular commit process.  When preparing a release, a maintainer would have to check that the new or updated functionality is expressed in the typings file.  This might fall fairly heavily on those maintainers who have a deeper TS understanding. but for simpler things, I imagine most of us can figure it out.  It should be easy to see if a new function is included, or an old one properly removed.  But when we update behavior, that might get harder.
Perhaps we can add a specific team (a subgroup of maintainers, and perhaps other interested folks) and add something like this to CONTRIBUTING.md:

If your change alters the public API, please either also update the [typings file][link here] yourself or in your Pull Request, ping (new team link here) to request help with this.

Not having that complete may or may not stop us from merging the PR, but it would definitely stop us from publishing it.
(Of course the discussion on the Deno thread about using GH commits as the source of truth might make this more difficult, too.)

The reasons I am a yes on bringing the typings in-house are (1) the number of support requests we seem to get for them, even though they haven't been ours to maintain, and (2) the fact that we have recent additions to the core team enthusiastic about keeping up with that support.

I needed some time to gather my thoughts around this.
I am not sure about type safety not being a pr requirement,
this would compromise overall type safety quality and/or put too much burden on
a few maintainers that would need to catch up with how the codebase evolves.
On the other side,
I totally agree with @Bradcomp as this requirement would compromise contribution accessibility...

I think we should find a way of unblocking this,
I really think that lack of ts support would eventually harm adoption...
Perhaps,
we should wait a bit,
or use a fork as a prof of concept?

I am not sure about type safety not being a pr requirement, this would compromise overall type safety quality and/or put too much burden on a few maintainers that would need to catch up with how the codebase evolves.

I think the only reason we are even considering doing this is that there are new maintainers excited by this.  Ramda has always been a JS library, and I cannot imagine requiring an understanding of another language in order to contribute.
I suggested not publishing until the typings were straightened out, but I can also imagine simply not merging until then.  If the contributor does not have the skills, then it might require corrections or additions from the core team before we merge it.  But somehow we need to allow any JS developer to offer contributions.

Perhaps, we should wait a bit, or use a fork as a prof of concept?

Maybe, but I would think the first step would simply to create our own version of the typings file used by TS.  That is what I assume would need to be changed on every API-altering commit.  If we can't create one that captures the library as it stands, then we will have to see if the projects is even worth doing.

I would love to see a POC that could at least handle our example code. If we can do that I think that it would be feasible to move the typings in house.
I have been thinking more about this, and there are two main places where I see things coming up:

Changes to the API - I suspect many of these, other than when adding new functions, would be minimal. The burden would be worst if we add new functions that are not similar to ones that already exist, but this is rare. I also think it will be more difficult for functions inspired by typed FP, such as additional fantasy-land support. For new functions we would possibly need somebody well versed in TS to help out, but for most updates to existing functions the changes will probably be easy to make by either the contributor or a maintainer.

Issues with the type files themselves. If somebody opens an issue because the types are rejecting a well typed expression, it would fall heavily on the maintainers who know TS to debug and determine the fix for these issues. This is similar to how we have only a few people who deal with the build issues that come up.

My trepidation on this comes from trying to add additional safety guarantees to the library before in the form of Property Based Tests. I still think they are a great way to guarantee our library against certain classes of bugs. I went through the work of adding PBTs to a number of Ramda functions that could benefit from them, as well as pushing for people to add them during PRs, and offering guidance on how to do that. Unfortunately my efforts didn't really pick up steam, and I think the tests I added are still the only ones in the codebase.
This is fine for PBTs because they don't effect the API, they just give us additional guarantees that our functions are total, or at least have a well defined domain and codomain. But with these typings we would be adding something that has an external effect, and will generate additional work if they are wrong or go out of date. It seems like much more of an all or nothing proposition.

Hello!
As someone that likes type-safety provided by tooling, I stumbled upon this discussion. I am not at all qualified to take part in this discussion, but it seems to me like some of the mentioned hurdles will be resolved with a future major release of TypeScript. See here:
https://devblogs.microsoft.com/typescript/announcing-typescript-4-0-beta/
I hope this is actually relevant and not missing the point of this discussion entirely.

Hi, for instance which ones @PunkIslamist?

Hey @hitmands!
Sorry, just posting a link was very unspecific. The first feature described in the post seemed relevant to the discussion - variadic tuple types. They say that it can resolve the problem of escalating overloads. They also show a way to implement a generic partialCall() method with this feature.
I hope this clears it up a bit :)

I hope this is actually relevant and not missing the point of this discussion entirely.

I only read through that first section, but it did seem relevant.  This could be very good news for those interested in Ramda with complete and accurate TS typings.  I don't know TS well enough to see if that covers all the gap that currently exists, but it looks like it will at least reduce that gap.

Hi, for instance which ones @PunkIslamist?

Hello, any advancement on that issue. Are you still working on typescript support? I would like to help.

Hi @denismaxim0v ,
yes, there is a lot of interest on providing a safe ts experience,
we are deciding what the first steps will be.
So far, I think we are going to :

I would love to see a POC that could at least handle our example code. If we can do that I think that it would be feasible to move the typings in house.

I was also thinking to reach out to the authors of @types/ramda and @types/npm-ramda and see if there is interest in using their versions as starting points.

@hitmands Those sound like great first steps. I think a POC is the way to go, and reaching out to either or both of those projects seems easier than starting from scratch.

@hitmands Oh, great. Will look into these two for now then

I would also be on board to help if needed, I could really use the 4.0 support for TS

Great, so those are the packages:

https://github.com/typed-typings/npm-ramda (formerly: https://github.com/ikatyang/types-ramda)
https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/ramda

I have emailed the authors and waiting for a feedback to plan next steps :)
Go Team!

Thanks for the notice. Basically, the approach behind npm-ramda is to use the codegen to generate the actual types from the type templates (e.g., the R.map type template and its generated type) to avoid writing those complex types manually. It worked mostly, the main issue is that TypeScript does not support HKTs, users will get errors if they'd like to use HKTs, or users can disable strictFunctionTypes to suppress those errors with the cost of getting not-that-accurate types. I'm still waiting TS to add such support so that we could get 100% type-accuracy, but I guess it should be enough accurate for most people to use for now.

As a passionate ramda and TypeScript user, I would love to see this happening!
Also might be able to offer a bit of help. I started using TypeScript quite recently, but I have background in Java and C# development. I prefer having strongly typed codebase and I used to work with quite complex hierarchies of generics.
Honestly, I hated JavaScript in the beginning, for not being strict enough, so migration to TypeScript is a huge improvement for me.
@CrossEye also, sorry for offtopic, but this caught my attention - is it posible to somehow apply for an invitation to FP Slack, which you mentioned? (functionalprogramming.slack.com)

@malyzeli:

@CrossEye also, sorry for offtopic, but this caught my attention - is it posible to somehow apply for an invitation to FP Slack, which you mentioned? (functionalprogramming.slack.com)

I'm sure it is.  I must have done so several years ago, but I'm afraid I don't recall the process.

@ikatyang:

the main issue is that TypeScript does not support HKTs

I don't know enough about TS to understand if the recent updates will fix that:
https://devblogs.microsoft.com/typescript/announcing-typescript-4-0-beta/.  Do you know?

Sorry for the late response. TS 4.0 improves it a lot, for example it is now possible to "carry" the generics with tuple types (i.e., you can think of that the tuple type can now be generic, though you cannot explicit specify it, they can only be inferred using infer), I'm not sure if it can solve all the issues but it should significantly improve the type accuracy here.
And also, there is an upcoming TS feature (currently scheduled at TS 4.1) called Template Literal Type, which can do more magic things, someone even uses it to write a type-level JSON parser. I'm not that good at meta-programming but maybe someone can use it to improve the type accuracy here.

@Bessonov checkout Zod if you are interested in parsing schemas at runtime and using static types based on those schemas in your code - does that make sense?

In my eyes the core of the issue can be summed up in the following code block:
compose(toUpper, prop('name')) // => Error underneath prop('name')

The error message provided to the user:
 No overload matches this call.
  The last overload gave the following error.
    Argument of type '<T>(obj: Record<"name", T>) => T' is not assignable to parameter of type '(x0: unknown, x1: unknown, x2: unknown) => string'.
      Types of parameters 'obj' and 'x0' are incompatible.
        Type 'unknown' is not assignable to type 'Record<"name", string>'.ts(2769)

To make the code work is quite simple:
// Conceptual
compose(toUpper, prop<Input, Output>('name'))
// General
compose(toUpper, prop<string, string>('name'))
// Specific
compose(toUpper, prop<'name', string>('name'))

The reason the error occurs is not due to an error of Typescript, Ramda or the type definitions. prop('name') cannot know what is it operating on if it is partially applied unless the user defines it and the only sensible way to do that is through generics. We are expecting to much of the type inference engine and too little of the person writing the software.
Example 1

non-FP style
immediate invocation
type inference works

// Relevant type defintion
function prop<P extends keyof T, T>(p: P, obj: T): T[P];
// Code
prop('name', {name: 'Bill Bryson'}) // => <"name", {name: string}> types inferred
prop<'name', {name: string}>('name', {name: 'Bill Bryson'}) // => equivalent to above

Example 2

FP style
currying
immediate invocation
type inference works

// FP style - partial application. Type inference works
// Relevant type definition
function prop<P extends string>(p: P): <T>(obj: Record<P, T>) => T;
// Code
prop('name')({name: 'Bill Bryson'}) // => <'name'><Record<'name', string>>
prop<'name'>('name')<Record<'name', string>>({name: 'Bill Bryson'})  // => equivalent to above

Example 3

Point-free
currying
partial application
delayed invocation
type inference cannot work

// Relevant type definition
function prop<P extends string, T>(p: P): (obj: Record<P, T>) => T;
// Code
compose(toUpper, prop('name'))
// In this example prop('name') is resolving to prop<'name', unknown>('name')
// toUpper expects a string, not unknown, therefore the compiler throws a warning

This quote from the Typescript Documentation sums up the issue:
Point-free programming — heavy use of currying and function composition — is possible in JavaScript, but can be verbose. In TypeScript, type inference often fails for point-free programs, so you’ll end up specifying type parameters instead of value parameters. The result is so verbose that it’s usually better to avoid point-free programming.
With all that being said the issue generics are a great way of keeping code concise and I feel the benefits of point-free programming are worth having to manually set types.
The issue for me is not the language Ramda is written in, nor is it (for the most part) an issue with the typings. The issue is the complete lack of documentation (outside of source code) detailing how to apply generics to enable point-free programming. I am currently going through each Ramda function and adding notes on how the typings could be improved as well as writing documentation detailing how to use that specific functions generics.
For example:
filter
@generic Input [set] - if you passing number[] this should be set as number
const ages = [56, 30, null]
const names = ['Bill', 'Bob']
// Untyped
const filterAges = filter(Boolean)
filterAges(names) // => silent error
// Typed
const filterAges = filter<number>(Boolean)
filterAges(names) // => Typescript warning

##### `isNil`
- No Typescript knowledge is required to use this function

This is just my alternative view but I do think any re-write has an huge opportunity cost, not a lot of benefit and misses the core of the issue which is a lack of documentation. I may be missing something but that is my take on the issue and if anyone else is in agreement I would love a bit of help writing up the documentation which could then potentially be added as a link on the home page.

I totally agree with what you said,
I'd just add one thing though:
The issue between point free programming and ts generics
doesn't really represent a limitation, it only leads to a more verbose code in my opinion.
Still something that should not prevent us from delivering a great developer experience,
especially nowadays that ts adoption is constantly improving (e.g. deno, etc.)

@alexandermckay:
That looks wonderful.  When you're done, do you see this as suggesting changes to Ramda -- more JSDoc tags? Separate documentation? -- or to a Ramda typings document?  Or do you see this as an eventually independent of both?

@CrossEye Separate documentation for the time being

@hitmands:
Any progress to report?

Hi @CrossEye, not really...
We can re-engage with https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/ramda...

This is now out of date,
as it only supports<EMAIL_ADDRESS>https://github.com/typed-typings/npm-ramda

I ended up creating https://github.com/hitmands/ramda-ts,
I plan to start porting one Ramda function at a time and see where/if I happen to hit dead spots.
Everyone is very welcome to contribute,
it's there to prove/test the typescript maturity to support a library like ramda...
Maybe, if it plays well, we might end up supporting it here on https://github.com/ramda
cc: @ramda/core

I think this is a great approach.
I really need to get a new version of Ramda (JS!) out the door, then maybe I'll try to contribute.  I've never found much to love in TS, but I'm reluctantly deciding I might need to learn it.

I dont believe it is prudent to fight the battle when we could contribute to @types/ramda thou. Being said, TypeScript has done a great job adding functionality that would alleviate the pain for FP libraries like ramda, it may be worth the try over here. But before even getting there, making @types/ramda clean, and simple to maintain would be step 0 IMO.

@CrossEye TS is unyieldy, ugly, and the deeper you get, the more you see it's really just dueling banjos. Programatic types vs program logic. But... the bigger a codebase gets, the more valuable TS becomes. I think the key is simply using as little of it as you can, as strictly as you can.

@yordis:
I have no problem pursuing both in parallel.  If I understand the goal @hitmands is shooting for, this would eventually lead to a parallel implementation of Ramda, the original one in TS and one in JS.  The JS code would not be disturbed at all by types, but there would be a TS-native version as well.  I imagine there are tools that could extract the TS types for use with the JS version if people for some reason wanted to do that.
Are you interested in helping contribute to @types/ramda?  I personally do not have the relevant skills to do so.  I keep hearing, though that best solution would be to have the types maintained by the Ramda team, who would be able to keep up with the changes more quickly.
@mikestopcontinues: I've been involved in some 100K+-line applications in JS and didn't miss types.  But I've never done any in the 1M+ range, and perhaps I would grow to appreciate it there.  I do understand the appeal of static types, and I love the fact that when I work with Haskell, usually once I get the types exactly right, the code is already doing just what I want.  But I'm still much more productive in JS.  Maybe some day I'll take to TS, but I'm not holding my breath.
I'm curious what you mean to imply by "dueling banjos" here.  I certainly know the song, but am not clear on its implications here.

@CrossEye By dueling banjos, I mean that with typescript, you can end up parroting the actual program logic. Maybe it's beautiful music, but it's also virtuosity without a real benefit.
For instance, for a function like uppercaseKeys(obj) you can write a return type UppercasedKeys<Obj> that takes an arbitrary input type and returns a type with the keys uppercased. To write that type, you parallel the function with identical type logic (an actual map and transform func).
But in doing so, have you statically typed your program? Or even prevented any bugs?  Maybe you just wrote the wrong logic twice. Worse is that it's so tightly coupled that any change means literally doubling your work, much like with tightly coupled unit tests.
So I think there's an 80/20 rule here. Use 20% of TS features for 80% of the benefits. The rest definitely costs more in productivity than it's worth.

@mikestopcontinues I know what you mean, and I've definitely seen some types so complicated that I ripped them out as useless. I think a lot of people think of typescript as "just a superset of JS" and unfortunately they also brand themselves that way, but really writing in a typed language is something you must think about fundamentally differently.
That said your average user shouldn't be writing uppercaseKeys themselves. They should be importing it from a trusted library where any bugs in the implementation can be fixed for all users if they are found by one user. Indeed for the maintainers of that library as you point out having to maintain the logic and the types that mirror the logic is additional effort, but I think you'd find that all the users would see is a win.

@mikestopcontinues:

By dueling banjos, I mean that with typescript, you can end up parroting the actual program logic.

Ahh, that makes sense.  And it is one of the thing that most annoys me when reading TS.
@conartist6:

writing in a typed language is something you must think about fundamentally differently

Yes, this!
However, I don't agree with this:

That said your average user shouldn't be writing uppercaseKeys themselves. They should be importing it from a trusted library

That's a rare enough need to my eyes that I would certainly never include it in Ramda, nor would I expect most libraries to want to include it.  If I needed it across an application, I would write it once for that application, or depending upon how simple it is to implement (I can imagine, for instance rekey (toUpper)) I might not even bother.  But Mike's point is clear enough regardless of the example.

Folks,
In Haskell (it doesn't get any better than that for FP), you add the type definition along with the actual code ... so is the problem with it?

I don't get why we keep fighting for something that could be "why not both?"
Why so much push and pull and opinion X and opinion Y when we can focus on helping each other. And becoming a fight about types and types systems and all that jazz.
Do I agree with you to some extent that we don't need TypeScript as much? Absolutely.
But I invested 5 hours and went to sleep at 6 AM working on https://github.com/DefinitelyTyped/DefinitelyTyped/pull/56754 Not trying to brag about it (nothing I would encourage others to do) but the fact that I put the effort into it.
Why?
Because I am mentoring person about FP, how to re-implement Ramda as a learning opportunity. He does care about the inputs and outputs even though the person can't comprehend much what is happening in the middle. He is not at your level.
I didn't tell the person: you must know, and you must understand and types are useless?

Can we focus on encouraging people to contribute to @types/ramda?
Can we focus on meeting people where they are?
Can we focus on caring for those on the other side?
Anyway, I may unfollow the thread, I rather focus in continue to improve the package even thou brings no value to me.

@yordis:
I'm afraid you're seeing a fight where I see a civil discussion about priorities.

I don't get why we keep fighting for something that could be "why not both?"

Well that's what I said in my last response to you:

I have no problem pursuing both in parallel.

If you find working on @types/ramda to be most important, I would certainly expect you to do so.  But I don't share that priority.
Most contributors to open source projects have multiple systems they are involved with.  I have a separate project that I"m hoping to open source in the next year, and it takes much more of my time than Ramda.  Plus I have two others I'm noodling with.  Besides that, I have family, work, civic responsibilities, the family farm: lots of other things.  So I have to decide what to work on.  I have almost no interest in TypeScript, so it's hard to spend any effort on it.
@hitmands came aboard the Ramda team in part because of experience with TypeScript.  We welcome any help you can give, but I'm mostly willing to let hitmands take the lead on our approach.  Most people who want the Ramda team to work better with TypeScript seem to want the Ramda team to supply the types ourselves.  I think working to improve @types/ramda in parallel is a very good idea, but if we can do both, I see no objections.

I have done some optimizes to @types/ramda recently, as follows:
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/57346
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/57362
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/57367
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/57374
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/57393
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/57405
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/57411
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/57424
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/57458
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/57476
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/57558
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/57567
I think put ramda type declarations in DefinitelyTyped repository separately is more flexible, since we can fix type declarations problems frequently and quickly, and no need to publish a new ramda version for only type changing specially .

@adispring: It's great to see this.  I haven't dug into the types themselves as they'd be pretty meaningless to me.
I am curious whether TS typing is now sophisticated enough to cover all of Ramda's more sophisticated structures.  I know that for a long time they weren't but that they have made progress.

Typescript still has some limitations, for example:

it not supports curried function very well, especially curried callback function.
automatic type inference is not accurate in some cases.

@adispring is there some info how to correctly type converge now? Somehow my types never check out.
even with simple example from docs:
const average = converge(divide, [sum, length])
average([1, 2, 3, 4, 5, 6, 7])

TS version is 4.5.3

Instead of pointfreee style, try to write the converging function arguments explitely, as follows:
converge((a, b) => divide(a, b), [sum, length])
Current converge ts type does not support curry well. I'll optimize it latter.

Instead of pointfreee style, try to write the converging function arguments explitely, as follows:
converge((a, b) => divide(a, b), [sum, length])
Current converge ts type does not support curry well. I'll optimize it latter.

Small rand. Sorry for this since it's so close to New Year, but seriously, tell me something. Ramda was never good with TS. And when I say "good", I mean kind of "weird", but how did you manage to make it such a disaster. I am trying to upgrade to the newer versions and it is a complete mess. The Docs, don't make sense anymore. and tbh, TS tells me to do the wrong thing. I know it, and yet it tells me to do so.
Basically, you turn TS to the dark side. Really, how is this possible. Putting it on hold until whenever. Again sorry for the rant so close to the holidays. Besides the recent issues, Ramda is an amazing lib. Merry Christmas everyone.

@EfstathiadisD come use iter-tools! You sound like you want a functional library which favors idiomatic usage and typing, and that's what I've built. As you point out ramda is amazing, it just doesn't have the design goals you seem to want.

I've been using it with TS for a while... You just gotta be generous with your ts-ignore comments when ts isn't smart enough figure out the types. I'm satisfied with this personally. TS is a tool, and when it gets on the way, just turn it off

I'm closing this for now.  I don't want to abandon it, but it's clearly not going to get done on the path to v1.0.  We can come back to it later.  @hitmands: see https://github.com/orgs/ramda/teams/core/discussions/3 for why I think it won't happen for several versions.
I still would like to do it, but not to keep this issue open all the while.

I'm closing this for now. I don't want to abandon it, but it's clearly not going to get done on the path to v1.0. We can come back to it later. @hitmands: see https://github.com/orgs/ramda/teams/core/discussions/3 for why I think it won't happen for several versions.
I still would like to do it, but not to keep this issue open all the while.

The link for discussion is not working.

@woss: Sorry, I shouldn't have posted that link here. It's for internal team deliberations.  We are discussing what we might like from a roadmap.   When we have a reasonable sketch of a plan, we'll post it somewhere for wider public discussion.

I'll come back when it's completely rewritten in Typescript. Currently, Remeda is the best alternative.

@woss: Sorry, missed your last.  That must be a private discussion channel.  But the idea is that we want to get 1.0 out the door with only minor changes, then use 2.0 to upgrade to modern JS.  After that, this could become fair game.
@nghiepit: I really doubt it will ever be completely rewritten in TS.  That's simply not part of Ramda's goals.  I think the most we would ever see is parallel versions.  Much more likely is moving the typings in-house.

Instead of pointfreee style, try to write the converging function arguments explitely, as follows:
converge((a, b) => divide(a, b), [sum, length])
Current converge ts type does not support curry well. I'll optimize it latter.
If TS could support currying would be just awesome!

One of main selling point of FP is that you can program less, not more.
So IMHO Point Free style is better than literal function application:
Ok'ish:
sum (x:xs) = x + (sum xs)
sum [] = 0

Better:
sum = foldr (+) 0

Without proper types switching to TS makes:

program more to achieve same output
worse style
... why?

I've been using it with TS for a while... You just gotta be generous with your ts-ignore comments when ts isn't smart enough figure out the types. I'm satisfied with this personally. TS is a tool, and when it gets on the way, just turn it off

That is how I have been using it!
Flow was quite smarter inferring types. However now most types are in TS.. so the ignore comments bloat

Closing for now, moving everything to typescript can be addressed in the future. In the meantime, ramda types are being actively maintained in ramda/types


76 client endpoints 2

The remaining client endpoints.
There were a lot more design decisions to be made here vs the last client endpoint PR.  Feedback appreciated.
Will close #76

This looks good @dcolligan, some very useful abstractions here to simplify running all these different queries. I think we can make the naming a little bit better, but otherwise I like it a lot. I don't have time to play with this properly right now, but I'll try and get to it in the next couple of days and get back to you with some more comments.

-1 on the latest commit, I'd like to consider the style guide change separately from this PR.

I agree with @Naburimannu --- please take out the last commit and submit it as a standalone PR. We need to make sure that all style choices are made in a transparent manner.

Alright, I moved the style change request to #123

Excellent, thanks. Can we change back to HTTPClient for now also? We'll need to do a project wide search-and-replace for these strings if it's accepted anyway.

Ok, changed the name back to HTTPClient

Phew, this is a mammoth patch @dcolligan! I haven't gone through all of it yet, but it's adding a huge chunk of functionality client-side. You've done a great job making a slightly messy protocol manageable. I wonder if we can simplify things a bit though. I've found myself chasing a lot of links in the chain to find out what is actually happening. It's probably worth going through things again and seeing if they can be streamlined a bit I think.
I'll try and get through the rest as soon as I can.

@jeromekelleher Pushed some more commits which addressed your concerns

Ok, this is making more sense to me now. My comments at the end are sort-of contradicting the comments at the start, as I've changed my mind about the request container abstraction. I think the complexity that we're seeing and the difficulty in backtracing exactly what's happening in a given method is due to the container abstraction and trying to do all of the http requesting logic in a single method. I think a better option would be to write different run methods for the distinct patterns we see (which there are three I think? GET a single object, POST a list of objects and POST a single object?), and compose their functionality from the building blocks you already have lying around like deserialise and so on. I think this will make the code a lot simpler and easier to follow.

We could do that, but why not have a general method that can do everything (except for very well-defined and reoccurring cases like GET/single-object)?  Adding additional client methods will likely necessitate a refactor when they don't fall into one of our pre-defined buckets.  Plus it's another method(s) to test.
Before the list reference bases endpoint, I separated everything into a GET/single-object bucket and a POST/multiple-object bucket, but then that broke down since list reference bases is GET/multiple-object.

I agree there's a tension between making it easy to test and making it easy to read, but I think that easy to read must win every time. I just can't tell easily what's going to happen in the big complicated _doRequest method for a given endpoint, and this is definitely an alarm bell. I should be able to start at the bottom of the call chain, press # or ctrl-] a few times (assuming vim) and know exactly what is going to happen, without having to remember exactly what parameters I'm carrying back up the chain.

I disagree I'm afraid. Here's my (incomplete, untested and un-pep8 styled) version:
    def runGetRequest(self, protocolRequest, objectName, responseClass, id_):
        url = "{objectName}/{id}"
        fullUrl = posixpath.join(
            self._urlPrefix, url).format(id=id_, objectName=objectName)
        params = self._getAuth()
        requestDict = protocolRequest.toJSONDict()
        params.update(requestDict)
        response = requests.get(fullUrl, params=params)
        return self._deserializeResponse(response, responseClass)

    def getReferenceSet(self, id_):
        """
        Returns a referenceSet from the server
        """
        return self.runGetRequest(protocolRequest, "referencesets",
                protocol.GAReferenceSet, id_)

    def getReference(self, id_):
        """
        Returns a reference from the server
        """
        return self.runGetRequest(protocolRequest, "references",
                protocol.GAReference, id_)

    def listReferenceBases(self, protocolRequest, id_):
        """
        Returns an iterator over the bases from the server
        """
        return self.runListRequest(protocolRequest, "references/{id}/bases",
                protocol.GAListReferenceBasesResponse, id_)

    def searchVariants(self, protocolRequest):
        """
        Sends the specified GASearchVariantsRequest to the server and returns
        an iterator over the returned set of GAVariant objects. Result paging
        is handled transparently, so that several HTTP requests may be made
        while this method executes.
        """
        return self.runSearchRequest(protocolRequest, "variants",
                protocol.GASearchVariantsResponse, "variants")

    def searchVariantSets(self, protocolRequest):
        """
        Returns an iterator over the VariantSets from the server.
        """
        return self.runSearchRequest(protocolRequest, "variantsets",
                protocol.GASearchVariantSetsResponse, "variantSets")

    def searchReferenceSets(self, protocolRequest):
        """
        Returns an iterator over the ReferenceSets from the server.
        """
        return self.runSearchRequest(protocolRequest, "referencesets",
                protocol.GASearchReferencesRequest, "referenceSets")

    def searchReferences(self, protocolRequest):
        """
        Returns an iterator over the References from the server
        """
        return self.runSearchRequest(protocolRequest, "references",
                protocol.GASearchReferencesRequest, "references")

    def searchCallSets(self, protocolRequest):
        """
        Returns an iterator over the CallSets from the server
        """
        return self.runSearchRequest(protocolRequest, "callsets",
                protocol.GASearchCallSetsRequest, "callSets")

    def searchReadGroupSets(self, protocolRequest):
        """
        Returns an iterator over the ReadGroupSets from the server
        """
        return self.runSearchRequest(protocolRequest, "readgroupssets",
                protocol.GASearchReadGroupSetsResponse, "readGroupSets")

    def searchCallSets(self, protocolRequest):
        """
        Returns an iterator over the CallSets from the server
        """
        return self.runSearchRequest(protocolRequest, "callsets",
                protocol.GASearchCallSetsRequest, "callSets")

    def searchReads(self, protocolRequest):
        """
        Returns an iterator over the Reads from the server
        """
        return self.runSearchRequest(protocolRequest, "reads",
                protocol.GASearchReadsRequest, "alignments")

This is a lot less code, and seems a lot easier to read to me. OK, so runListRequest and runSearchRequest will share some code, but we can compose their functionality from smaller pieces (like deseraliseResponse), and then they are nice and simple.

Wow, this is a big patch. I haven't been following this patch as it's been iterated on, so take what I have to add with a grain of salt. My only comments are on the client.py changes.
The greatest argument I have against the generic request container is the fact that it's not totally generic. At https://github.com/dcolligan/server/blob/76_client_endpoints_2/ga4gh/client.py#L91 I see an if statement depending on the type of the request. To me, that seems to say that we have not appropriately encapsulated all the behavior here. It's always preferable not to duplicate code, but I'm not convinced that we haven't shared code here simply by making multiple distinct types into one here. There is probably a more polymorphic solution here and I can imagine that being worthwhile to pursue. Put another way, if you are going to have containers, I would have them contain all behavior specific to them.
Another (more off the beaten path) way to think about this whole problem is to switch from thinking about a monolithic class based approach to a more middleware-oriented approach. Put another way, instead of gathering all the logic into a single function and some helpers, can we decompose the problem a bit more. We know we have to create a request -> execute that request -> return that response. Perhaps we can break that up even more to
[create basic request -> add auth -> add path -> add params -> ...] -> [execute request -> check status -> ...] -> [deserialize response from json -> deserialize to GA objects -> convert to final return format -> ...]

I think you've made great progress toward doing something like this just by breaking behavior out into functions, but you could go further and actually describe their execution flow as data rather than code. The biggest benefit of this is that adding new functionality typically does not increase complexity significantly. It can be difficult to do this while achieving the desired high cohesion/low coupling.
You may have already considered a lot of this and rejected it. If any of it sounds useful but really really obtuse, let me know and I can try to spend some more time with the patch and formulate some more concrete ideas.

Alright, pushed a new change that implements Jerome's design that we talked about

Excellent! Thanks @dcolligan, this looks great. I've been playing with it a bit, and I think we can simplify it down a bit and also get all of our http requests going through one place. Here's one way we might do it:
    def _doRequest(self, httpMethod, url, protocolResponseClass,
                   httpParams={}, httpData=None):
        """
        TODO document this central method.
        """
        headers = {"Content-type": "application/json"}
        params = self._getAuth()
        params.update(httpParams)
        response = requests.request(
            httpMethod, url, params=params, data=httpData, headers=headers)
        self._checkStatus(response)
        return self._deserializeResponse(response, protocolResponseClass)

    def runSearchRequest(self, protocolRequest, objectName,
                         protocolResponseClass, listAttr):
        """
        Runs the specified request at the specified objectName and instantiates
        an object of the specified class. We yield each object in listAttr.
        If pages of results are present, repeat this process until the
        pageToken is null.
        """
        fullUrl = posixpath.join(self._urlPrefix, objectName + '/search')
        notDone = True
        while notDone:
            data = protocolRequest.toJSONString()
            responseObject = self._doRequest('POST', fullUrl,
                    protocolResponseClass, httpData=data)
            for extract in getattr(responseObject, listAttr):
                yield extract
            notDone = self._updateNotDone(responseObject, protocolRequest)

    def runListRequest(self, protocolRequest, url,
                       protocolResponseClass, id_):
        """
        Asks the server to list objects of type protocolResponseClass and
        returns an iterator over the results.
        """
        fullUrl = posixpath.join(self._urlPrefix, url).format(id=id_)
        notDone = True
        while notDone:
            requestDict = protocolRequest.toJSONDict()
            responseObject = self._doRequest('GET',
                    fullUrl, protocolResponseClass, requestDict)
            yield responseObject
            notDone = self._updateNotDone(responseObject, protocolRequest)

    def runGetRequest(self, objectName, protocolResponseClass, id_):
        """
        Requests an object from the server and returns the object of
        type protocolResponseClass that has id id_.
        Used for requests where a single object is the expected response.
        """
        url = "{objectName}/{id}"
        fullUrl = posixpath.join(
            self._urlPrefix, url).format(id=id_, objectName=objectName)
        return self._doRequest('GET', fullUrl, protocolResponseClass)

This needs some PEP8ing and more testing, but I've run it though your Google examples and it seems to work. What do you think?

Looks good, except for that the headers should only be set if httpData is not None.  I'll incorporate and resubmit the branch.

Alright, pushed a new (squashed) commit.

Superb! This looks really great now, thanks for all the hard work @dcolligan. I think we're ready to merge.
First, we need to rebase and fix some conflicts (toJSONString etc). Also, you seem to have picked up a bunch of random commits on .travis.yml, STYLE.md and .gitignore which should be removed. After that, I'm +1.

The STYLE.md change is unintentional but everything else is intentional.  The change in .travis.yml is just a comment to notify people about its use in run_scripts.py.  The .gitignore change deletes extraneous lines.
I just pushed a new commit rebased on top of master which rectified the STYLE.md deletion and also addresses #132

Great, let's merge this beast!
@pashields, @Naburimannu, would you mind having a quick look at this?

+1 looks good to me.

Merged! Thanks @dcolligan, this is a huge step forward client side.


⚠ PassThePopcorn has degraded performance

In eaa5ffc, PassThePopcorn ($PTP) experienced degraded performance:

HTTP code: 200
Response time: 7378 ms

Resolved: PassThePopcorn performance has improved in 517645e after 7 minutes.


JPA @Version 기능 공부

https://www.logicbig.com/tutorials/spring-framework/spring-data/optimistic-lock.html


Failed open corrupts file system

import pyfakefs.fake_filesystem
import os
fs = pyfakefs.fake_filesystem.FakeFilesystem()
fs.CreateDirectory('/Volumes')
fs.CreateDirectory('/Volumes/ramdisk')
fs.CreateDirectory('/Volumes/ramdisk/test')
os0 = pyfakefs.fake_filesystem.FakeOsModule(fs)
opener0 = pyfakefs.fake_filesystem.FakeFileOpen(fs)
component0 = "alpha"
path0 = "/Volumes/ramdisk/test"
try:
file0 = opener0(path0,'w')
except Exception as e:
print e
path0 += "/" + component0
os0.symlink(path0,path0)
the "open" yields a correct Errno 21 failure, but it causes problems in the symlink later on; clearly modifying state before it knows if there is a valid open.

This one really does seem to be a bottleneck for testing FakeFileOpen, if it's allowed to write to files.


Adding second account: work and non-work

I have set the plugin to work with my work Google Workspace. Works like a charm
I assume to add a personal account I would need to follow the same process, however I don't see a place to put another set of Client ID and Client Secret?
How would I do that?

you don't really need another client id/secret.  you can just use the same ones.  just add another account directly.

Hi, I get this error which is what I expect
Access blocked: Obsidian Google Contacts and Calendar has not completed the Google verification process
I am assuming the plugin is using API calls, I don't understand the project on my work google account can make api calls work on personal google account. In fact I would think this would be a big issue??

there is some instructions and background here https://ntawileh.github.io/obsidian-google-lookup/google-setup/ - this is the summary though:

the google client id and secret are not tied to any google account.  all those do is identify the APPLICATION being used, NOT the USER using it.  For example, any site you have visited which allows you to "Sign in with Google" has a client id+secret stored somewhere by the developer of the app.
the user and access to user/account information comes into play when you sign in (the web-based Auth flow that tells you "So and so application wants access to your calendar - do you want to allow that?"

Now back to your issue, there are instructions on that same URL on how to set up the google client app.  If you are trying to publish it as an external app available to ANY user, Google will need to verify some things and that's the error you are getting.  Alternatively you can choose to not publish it, keep the app in "Testing" mode, and then you can add test users.  In this case, you could add your work account, personal account, anything else.
Hope this helps

Hi, Got it and now it is working
Sorry I obviously was confused at where the auth was.
I am keeping it as "Testing " and assume it is all well. Thanks!


[TRI-573] Move job loading to the jobs loader

By fetching jobs too high up in the route hierarchy, navigating to the job list page does not pick up on new jobs created since the last time the jobs were fetched. This is because of how remix works and so we need to move all fetching logic to the correct places
Currently, jobs are fetched in the ProjectPresenter and should be moved to their own presenter (JobListPresenter). The route that currently fetches the jobs is here:
https://github.com/triggerdotdev/trigger.dev/blob/main/apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam/route.tsx
But they should be moved to here:
https://github.com/triggerdotdev/trigger.dev/blob/main/apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam._index/route.tsx
From SyncLinear.com | TRI-573

/bounty $50

/attempt #155


fix: Replace RawGit with W3C TR accname page

RawGit is shutting down, so swapped the live version with the published w3c TR page

Thanks so much @nschonni !


Account info does not display when row is highlighted

Description
The account information does not show when the row is highlighted. When I click on another row it displays.
Steps to reproduce

Edit the account name
Click enter
Highlight another account row

Additional context

Hedera network
other
Version
0.2.0-alpha.2
Operating system
macOS

confirmed account infos do display


Many titles are wrong

Hey, I'd like to use this schema in my project but at the moment it's kind of unusable because the title fields should really be descriptions. Most schema parsers use the title field as the type name, so when the title is The minecraft item identifier 1.8.0, the generated type name is TheMinecraftItemIdentifier180, which is less than ideal. I'm not sure if this is on purpose or if it was done by some automated tool, but it should probably be "ItemIdentifier" or something instead. I'm creating this issue to make sure there's not a good reason for this, and if there's not then I'll start fixing them + make a PR. Let me know!

It's all done under good reason, and slightly yes an automated too was used.
It's not something I want to be changed. maybe for spelling or capitalisation yes.
But it marks for example if the definition is for example not documented, usually comes along with UNDOCUMENTATED in either the title or description.
The version numbering that is shown is important for me to figure out when something is used.
Behaviors for example while having different formats. Still can refer to 1.8.0 components as they have never changed since their creation. A good example of changed components is the "Minecraft:damage_sensor" that changes somewhere in version 1.10.0 - 1.12.0. And I can see on the title of the component in the file itself how it's constructing the schema.

It's all done under good reason, and slightly yes an automated too was used.
It's not something I want to be changed. maybe for spelling or capitalisation yes.
But it marks for example if the definition is for example not documented, usually comes along with UNDOCUMENTATED in either the title or description.
The version numbering that is shown is important for me to figure out when something is used.
Behaviors for example while having different formats. Still can refer to 1.8.0 components as they have never changed since their creation. A good example of changed components is the "Minecraft:damage_sensor" that changes somewhere in version 1.10.0 - 1.12.0. And I can see on the title of the component in the file itself how it's constructing the schema.

Okay, no worries, I'll just change them locally then. Thanks!

Okay, no worries, I'll just change them locally then. Thanks!


[Enhancement] Run scan based on Jar list

Currently, you need to point DC at a folder full of the files you want to scan. What would be easier would be to simply provide a list of the files you are using, rather than go through the rigmarole of asking complicated questions of your build envt to get all the files in once place.
If the filename is all that's required to search the NVD, why do we need the whole file? I guess it's perhaps because you CRC the file and use that to lookup the NVD?
If not, can we have an option to scan using a list instead?

I had looked for "how it works" info, but clearly not well enough :-)
I'm currently automating AppScan Source scans with Jenkins, and was wanting to bolt on a DC scan as well. Since AppScan requires dependencies, I am using Maven to generate them, so that's not an issue. I call the command line DC and point at the folder of dependencies I already have.
We are using the DC Maven plugin as part of our CI, but it can add time to builds so I was looking at decoupling it.
In short, I was trying to make it as simple and lightweight as possible. Our auto deploy system generates a list of Jars, so I was hoping to be able to use that rather than have to actually compile a folder of Jars.
Thanks.

The maven plugin build time generally comes from having to update the local DB (a copy of the NVD CVE data). One thing some orgs have done is setup a central database, setup a single Jenkins job to update the database, and then run all other DC scans by pointing to the central database with the noupdate flag set.
As I believe your question has been answered I'm going to close this ticket. Feel free to re-open it if I am mistaken.


Allow uploading in own files view

We can already drag and drop to any view in the app but you could open file dialog when pressing the "upload file" button when there's no files yet

This is handled by a upload button in the files view now.


ES client upgrade

ES client upgrade required to work effectively with newer versions of Elasticsearch.
Current: 15.5.0
Upgrade: 16.7.2
(https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/breaking-changes.html)

Upgrade completed along with additional code changes to ES calls and payload processing.  Changes need to be merged into master.

Merge Complete.


Fix USD staked amount

Description
Validator USD staked amount is wrong in the Elys staking dropdown. This PR aims to fix that.

I'm unable to provide screenshots of the wrong value but here's the right number from local chain/webapp:


test_ext_peer sometimes segfaults

I think this distinct from #58. Sometimes instead of the unit tests failing as in that issue, the process itself actually segfaults!
You can view the full log at https://gist.github.com/natevw/26913782be7b00299c55 but here's the most interesting bit:
test_ext_peer(94169,0x7fff74824300) malloc: *** mach_vm_map(size=12270707960932937728) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
/bin/sh: line 1: 94169 Segmentation fault: 11  ./bin/test_$test

…seems that was a lucky capture, as usually it's a pretty silent segfault but this time there was some debug information. That seems like a pretty extreme size to be mapping (allocating?) so I wonder if this is related more to #64.
Actually running under a debugger to get a stacktrace should narrow down the cause even more, but I'm leaving this here for later to continue with the other problem I'm trying to track down.

Note that in some informal testing (not really focused on tracking down the segfault itself) this seems to happen about 10% of the time the test is run: https://github.com/telehash/telehash-c/issues/58#issuecomment-128482262 So it might need a bit of persistence to reproduce, but it's not a total unicorn either.

I've been reproducing it with a simple shell loop:

export LOOPCOUNT=0
while [ $? = 0 ]; do ((LOOPCOUNT++)); echo $LOOPCOUNT; ./test/bin/test_ext_peer; done

The first time I got a segfault after the 35th loop. On a subsequent test it didn't segfault until the 92nd loop. In all segfault cases, it seems to happen right after the final link has reported itself up, which should allow the final test to pass:
src/link.c:277 link_receive_handshake() link ready

I'm def still seeing this even w/ #58 fixed...

once I caught this w/ lldb it was obvious, probably any memory testing lib would have caught it too, I need to figure out how to get one of those integrated w/ the test suite :)


Don't suppress missing FSI transitive references

Close https://github.com/Microsoft/visualfsharp/issues/509.
Use throwOnError option so that missing reference exception is propagated correctly to the top level.

I reproduced the problem using the exact repro by @dsyme in #509.
Should I add it as a unit test? It requires adding a few big assemblies, what is a natural place to add compiled assemblies for testing?

LGTM. (I'd personally be willing to accept this change without a test case, since it's most definitely an improvement over the old terrible error message)

@latkin @KevinRansom @NumberByColors Do you guys have any opinion?

LGTM


Add detector unit tests

Pull Request
Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

#103  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @JackPlowman and the rest of your teammates on  Graphite


Todo list branch

Created repo
Init project using webpack
Created index and js files dynamically using webpack
Created todo list
User can add task
User can remove task

Hello reviewer, Please add a task on my project


force progress bar for seeking when duration is NaN

hi
i m doing live transcoding with ffmpeg then sending the result live in medialement the problem i have is vide duration is NaN so no seeking progress not working etc
i know the duration of the file i transcode before it start so i can set it in medialement if i do
var t = /([0-9]+):([0-9]+):([0-9]+.[0-9]+)/.exec(duration);
if(t){
player.options.duration = parseInt(t[1],10)6060+parseInt(t[2],10)*60+parseFloat(t[3]);
}
it works and i have 00:01:28  | 01:40:25 for exemple in the controls bar, right
now if i mouseover the progress bar it do not progress and still show NaN:NaN in the little popup and seeking disabled...
how can i "force" progress to work and enable seeking ?
thanks

@smolleyes Closing old issues. If you'd like to reopen this, let us know. Thanks!


configuration docs update

https://vsecm.com/docs/configuration/
add a notification that these values can be configued in the values.yaml of the helm charts.
also add a comment on the values.yaml files that the user can find detailed explanation of what these fields are in the official docs (in the above link)

Done.


Imported tables have huge filesize difference than pg_relation_size

Imported a small XLS file:
CartoDB Editor dashboard:
7.39 MB

pg_size_pretty( pg_relation_size('testing') )
> 288 KB

Doing VACUUM and/or VACUUM FULL does nothing.

@javisantana @saleiva : @iriberri tells me we have 3/4 issues per week lately about users complaining of this. Should I stop my current tasks and check what has changed recently regarding disk space in dashboard listing?

Let's leave for after the limits.

In a dev environment, without georeferencing file size is correct. Georeferencing is what triggers the big increase of space, but that shouldn't be a problem as it should go down, something is caching it.

Issue found. Both at rails (app/models/table/relator.rb method row_count_and_size) and the extension (CDB_Quota.sql function CDB_UserDataSize) use pg_total_relation_size, which takes into account dead rows, indexes, etc., instead of pg_relation_size (which only counts live ones).
After talking with @javisantana we'll replace by the "live only" version.

This ticket is only pending postgresql extension release to bump up rails code and push live.

@iriberri I'll push the pull request tomorrow live.


Switch to Go AWS SDK v2

See https://aws.github.io/aws-sdk-go-v2/docs/migrating/ for the
migration guide. Version 2 of the SDK features better modularisation,
more Go-idiomatic types and potentially more Go-idiomatic error
handling (albeit not on any of the errors we care about, as far as I
can determine).
Signed-off-by: Stephen Kitt<EMAIL_ADDRESS>

Tested with https://github.com/submariner-io/submariner-operator/pull/1664.


Upate GH Version to Latest Master

This is just a minor version bump.

Can't you use the recent_core branch? And if we update master should we not rather merge recent_core into master?

Oh sorry, I thought with https://github.com/graphhopper/map-matching/commit/996130db056f522e96daaef1c6e702de0753d105 we would be working on master :). In this case, recent_core is definitely better.

BTW: should we set recent_core as default branch or mention this in the readme or something like that?

Using recent_core just seems to be a bit inconsistent and I find my self stumbling over every time I do something with map-matching, which is infrequent.


Port 2001 for OpenHAB connection not available

I'm trying to connect RaspberryMatic with OpenHAB using the homematic binding.
The documentation says it needs access to port 2001 for accessing the rf components.
When I try to setup the connection in OpenHab it tries to connect to this port on RaspberryMatic, but the connection fails.
netstat on RaspberryMatic shows there is nothing behind this port:
# netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 <IP_ADDRESS>:1999            <IP_ADDRESS>:*               LISTEN      601/ReGaHss
tcp        0      0 <IP_ADDRESS>:80              <IP_ADDRESS>:*               LISTEN      420/lighttpd
tcp        0      0 <IP_ADDRESS>:8181            <IP_ADDRESS>:*               LISTEN      420/lighttpd
tcp        0      0 <IP_ADDRESS>:22              <IP_ADDRESS>:*               LISTEN      436/sshd
tcp        0      0 <IP_ADDRESS>:8183            <IP_ADDRESS>:*               LISTEN      601/ReGaHss
tcp        0      0 <IP_ADDRESS>:443             <IP_ADDRESS>:*               LISTEN      420/lighttpd
tcp        0      0 <IP_ADDRESS>:8700            <IP_ADDRESS>:*               LISTEN      479/cuxd
tcp        0      0 <IP_ADDRESS>:8701            <IP_ADDRESS>:*               LISTEN      479/cuxd
tcp        0      0 <IP_ADDRESS>:705           <IP_ADDRESS>:*               LISTEN      512/snmpd

Do I need to activate something in RaspberryMatic?

Sorry, but which RaspberryMatic Version are you using? Here I can perfectly see lighttpd listeninh for requests on port 2001:
# netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 <IP_ADDRESS>:443             <IP_ADDRESS>:*               LISTEN      484/lighttpd
tcp        0      0 <IP_ADDRESS>:705           <IP_ADDRESS>:*               LISTEN      563/snmpd
tcp        0      0 <IP_ADDRESS>:1999            <IP_ADDRESS>:*               LISTEN      725/ReGaHss
tcp        0      0 <IP_ADDRESS>:80              <IP_ADDRESS>:*               LISTEN      484/lighttpd
tcp        0      0 <IP_ADDRESS>:2001            <IP_ADDRESS>:*               LISTEN      607/rfd
tcp        0      0 <IP_ADDRESS>:8181            <IP_ADDRESS>:*               LISTEN      484/lighttpd
tcp        0      0 <IP_ADDRESS>:22              <IP_ADDRESS>:*               LISTEN      499/sshd
tcp        0      0 <IP_ADDRESS>:8183            <IP_ADDRESS>:*               LISTEN      725/ReGaHss
tcp        0      0 :::2010                 :::*                    LISTEN      624/java
tcp        0      0 :::443                  :::*                    LISTEN      484/lighttpd
tcp        0      0 :::9292                 :::*                    LISTEN      624/java
tcp        0      0 :::80                   :::*                    LISTEN      484/lighttpd
tcp        0      0 :::8181                 :::*                    LISTEN      484/lighttpd
tcp        0      0 :::22                   :::*                    LISTEN      499/sshd

Have you checked your firewall settings? Please check the settings for access to the remote scripting engine.
Also note that this is not a support forum but a bug/issue tracker. Thus, please discuss this issue at homematic-forum.de instead.

When it should be on by default and isn't on my installation I'd call it a bug.
I've installed the latest release of Raspberrymatic on a pi zero and I use the latest firmware: <IP_ADDRESS>80715

Please check if "rfd" is running because that's the process listening on port 2001 usually. Which hardware are you usibg to conmunicate with homematic devices? HM-MOD-RPI-PCB?

I'm using HM-MOD-RPI-PCB on a pi zero.
The rfd process is not running, starting it generates the log
Aug 25 12:21:04 homematic-raspi user.info rfd: BidCoS-Service started
Aug 25 12:21:04 homematic-raspi user.info rfd: XmlRpc Server is listening on TCP port 2001
Aug 25 12:21:04 homematic-raspi user.warn rfd: opendir(/etc/config/firmware//) failed
Aug 25 12:21:04 homematic-raspi user.err rfd: No BidCoS-Interface available
Aug 25 12:21:04 homematic-raspi user.warn rfd: Error initializing interfaces

So I'm looking into that error.
Thanks for pointing me into this direction.

Weird thing is in the WebUI everything seems to work.

But if rfd is not running no communication to BidCos devices are possible.

Ok, I doesn't work now. So I'm going to check the hardware. Must be a user-problem.
Sorry for keeping you in the loop. Thanks for your support.

Connect a hdmi monitor and press ALT+F2 to see bootup messages and error messages during bootup. Also show in the /var/hm_mode file if your interface was recognized correctly.


Shared Key vs Redirect uri

We are creating some mobile apps (android and IOS) that should access our API.
For user authorization we are using identityserver4.
During my research i fond out that using the Resource Owner Password Credentials flow is not good because an mobile app (public client) can not keep the Shared Key secret. And i fully agree with that.
As the alternative they say to use the Hybrid flow. This flow does not a shared key but an redirect uri to check if the client is allowed to request tokens. As i understand it correct the redirect uri is also not a protected against disassembly etc.
So my questing, what is the added security with the Hybrid flow?
Unless i'm missing something both systems do not protect against someone else duplicating my app to get access to the tokens.

Nothing protects you from that.
But there still needs to be a valid user to use that "duplicated" app - which is an explicit trust decision.
Also read this:
https://tools.ietf.org/html/draft-ietf-oauth-native-apps-09

Thanks for the link!
I still have to read in in depth but during my quick read i still havent found an answer about what makes the autorizationcode flow better then the resource owner password credential flow. (Given the app is from the same company as the api and identityserver). But all oauth documentation says that the resource owner password credential flow is bad practise (only use in legacy applications), but no explaination why.

You don't get SSO and the client app must handle the credentials.


Add 19 cases of end-to-end test for weldr-web.

How To Run:
Requirement:

Both BDCS-API and Weldr-Web need to be running on localhost.
Real API is required. Mock API will not work in end-to-end test.

./node_modules/mocha/bin/mocha test/end-to-end/test/test_*.js
Test Results:
Create Recipe Page
Input Data Validation Test
Required Field Missing
✓ should show alert message when create recipe without name (1945ms)
Simple Valid Input Test
✓ should switch to Edit Recipe page - recipe creation success (3067ms)
Edit Recipe Page
Single Word Recipe Name Scenario
Menu Nav Bar Check
✓ should show a recipe name with a correct link address (1575ms)
Title Bar Check
✓ should show a recipe name title (1687ms)
✓ should have Create Composition button (1563ms)
Create Composition Test
✓ should pop up Create Composition window by clicking Create Compostion button (1652ms)
✓ should have toast notification pop up when new composition added (4421ms)
Recipes Page
Title Check
✓ should be Recipes (1702ms)
Tool Bar
Create Recipe Test
✓ should have the Create Recipe button on the Recipes page (1749ms)
✓ should pop up Create Recipe window when click Create Recipe button (1665ms)
Recipe List
Single Word Recipe Name Scenario
Recipe Content Check
✓ should have correct recipe name on list after new recipe added (1697ms)
✓ should have correct recipe description on list after new recipe added (1713ms)
Create Composition Test
✓ should pop up Create Composition window by clicking Create Compostion button (1672ms)
✓ should have toast notification pop up when new composition added (4536ms)
View Recipe Page
Single Word Recipe Name Scenario
Menu Nav Bar Check
✓ should show a recipe name (1632ms)
Title Bar Check
✓ should show a recipe name title (1748ms)
✓ should have Create Composition button (1799ms)
Create Composition Test
✓ should pop up Create Composition window by clicking Create Compostion button (1677ms)
✓ should have toast notification pop up when new composition added (4771ms)
19 passing (42s)

@henrywang I'm not able to execute these tests. How about adding a tests/end-to-end/README.md file which describes how to run in more details. I did:
$ npm install
$ ./node_modules/mocha/bin/mocha test/end-to-end/test/test_*.js

  Create Recipe Page
    1) "before all" hook

  Edit Recipe Page
    2) "before all" hook

  Recipes Page
    3) "before all" hook

  View Recipe Page
    4) "before all" hook

  0 passing (62ms)
  4 failing

  1) Create Recipe Page "before all" hook:
     RequestError: Error: connect ECONNREFUSED <IP_ADDRESS>:3000
      at new RequestError (node_modules/request-promise-core/lib/errors.js:14:15)
      at Request.plumbing.callback (node_modules/request-promise-core/lib/plumbing.js:87:29)
      at Request.RP$callback [as _callback] (node_modules/request-promise-core/lib/plumbing.js:46:31)
      at self.callback (node_modules/request/request.js:188:22)
      at Request.onRequestError (node_modules/request/request.js:884:8)
      at Socket.socketErrorListener (_http_client.js:310:9)
      at emitErrorNT (net.js:1278:8)
      at _combinedTickCallback (internal/process/next_tick.js:74:11)
      at process._tickCallback (internal/process/next_tick.js:98:9)

Do I need to have the API backend running or not ?
Also description of how the end to end test suite is organized would be great. No need for an exhaustive architecture overview but at the very least we need to know which file includes what.
Also the new files should not introduce eslint errors.

Good idea! I'll upload a README.me file in this folder. Thanks!
@atodorov, please have BDCS-API and Weldr-Web running on localhost.

OK, after starting the API and WebUI locally I was able to start the tests. I got one error though:

Recipes Page Title Check should be Recipes:
Error: timeout of 15000ms exceeded. Ensure the done() callback is being called in this test.

I will review and retry again once the missing README.md is added and eslint errors are fixed.

Updated README.md, added End-To-End Test section to describe how to run end-to-end test.

README.md added. Thanks @atodorov for your comment.

+1 for the README, still waiting for replies on the other comments.
+1 for fixing the eslint errors

According to Travis result, there's no eslint error from my files.

Updated according to @atodorov's dependency method.

Squashed 4 commits into a single commit.

@bcl I think we are good to go for the moment unless you have more comments/preferences on the API_URL env variable. Feel free to merge, otherwise I will merge after Easter holidays.