Market Property: Balances

I would like to see a new market property called balances, that would function the same way as symbols. When you call fetchBalances(), then the balances property would get filled with an associative array of currencies with balances (free, used, total, available, etc.).
Balances would also be updated via websocket (when available) or by repeated calls to the API, depending on rate limits.
Ultimately, we could do something like:
kraken.fetchBalances();
let myAvailableBTC = kraken.balance.available['BTC'];
let myTotalBTC = kraken.balance.total['BTC'];

You will have it, soon ) Thanks for posting!

We have unified the balance API, as written here, in issue #36 . Now your feature request can be implemented easily ) We will add a dirty balance method that would save the balance on a market instance, first. And when we are ready to release the Websocket implementations, we will make it auto-updating, of course. Keep in touch! )

So, the "dirty balance" is an adjustment of the local ledger to reflect any orders I place? What about orders that fill? Is that also being tracked for the balances?

Yes, when we add a rate-limited poller and release methods for websocket apis, the balance will reflect everything in real time (as real as it can possibly be). Currently, balances are not reloaded and not cached automatically, you have to do it by manually, as we've just finished unifying them (that was a lot of work in itself). But, having them unified, we will soon add auto-adjustment as well. As for order fills, I don't think I get your question entirely, but a balance api usually tells an amount (left) in orders, a free available balance and a total, and there's usually a separate interface for execution details (transactions, trades). It will also be stream-bases, basically, all data, including the balance, open orders, partially filled orders and closed orders, all of that must be mutually consistent.

As this issue is still open and I had a hard time finding anything else about it (except issue #145 which is linked here in closed): Is there anything new on the whole WebSocket topic?

@Helmi hi! This issue is still open, we are working on it and hope to deliver it soon. We are closing in to v2, as you may see from our versioning, where we will be introducing a major refactoring and many new features ) But our WS implementation will not only stream balances, but also fills, orders, and all that stuff, where available. Stay tuned for more updates! )

Thanks for the quick headsup. That sounds awesome. Go for it 👍

+1
Any word on the version 2.0 release?

@alexmipego we are currently working on it intensively to deliver the first version of it asap, hopefully, this month.

Thank you for the effort! I am a fan!
@kroitor you wrote:

We are closing in to v2, as you may see from our versioning, where we will be introducing a major refactoring and many new features )

Is there a way to take glimpse on the v2 and WebSocket code and major refactoring? I couldn't find it nowhere.
Where are the versioning is managed? I couldn't find it.

@nitzanav

Is there a way to take glimpse on the v2 and WebSocket code and major refactoring? I couldn't find it nowhere.

The refactoring is happening right now in the main branch (there's been a lot of changes since then), and the work on websocket is also on its way, but there's no single place that holds all of it right now, the work is done in forks mostly... Here's is an example of it: https://github.com/ccxt/ccxt/pull/751

Where are the versioning is managed? I couldn't find it.

The version is single-sourced upon each build on Travis CI from package.json with the vss.js script (vss == version single-sourcing). More about it: https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md

Thank you for the information and for this amazing project!

@kroitor Is it going to be available in the three languages as usual?

@rubobaquero yes, that's the whole point, if we could do throw just one single language into it, it wouldn't take so long ) The main difficulty is to unify them all.

Any timeline updates on websocket implementation.

@GoChartingAdmin yes, we are in the process of merging this PR: https://github.com/ccxt/ccxt/pull/751, so, we hope to see first implementations of a generic ws wrapper in the coming days.

@kroitor I haven't completed my EtherDelta implementation yet. Does this PR allow for websocket implementation with all basic operations? Is it complete enough for me to give a stab at it?

@fredfortier unfortunately, not yet... It is a first approach, and it is

not unified (exchange-specific)
not portable (language-specific)

We are still working on the unification of WS (this is the hardest part of it). If you want to dive deeper into the history of WS in ccxt, here's an overview of it (contains several other implementations in JS, that are closer to a unified state): https://gist.github.com/kroitor/7dce1d23a10937ab8c07a5451f17ccf2

For PHP we can create a wrapper around some library, eg: https://github.com/amphp/websocket
Thing is, it would be available only for PHP 7+ (we can do it as a suggestion / optional thing)

It seems there to way to receive
notifications for the new comments in Gists,
so I would like to propose to move the discussion here,
where we do have this and other nice facilities.
In relation to @kvdveer comment in
https://gist.github.com/kroitor/aabecc346a5bded6ead7a82205d8ffc1#gistcomment-2403346
I agree that each language should use its own preferred idiom
and that my suggestions are mostly focused on the JS.
However, it would be sad to have the "subclassing" forced onto languages that don't need it.
I find the class based JS code often very hard to understand, because the encapsulation is too often broken with classes referring to each other inner details whose definitions are spread over huge number of files.
I recently came across a whole ecosystem of stream libraries around
https://github.com/pull-stream/pull-stream
maintained by @dominictarr (who is one of the most prolific contributors in the JS-Node community). Both code and the API are very elegant and simple and make no use of classes.

@kroitor I have been testing about websockets feature. I have used react/ratchet and evenement in php and ws and EventEmitter in javascript. I don't know if this could be a valid aproach to ccxt, but I have created php and js examples to connect cex websocket and get orderbook updates, and it works (many things to check, but I see updates in execution log). If you think it could be a good starting point I would like to test python too.

Sorry, I forgot the github link: https://github.com/lfern/ccxt/tree/feature/websockets

@lfern nice, did you work on a structure for the per-exchange websocket configs? I think that was the real hurdle

@pursehouse I have tried to put all code related to websockets out of code writen for every exchange. This way I have defined a base class AsyncConnection
https://github.com/lfern/ccxt/blob/feature/websockets/js/base/async/async_connection.js
that only defines this methods: connect, close, send and sendJson. This base class inherits from EventEmitter and will emit events related to websockets events. In this test I have defined thre eevents:

'message' when message received from connection,
'error' when error ocurred and
'close' when connection is closed by server.

I have defined a child class WebsocketConnection, from AsyncConnection,
https://github.com/lfern/ccxt/blob/feature/websockets/js/base/async/websocket_connection.js
that implements simple websocket connection (if any exchanges define a different connection type, you could define new child classes from AsyncConnection, such as pusher in bitstamp). In Exchange.js you can capture this events
https://github.com/lfern/ccxt/blob/feature/websockets/js/base/Exchange.js#L1176
and invoke custom functions defined into exchanges classes in *.js.
In cex.js I have defined _asyncOnMsg method
https://github.com/lfern/ccxt/blob/feature/websockets/js/cex.js#L539
that gets the message received from websocket, parses it and do whatever have to do. I have modified Exchange class to inherits from EventEmitter too, so you can send events from cex.js. In this testing I have only implemented order book process, so when _asyncOnMsg method detects that orderbook is received, it process and emits 'ob' event.
https://github.com/lfern/ccxt/blob/feature/websockets/js/cex.js#L563
This event could be captured from main program to implement any action on receipt. I have defined _asyncSubscribeOrderBook method on cex.js
https://github.com/lfern/ccxt/blob/feature/websockets/js/cex.js#L596
that is invoked from asyncSubscribeOrderBook method in Exchange.js. In this _asyncSubscribeOrderBook method you can generate the message to be sent to websocket to make orderbook subscription (you would define for each exchange).
Right now the configuration structure for asynchronous connections is like this:
'async': {
'type': 'ws',
'url': 'wss://ws.cex.io/ws/',
'wait4readyEvent': 'auth',
},
where type is the type of connection (Websockets in this case), url is the websocket url and wait4readyevent defines the event that you must wait until you can subscribe or send any command to the websocket (in cex yo must do authentication before send any subscription request). But this structure must be revised when new asynchronous connections would be added. I have tested in node and php, and it seems it could work.
https://github.com/lfern/ccxt/blob/feature/websockets/examples/js/async-orderbook-cex.js
https://github.com/lfern/ccxt/blob/feature/websockets/examples/php/async_orderbook_cex.php

I think that it could be a good starting point to implement websockets in exchanges, but I don't want to go on if you think that it isn't.

@lfern looks like a good start to me. I'd try to get those big if statement blocks out of the cex._asyncOnMsg method, if you don't want that method to end up giant in each exchange in the long run. _async_orderBookSubscribe or whatever...
cuz I think ideally... you'd have a list of methods and their websocket key so that a more generic single method could be running the data check. So the exchange config would have an array of the keys and their corresponding method to process that data. Then the Exchange class would be the only method needing extensive logic written into it for checking what to do with the result. Or even have them 1:1 named
instead of if( e === 'order-book-subscribe' ) ....  have it call cex._asyncOrderBookSubscribe similar to on the already existing method mapping concept
maybe try out a few more exchanges with very different web sockets systems to see what issues/refinements come up

Here I'd like to share this websocket implementation for Binance in fully functional code:
https://github.com/zoeyg/binance/issues/57
An important part is function re-usability and separation of concerns.
For instance, the whole WebSocket related part is completely encapsulated in this function:
// general purpose websocket factory
const WebSocket = require('ws')
const createWS = path => cb =>
  new WebSocket(path).on('message', cb)

The rest are pure functions that can be composed and tested in isolation.

@dmitriz I don't use streams before, but do you think this aproach could be used in other kind of connections? Bitstamp use a Pusher connection for trades, orderbook and tickers, and in gemini you have to connect to a diferent ws url for each symbol. Do you know a similar library to implement it in pyhton & PHP?

@lfern
Different ws url should not make any difference, you just compose the createWS factory with another suitable function getUrl.
I have no experience with Pusher, is that some sort of wrapper around WS?
Then it would replace the createWS factory by factory of its own, with the same idea:
const createPusher = (channel, event, key) => cb =>
  new Pusher(....).on(event, cb)

or something of that kind
(I don't know the exact syntax but they are usually similar, so I am just making it up.)
I don't have as much experience in with Python or PHP
but the principle should be the same - you create general use function
that subscribes your callback function to the stream.
If it is general and reusable, you can nicely isolate it from the rest of the code.

@dmitriz @lfern the socket code doesn't go inside the exchange class, it goes in the language specific classes. The exchange classes just have config information and maybe a little exchange specific logic.
don't get hung up on what nodejs library to use, because that part is not relevant to getting the overall ability able to function.
the key is getting the exchange class config structure able to handle anything that the variety of exchanges can throw at it.
Most likely it would make sense for the library to just support building/reading the data for the sockets, not make a decision on what socket library to be using

Only the classes like binance or gdax are not language specific. The others are language specific.
What is needed, is a configuration/method structure that can exist in each exchange class which would create the data needed to create a socket connection. Meaning the URL and potential post/get data included. Then another set of rules/methods to parse the data returning in each cycle from those websockets.
There are way too many concepts of which specific event based library someone wants/needs to use, so having those features to do the simple handling of populating the socket connection information, and processing the results of that connection makes it so that there could be a specific or barrage of different socket libraries implemented to use. Because you still need to implement code to process that data.
$exchange = new \ccxt\binance();
$socketInfo = $exchange->getSocket( $someIdeaOfParams );
$socketLibrary = \SomeSocketLib();

$connection = $socketLibrary->open( $socketInfo['url'], $socketInfo['post'] );
while( $message = yield $connection->receive() ) {
/// stuff..
}

etc...
the main hurdle is a viable repeatable config concept that works across all exchanges. The rest is secondary and not even really important to be inside the library... unless people feel like committing examples for different socket libs

Only the classes like binance or gdax are not language specific. The others are language specific.

Can you elaborate why? Why only these and why this matters?

What is needed, is a configuration/method structure that can exist in each exchange class which would create the data needed to create a socket connection. Meaning the URL and potential post/get data included.

Would some (url, options, data) work as such data?
The options should be generic for any socket connection, not specific to an exchange.

Then another set of rules/methods to parse the data returning in each cycle from those websockets.

That would be separate from any socket lib, just some plain function parser.

There are way too many concepts of which specific event based library someone wants/needs to use, so having those features to do the simple handling of populating the socket connection information, and processing the results of that connection makes it so that there could be a specific or barrage of different socket libraries implemented to use. Because you still need to implement code to process that data.

A simple way would be to only use the most basic part and expose the socket object to user for finer controls. So you don't need to worry about those, just to connect and send the data.

@dmitriz  I think you need to understand the functionality better to weigh in on this... just let @lfern continue his work

@pursehouse

@dmitriz I think you need to understand the functionality better to weigh in on this... just let @lfern continue his work

I find this comment rather rude, especially from someone who never contributed to this project.

@dmitriz sorry man your questions are just far too confused of how the split up between what can be reused and what can't. frustrating to explain it, just wait.

this is a amazing project!
what's the timeline of the websocket api support?

@imhazige it's pretty much in @lfern 's hands at the moment, you could jump in and help though if you have time

I will reproduce my comment from #3023

Hi guys, I need some help regarding this library. I'm a bit new at programming in the javascript world and I'm tring this to build some backend data calculation.

I'm trying to go through all the exchanges and download their tickers. In the loop I'm building a json structure where I can go through all the data to do calculations. I would like to do this fetching more faster since I'm planning to build a web client with some services, starting with a realtime coin marketcap. Websocket fetching will be available for ccxt?
Is it viable to do it this way? You can check the source here.

@jraicr i'd recommend branching from the work that @lfern started and try to continue his work on getting a generic format for the per-exchange configs to handle url building and result processing. The code to actually create a socket should likely be kept separate because of how different each language uses sockets. But adding a functional optional library as an example would certainly help people after the base websocket config system logic is built out

I have been testing in this feature https://github.com/lfern/ccxt/tree/feature/websockets . I have tested getting orderbook with gateio, binance, cex and coincheck, but I think its design is not enough robust yet. I would like to test it with other websockets exchanges. Do you know where to find most important exchanges with websocket support (not push protocol for now)?
Thanks

@lfern you might want to look into gdax, bitmex, bitfinex as well... we hope to compile the complete list in ccxt... but until then, I think, the procedure is to go from CoinMarketCap ranking → CCXT exchanges' list → Exchanges' API docs.

@lfern yeah like he said, test those ones and from the top rated exchanges and work your way down the list till you feel you've tweaked things enough

I tested lbank & zb websocket orderbook successfully, but I have some questions:

I have created a context object/dictionary to put all temporal info you need about the websocket management. This dictionary could be accessed from any exchange code (eg binance.js) so you could get/set any data from/to it. If you need to store a value in this context, you could do this way:

let ethBtcSymbol = this.context['ob']['ETH/BTC'];
ethBtcSynbol['ob'] = obReceived;

But when this is transpiled to PHP, you could not get a reference to this.context['ob']['ETH/BTC'], you really get a copy of this dictionary, so you don't set this value in this.context. I think the only way to do this is to code this way in .js:
this.context['ob']['ETH/BTC']['ob'] = obReceived;

Do you know hoy we could resolve this?

I have defined all new methods with "async" prefix, Do you think this is fine or its better another prefix or not to use prefixes in this functions?

How could we merge this code on ccxt? Do you think this could be merged in a separte branch, or it is better to keep on this branch?

[https://github.com/lfern/ccxt/tree/feature/websockets-multiple]https://github.com/lfern/ccxt/tree/feature/websockets-multiple
I think we need other people tests this code and get feedback from them. Its first time I use asyncio and reacphp and I am sure I made many mistakes in the implementation.

@lfern if you're best with JS then just stick with solving all the JS stuff. I'll try and checking the PHP issues asap. But l would suggest not abbreviating things like _asyncHandleOb. took my a while to figure out what "ob" meant :) might as well keep things as verbose as possible.
As far as the naming question for async or whatever... I'd think that going with socket or websocket might be better because there are more async concepts than just websockets... so that'd prevent future problems. And less confusion against the js : async setting already

How could we merge this code on ccxt? Do you think this could be merged in a separte branch, or it is better to keep on this branch?

A little more on these questions... We will start by collecting a list of ws-enabled exchanges, and will add a WS column to our exchanges' table, then will start filling it out. Hopefully, we'll have a dozen of exchanges with WS support in the first stable version of it. Once this is done, we are going to merge it to the master branch. And we will keep adding the rest of them from there.

We will start putting this into a separate branch of CCXT. However, we will need to rework the WS code itself somewhat to make it compatible with SignalR, pure/custom WS and socket.io protocols (those are three most popular streaming flavours used commonly across many exchanges). And for that we will have to make sure that we are not using any global variables or any instance variables. Otherwise this might produce tons of conflicts in the async concurrency (we're already having them in Python due to instance variables like self.last_http_response and alike).

I don't know how to do this without instance variables, because you must keep the live orderbook and modify it on every delta update, and process it in the same order as you receive them from websocket. So I think you can´t process incoming websockets messages using differents threads. (but I don't know so much about asyncio). Maybe this approach is not good to achieve this requirement.
Anyway, I think It is not a good idea to test more exchanges before these changes in WS code (to make it compatible with those protocols: SignalR, ...) would be ready. But I don't know how to collaborate in this job. I thought this could be achieved developing a wrapper over these protocols and, using EventEmitter, emitting 'on-open', 'on-message', 'on-error' and 'on-close' events. But if this is not a good aproach, maybe it would be better to start again and define a new class/interface that works fine with all of these protocols.

Maybe introduce some hint for JS to PHP transpiler to use the reference & instead of copy?
http://php.net/manual/en/language.references.whatdo.php

Maybe introduce some hint for JS to PHP transpiler to use the reference & instead of copy?

Yeah, this could be a workaround, however, the goal is to keep the codebase clean in pure target languages, otherwise those hints would be there all over this repo already.

@lfern

I don't know how to do this without instance variables, because you must keep the live orderbook and modify it on every delta update, and process it in the same order as you receive them from websocket.

The usual way of doing it is not to use instance variables, instead, you pass a reference to the orderbook into the update handling method, along with the update data itself. So, if you pass the proper references through the entire call chain, then we won't need instance variables at all, right?
The main problem with instance variables (those are almost alike "global" variables), is when async responses get out of order, you might have a lot of side-effects, conflicts and inconsistencies with methods that do mutation of those instance variables. As I said, we had some of those earlier, so we'd better not repeat the mistake again.

@lfern

Maybe someone with a great experience in python asyncio could take a look into exchange.py.

There are two exchange.py files, one for the sync python version, and one for the async python version, derived from the sync one. So, you're probably looking for https://github.com/ccxt/ccxt/blob/master/python/ccxt/async/base/exchange.py
Or I may have misunderstood the question, if so, please let me know.

I couldn't find where I should have to place this local variable (live orderbook and other status variables) to be visible in any exnchange method that handle async messages from websocket.

No worries on that, basically, what I mean is, instead of doing this:
function initializeOrderBook (snapshot) {
    this.context['ob'] = { ... snapshot ... }; // new orderbook state, for example
    this.ws.handler = this.updateOrderBook
}

function updateOrderBook (deltas) {
    const oldOrderBook = this.context['ob'];
    const newOrderBook = parseDeltas (deltas, oldOrderBook);
    this.context['ob'] = newOrderBook;
}

...

↓ instead of the above it is better to do something like this (pseudocode):
// a base method
function initializeOrderBook (snapshot) {
    const newOrderBook = new ccxt.IncrementalOrderBook ({ ... snapshot ... }) // initialize it
    // should pass by ref in PHP in the base code, class instances are passed by reference
    this.ws.handler = (deltas) => {
        const updatedOrderBook = this.updateOrderBook (newOrderBook, deltas)
        this.orderBookUpdated (updatedOrderBook)
    }
    return newOrderBook; // return it, don't store it in an instance variable
}

function updateOrderBook (orderBookInstance, deltas) {
    // mutate orderBookInstance
    orderBookInstance.update (deltas)
    return orderBookInstance;
}

In the second example the orderbook itself is an object "passed by reference", and returned from the calls, so no calls interfere with each other by writing to a shared state. The sync primitives should be in the orderbook itself. Also, it is not recommended to mix asyncio + threads. We better either use just the threaded concurrency or just the async concurrency (preferred), and you can't really use both concurrencies at the same time, as they begin to... uhm... compete.
What I meant in my previous post is that we should just pass it along with the arguments to all calls interested in it, and return the updated value from those calls, instead of writing to the instance variable. This should the PHP issues, even when the value is passed by copy.

One of the key-points of PHP5 OOP that is often mentioned is that "objects are passed by references by default". This is not completely true.
A PHP reference is an alias, which allows two different variables to write to the same value. As of PHP5, an object variable doesn't contain the object itself as value anymore. It only contains an object identifier which allows object accessors to find the actual object. When an object is sent by argument, returned or assigned to another variable, the different variables are not aliases: they hold a copy of the identifier, which points to the same object.

Hope this helps, will be happy to answer if you have more questions on this.

Thanks @kroitor I am sure it's implict here but, sorry, I still can´t realize it. Please let me follow it from your code example:
// a base method in Exhange.js
function initializeOrderBook (snapshot) {
    const newOrderBook = new ccxt.IncrementalOrderBook ({ ... snapshot ... }) // initialize it
    // should pass by ref in PHP in the base code, class instances are passed by reference
    this.ws.handler = (deltas) => {
        const updatedOrderBook = this.updateOrderBook (newOrderBook, deltas)
        this.orderBookUpdated (updatedOrderBook)
    }
    return newOrderBook; // return it, don't store it in an instance variable
}
// a base methid in Exchange.js
function updateOrderBook (orderBookInstance, deltas) {
    // mutate orderBookInstance
    orderBookInstance.update (deltas)
    return orderBookInstance;
}

In cex.js I have define an entry method to parse any message received from websocket.
/* ...*/
processWebsocketIncomingMessage (msg) {
  // parse msg
  if (orderbook snapshot) {
    let orderBook = this.initializeOrderBook (msg.snapshot);
    return orderbook;
  } else if (orderbook delta) {
    /* here I need to recover incremental orderbook and then*/
  } else if (orderbook subscription) {
    /* here I need to recover a promise set when I send a subscription message (in other cex.js method) to tell the user subscription succeded*/
   }
}

This method is invoked from Exchange.js:
/* receive msg from websocket */
let orderbook = this.processWebsocketIncommingMessage(msg);

With this model I think I have a problem because I don't know if incoming mesage is an orderbook mesage in Exchange.js (this is don in cex.js), but in case I could where I could save this orderbook variable and then recover it in Exchange.js, so I could inject it (when delta message is received) in this processWebsocketIncommingMessage method without saving it in an instance variable?
Sorry, sure it is trivial, but now I can't see how to do it.

@lfern sorry, i'm probably confusing you more than clarifying anything ) But anyways, let's not focus on instance variables, in fact we can get away with instance variables as well. Basically, there is no problem with your implementation of processWebsocketIncomingMessage (msg) in the async environment – this code won't trigger a conflict, however, accessing an instance variable may be a problem in a threaded environment (with PHP). More to it, PHP passes all arguments by copy, unless those are instances of a class.
So, we have the following choices:

either design a standalone OrderBook class to avoid using the instance variables and make it a unique instance per thread:

// in each thread:
let orderbook = new ccxt.OrderBook (exchangeInQuestion)

the orderbook will then encapsulate processIncomingOrderBookDelta method as well as other methods needed, and the user will have  separate orderbook instance in each thread...

or add primitive synching base methods to actually make the code thread-safe:

processWebsocketIncomingMessage (msg) {
  // parse msg
  if (orderbook snapshot) {
    let orderBook = this.initializeOrderBook (msg.snapshot);
    return orderbook;
  } else if (orderbook delta) {

    // lock // ←-------------- added

    /* here I need to recover incremental orderbook and then*/

    // unlock // ←-------------- added

  } else if (orderbook subscription) {
    /* here I need to recover a promise set when I send a subscription message (in other cex.js method) to tell the user subscription succeded*/
   }
}

So, you're right, there's no easy way to avoid the instance variables and make this thread-safe, without taking thread-safety for consideration from the very beginning of code design. We are not thread-safe at this moment, so to workaround this entire issue a separate exchange instance per thread is required in order to avoid possible racing. This might be a good resolution to the issue – basically we should discourage everyone from sharing the instances across threads in the first place.
Ultimately, it has to be on of:

one orderbook per thread
or one exchange per thread
or has to contain syncing mutexes

Hope it makes sense.

@lfern which branch should I look into if I want to test the latest websocket feature?

I think it's less important to solve the concept of how to use sockets for the users, as much as it is to isolate the call creation and packet parsing so that they could be used by a potential premade socket calling method, or by their own cycles than an example could be distributed of for each language

I really eager to see be able to use the socket feature with the latest ccxt version, in the browser by using javascript.

@imhazige  the current branch is https://github.com/lfern/ccxt/tree/feature/websockets-multiple You can find an example in async_orderbook.js (same for ,php and .py). This example expects parameters:
node examples/js/async_orderbook.js      [market2 ...]
Orderbook is implented in cex, gateio, binance, bitmex, lbank and zb. Orderbook delta updates need to be revised, but if you set verbose mode you can see incoming messages and hoy orderbook event is emited.

@lfern
Thank you.
I just test the examples/js/async_orderbook.js, found the proxy did not work.
The verbose option is open, I can see the request is through the proxy. but it did not work, each time through a request timeout error.
I then test other example without websocket, like live-ticker.js, still did not work. while at the latest ccxt branch, the example do works.
I think that's because the code base is far behind the mater branch of the ccxt repository.
Is it possible to merge the latest code of ccxt?

@imhazige
merged with ccxt master. I didn't test it through a proxy. I am just doing some tests and I think we need another config variable for websocket proxy, because:
prepended to URL, like https://proxy.com/https://exchange.com/api...
does not work.

@lfern
Thank you, I see the code have merged the ccxt master v1.14.240. now the http proxy works.
but when I try the websocket proxy, it did not work.
I tested with the proxy url in the example code http://<IP_ADDRESS>:8080/, it did not work
Did it work in your case?

@kroitor
Thank you for the examples.
for http, use CORS proxies is enough.
but when using websocket, I was thinking use this way https://github.com/imhazige/node-http-connect-proxy, after trying, it did not work in some country. :(, I have no idea now.

htto-proxy-agents seems to work fine with node websocket.
example.js
    const proxy = process.env.http_proxy || 'http://<IP_ADDRESS>:8080' // HTTP/HTTPS proxy to connect to
    const agent = new HttpsProxyAgent (proxy)
    exchange = new ccxt[id] ({
        apiKey: apiKey,
        secret: secret,
        enableRateLimit: true,
        verbose: true,
        agent: agent,
    });

But Autobanh/asyncio(python) is not working when try to use proxy:
2018-06-22T11:32:03 connection to tcp:<IP_ADDRESS>:443 established
2018-06-22T11:32:03 CONNECT ws.cex.io:443 HTTP/1.1
Host: ws.cex.io:443

2018-06-22T11:32:03 received HTTP response:

b'HTTP/1.1 400 Bad Request\r\nServer: cloudflare\r\nDate: Fri, 22 Jun 2018 10:32:01 GMT\r\nContent-Type: text/html\r\nContent-Length: 171\r\nConnection: close\r\nCF-RAY: -\r\n\r\n'

2018-06-22T11:32:03 received HTTP status line for proxy connect request : HTTP/1.1 400 Bad Request
2018-06-22T11:32:03 received HTTP headers for proxy connect request : {'server': 'cloudflare', 'date': 'Fri, 22 Jun 2018 10:32:01 GMT', 'content-type': 'text/html', 'content-length': '171', 'connection': 'close', 'cf-ray': '-'}
2018-06-22T11:32:03 failing proxy connect ('HTTP proxy connect failed (400 - BadRequest)')
2018-06-22T11:32:03 dropping connection to peer tcp:<IP_ADDRESS>:443 with abort=True: None
2018-06-22T11:32:03 _connectionLost: None
Traceback (most recent call last):
  File "src/py/async-orderbook.py", line 84, in <module>
    loop.run_until_complete(main())
  File "C:\Users\Luis\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py", line 468, in run_until_complete
    return future.result()
  File "src/py/async-orderbook.py", line 64, in main
    await exchange.async_subscribe('ob', symbol)
  File "c:\users\luis\documents\projects\ccxt\python\ccxt\async\base\exchange.py", line 669, in async_subscribe
    await self._async_ensure_conx_active(event, symbol, True)
  File "c:\users\luis\documents\projects\ccxt\python\ccxt\async\base\exchange.py", line 480, in _async_ensure_conx_active
    await self.async_connect()
  File "c:\users\luis\documents\projects\ccxt\python\ccxt\async\base\exchange.py", line 490, in async_connect
    await async_connection.connect()
  File "c:\users\luis\documents\projects\ccxt\python\ccxt\async\async\websocket_connection.py", line 90, in connect
    await future
Exception: connection was closed uncleanly (None)

@lfern
I tested the examples with agent option
const proxy = process.env.http_proxy || 'http://<IP_ADDRESS>:8080'; // HTTP/HTTPS proxy to connect to
  const agent = new HttpsProxyAgent(proxy);
  exchange = new ccxt[id]({
    apiKey: apiKey,
    secret: secret,
    enableRateLimit: true,
    verbose: true,
    agent: agent
  });
  exchange.on('err', (err, conxid) => {
    try {
      console.log(err);
      exchange.asyncClose(conxid);
    } catch (ex) {
      console.log(ex);
    }
  });

get a error
fetch:
 binance GET https://api.binance.com/api/v1/exchangeInfo
Request:
 {}
 undefined

Error: [RequestTimeout] binance GET https://api.binance.com/api/v1/exchangeInfo request timed out (10000 ms)

           at e  js/base/Exchange.js:388  throw new RequestTimeout (this.id + ' ' + method + ' ' + url + ' request timed …

is the proxy server http://<IP_ADDRESS>:8080 running? what's kind of proxy server it is?

Sorry, I don't have a proxy server in my network, and I didn't want to waste time installing it for testing, so I try a free proxy  from free-proxy-list.net . I think these proxies aren't always available.

In a development environment I clone from lfern/ccxt in a local folder
(./ccxt) and create a new project (./test). Then exec 'npm init', 'npm
install ccxt -save' and 'npm link ../ccxt'. Or if you prefer, you can
import directly.
El 26 jun. 2018 8:24 a. m., "imhazige"<EMAIL_ADDRESS>escribió:
so how can I use this feature via npm? or I have to import the compiled
code directly?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ccxt/ccxt/issues/56#issuecomment-400193024, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABT4232tW8dZlz391Yze9qtSVOzUDWHqks5uAdOpgaJpZM4OjakB
.

@lfern
good to know, thank you.

@lfern
I want to add the okex websocket feature, is there any develop guide for the implementation?

@imhazige Sorry, there aren't any doc ready yet. I have added some tips in
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/WEBSOCKETS.md
but you can look at cex.js, because I think orderbook subscription is similar to okex. Anyway, send me
any question about it and I would help you to enable it.

@imhazige maybe you can use this skeleton as starting point:
...
,
            'asyncconf': {
                'conx-tpls': {
                    'default': {
                        'type': 'ws',
                        'baseurl': 'wss://real.okex.com:10441/websocket',
                    },
                },
                'methodmap': {
                },
                'events': {
                    'ob': {
                        'conx-tpl': 'default',
                        'generators': {
                            'url': '{baseurl}',
                            'id': '{id}',
                        },
                    },
                },
            }
...
  _asyncOnMsg (data, conxid = 'default') {
    let msg = this.asyncParseJson (data);
    console.log(msg);
  }
  _asyncSubscribe (event, symbol, nonce) {
        if (event !== 'ob') {
            throw new NotSupported ('subscribe ' + event + '(' + symbol + ') not supported for exchange ' + this.id);
        }
        this.asyncSendJson ({
          event: 'addChannel',
          channel: 'ok_sub_spot_' + pairId + '_depth',
        });
    }

Thank you so much, I will have a look today.
On 2018/6/28 00:43, lfern wrote:

@imhazige https://github.com/imhazige maybe you can use this
skeleton as starting point:
...
,
'asyncconf':  {
'conx-tpls':  {
'default':  {
'type':  'ws',
'baseurl':  'wss://real.okex.com:10441/websocket',
},
},
'methodmap':  {
},
'events':  {
'ob':  {
'conx-tpl':  'default',
'generators':  {
'url':  '{baseurl}',
'id':  '{id}',
},
},
},
}
...
_asyncOnMsg  (data,conxid  =  'default') {
let  msg=  this.asyncParseJson  (data);
console.log(msg);
}
asyncSubscribe  (event,symbol,nonce) {
if  (event  !==  'ob') {
throw  new  NotSupported  ('subscribe '  +  event  +  '('  +  symbol+  ') not supported for exchange '  +  this.id);
}
this.asyncSendJson  ({
event:  'addChannel',
channel:  'ok_sub_spot'  +  pairId+  '_depth',
});
}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ccxt/ccxt/issues/56#issuecomment-400748176, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AD7pFvG0YcY0lhX9tlQ_YdNpyrb0mkeaks5uA7YlgaJpZM4OjakB.

@lfern
For this method asyncSubscribe (event, symbol) , could we add a third parameter params, so we can add customer params like depth

@imhazige Of course! I think it is better to get full orderbook to manage a local order book correctly, but this depth parameter could be usefull to notify a fixed length orderbook. Do any change you need, because the current code is only a proposal and a few websocket exchanges has been tested.

Do we have a list of all exchanges that offer a websocket (of any sort) connection.  I have collected a total of ~25-30 such exchanges and could add to the list that you may be maintaining at your end

@kroitor what do you think about adding these columns to the ccxt exchange list (ws ver, wsDoc)?
https://github.com/lfern/ccxt/tree/feature/websockets-multiple

@lfern sure, I'll do that shortly. However, if you don't mind, I'd put that column into the wiki under the Streaming section (does not exist there yet). I would not put that on the main list in the README.md just yet, because if I do, noob people won't read anything and will think that those columns indicate websocket support in CCXT and will just flood us with questions on ws and all that, which is not what we need right now ) So, if you don't mind, I'd open a new section on streaming in our Wiki for now. Will do that shortly. That would be available for editing and for submitting urls and other things to our knowledge-base.

I just found that the websocket proxy will not work in the browser for okex, it need allow cross origin header too.

Hi  @danieldourado
Was trying to use the binance ws.
It sounds like the function _asyncHandleObRestSnapshot that manage the binance logic about the lastUpdateId u and U, is only called one time duringthe snapshot retrieval.
For each other stream received the logic seems to be not applied, unless I am wrong.
I am working on it to fix it, but I would appreciate I you could confirm my thoughts

I am working on refactoring functions in base classes (removing 'async' prefix in methods), and adding some class methods to access context data. Hope to be ready this week.

@anymos I have just added a test for updateBidAsk and now I think it is working fine.

@lfern
Great!

Is https://github.com/lfern/ccxt/tree/feature/websockets-multiple the official branch for WebSocket development? I am interested in contributing.

@wardbradt it's a significant branch for it yes. Nothing is really "official" yet I think. But help is greatly appreciated, this is highly likely to be the merged fork I think

What help is needed on that branch? Is there a doc somewhere with remaining steps and who's working on what? I'd like to make time to help if I can.

I think it would be nice if you could try to add websocket support for any exchange and write here some feeadback about problems you find or if this aproach is not really a good way to add websockets in ccxt. With this feedback, owners of ccxt project accept/reject to merge this branch.
There is an initial document with some tips about how you can add websocket support:
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/WEBSOCKETS.md
But any question about it, please write a comment in this issue and I will answer it.

Hey, I don't have time at the moment to integrate, but I have this example for a base class and extended class for converting a specific exchange kline packet payload into a generic cross-exchange format to use by the system. For PHP it's a great benefit to have a structured class to hold these things rather than a method converting. I assume node code is the same, I just don't deal with that as much
class SocketCandleKLineBinance extends SocketCandleKLine {
    public function __construct( $payload )   {
        $this->interval             = $payload->k->i;
        $this->start_time           = $payload->k->t;
        $this->trades_count         = $payload->k->n;
        $this->price_open           = $payload->k->o;
        $this->price_high           = $payload->k->h;
        $this->price_low            = $payload->k->l;
        $this->price_close          = $payload->k->c;
        $this->volume               = $payload->k->v;
        $this->taker_base_volume    = $payload->k->V;
        $this->taker_quote_volume   = $payload->k->Q;
    }
}

and..
class SocketCandleKLine {
    public $interval;
    public $start_time;
    public $trades_count;
    public $price_open;
    public $price_high;
    public $price_low;
    public $price_close;
    public $volume;
    public $taker_base_volume;
    public $taker_quote_volume;
    public $lowest_ask;
    public $highest_bid;
}

I would also prefer to use classes to store the data, but when doing transpilation of .js code I get error accesing attributes of any class
let x = $payload.k;

I get this transpiled code in PHP
$x = $payload.k;

Sure it could be done modifying the transpile script, but it isn't easy for me.
Maybe we could define some wrapper utils out of transpilable .js code so you can access any returned dictionary from ccxt functions to classes like this.

oh that's weird... yeah I dunno how the transpiler works.. maybe @kroitor ?

@lfern the problem with generic transpiling interpreted classes is that it's very hard to guess which variable is which type without a full-scale AST traverse in run-time and without introspection/without going into the v8/python/php "bytecode" so to speak... I'll figure something as a quick solution to this.

I get this transpiled code in PHP

Actually, this particular problem is PHP-specific and it should be easily solvable, let me work a bit on the transpile.js, will get back to you on this shortly. Just need to add a few exceptions for the dot operator.

Thanks @kroitor, I see that you don't create classes into .js transpilable code of exchanges, so I supposed that it could be hard to implement. I have tried to avoid it in this branch. Anyway, I think it is more important now that other contributors try to implement other websocket orderbook exchange and get feedback from them. If they like the way we are implementing websockets, then we could add classes proposed by @pursehouse

I see that you don't create classes into .js transpilable code of exchanges, so I supposed that it could be hard to implement

Yep, you're absolutely right, I thought the same initially... I've just reviewed it and tried to add a quickfix, but... it didn't work well ) I'm now looking for a way to properly transpile the dot without escaping it or tweaking the code somehow, purely by means of the transpiler... I think i'm going to need a bit more time, but I'll figure it out and will let you know asap.

Add websocket api doc link in table list for some exchanges:
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/README.md

Lfern are you on telegram or discord? I have some questions.
On Thu, Jul 12, 2018 at 9:01 AM lfern<EMAIL_ADDRESS>wrote:

Add websocket api doc link in table list for some exchanges:
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/README.md
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ccxt/ccxt/issues/56#issuecomment-404563014, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFQFAllW7QM1RwVI_U-3BTGoV3LqumQOks5uF3LRgaJpZM4OjakB
.

sorry @skeller88 . I don't have telegram/discord account for now. If you don't mind, write them here and I try to answer quickly.

@GoChartingAdmin, would you mind to post your websocket exchange list to add to this list?

@lfern I'll try and add Binance support on binance-websocket-support, a
branch that I forked off lfern/feature/websockets-multiple
https://github.com/skeller88/ccxt/tree/feature/binance-websocket-support.
Then I'll submit a PR against websockets-multiple. Does that work?
I'm trying to run examples/py/websocket-orderbook.py using "binance" and
seeing the below error. The self.wsconf Dict is empty at runtime. There's
already some code in binance.js so was the websocket-orderbook.py ever
working for you using binance? I'll assume it wasn't.
Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py",
line 1664, in <module>
    main()
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py",
line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py",
line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm
CE.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in
execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File
"/Users/shanekeller/Documents/lfern_ccxt/examples/py/websocket-orderbook.py",
line 81, in <module>
    loop.run_until_complete(main())
  File "/Users/shanekeller/anaconda/lib/python3.6/asyncio/base_events.py",
line 467, in run_until_complete
    return future.result()
  File
"/Users/shanekeller/Documents/lfern_ccxt/examples/py/websocket-orderbook.py",
line 61, in main
    await exchange.websocket_subscribe('ob', symbol, {'limit': limit})
  File
"/Users/shanekeller/Documents/lfern_ccxt/python/ccxt/async/base/exchange.py",
line 712, in websocket_subscribe
    raise ExchangeError('Not valid event ' + event + ' for exchange ' +
self.id)
ccxt.base.errors.ExchangeError: Not valid event ob for exchange binance
binance requires to release all resources with an explicit call to the
.close() coroutine.
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10c483dd8>

On Fri, Jul 13, 2018 at 9:08 AM lfern<EMAIL_ADDRESS>wrote:

@GoChartingAdmin https://github.com/GoChartingAdmin, would you mind to
post your websocket exchange list to add to this list?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ccxt/ccxt/issues/56#issuecomment-404878798, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFQFApUvfY_oNwHXIIOjxPXdP1Bel4lYks5uGMYBgaJpZM4OjakB
.

Sorry @skeller88 . I am not commiting changes of .py and .php transpiled files. Maybe I would have to commit these files too. You have to do npm install and then npm run-script transpile to get transpiled binance.py file. Then websocket-orderbook.py  would work

After running transpile, websocket-orderbook.py works. I see binance
already has websocket order book support, so I'll try and add websocket
orderbook support for bittrex. Their websocket is in beta so if I can't get
that working, I'll add it for gdax. Sound good?
Have any tests been written? I'm not finding any yet.
Are you waiting to add the python and php code until lfern/ccxt is approved
for merge into ccxt/ccxt?
On Fri, Jul 13, 2018 at 2:23 PM lfern<EMAIL_ADDRESS>wrote:

Sorry @skeller88 https://github.com/skeller88 . I am not commiting
changes of .py and .php transpiled files. Maybe I would have to commit
these files too. You have to do npm install and then npm run-script
transpile to get transpiled binance.py file. Then websocket-orderbook.py
would work
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ccxt/ccxt/issues/56#issuecomment-404957677, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFQFAhc61obJP9u6Y9Yu5iqWP6-s-0Htks5uGQ_XgaJpZM4OjakB
.

I think Gdax websocket doc is better than bittrex, but we are still testing, so any exchange could be interesting te check if its easy to implement. The only restriction it not to implement exchanges that uses pusher protocol (like bitstamp), or pubnut.com, Only "raw" websocket connections is supported in this branch.

Have any tests been written? I'm not finding any yet.

Sorry, I think it is not easy to write tests that simluates a websocket connection. I don't think about it, but I suppose we can create a Websocket subclass that emulates a websocket connection, but this class would be created for each exchange... Maybe someone in this thread could give us some ideas about it,.

Are you waiting to add the python and php code until lfern/ccxt is approved
for merge into ccxt/ccxt?

Yes, I thought that this could be commited when this branch would be merged. But if you think that commiting the transpiled code makes life easier anyone who wants to test it in php or python I will commit this code too! Maybe @kroitor could tell us if this would be a good idea, or it is better not to commit transpiled code for now.

yeah so there should be a generic concept for building the opening url, which needs ability to accept objects to build the url. as well as needs the socket calls for adding subscriptions as a generic concept that accepts objects.
then separately those all those object types need their own central structure that each exchange can extend to format to and from the ccxt format

Yes, you are right @skeller88 . Variable interpolation is not really usefull for now, because current websocket implemented exchanges except binance have only one websocket for all events. I think we need something that Exchange.js can use to know what to do when it receices a subscription/unsubscription command on event/symbol. Exchange.js manages websocket connections, so we need something in each exchange that let us know what to do on every subscription command:

when multiple channels in only one websocket: on subscripion and if websocket is not opened then open websocket, reset context. If still opened do nothing, on unsubscription close websocket (or not) if no more subscription are active, and reset websocket context.
when multiple channels on one websocket, but differents url connections for each channel (binance): on subscription/unsubscription you have to reconnect to websocket using a different URL (generated with previous subscriptions/unsubscriptions), and reset websocket context. On unsubscription if is the last stream active, close websocket.
when pusher: every event/symbol has its own websocket connection info, so we need something that maps event/symbol to pusher configuration.
when every event or event/symbol corresponds to a different websocket: on subscription open websocket for event/symbol, reset context; on unsubscription, close websocket for event/symbol.

I don't still find a exchange for this last case, but variable interpolarion would be useful to get source websocket for that event/symbol info.
Maybe it would be better to take out this from config and create methods, invoked from Exchange.js, that provides this information to manage websocket connection pool.

@lfern that's why I would suggest focusing on the individual features needed for processing data that is sent or received with the socket; rather than solving how to use sockets in general.
That way the requirements can be split into phases, and potentially releasing the core functionality features with examples on how to use them, can be followed with a future ( and much more complicated problem ) of a generic way to call the sockets. Which will get much more community support because the building blocks will exist.
Cuz whatever socket style you create for JS, now there is also a new socket style needed for PHP, and Python. So forcing the release to include a full socket solution means those also need solving. and for PHP, that would be complex to say the least.. because it's not event driven like JS is, so in order for me to send the responses into my database or whatever after CCXT processes it, I need to be along the lines of...

\Amp\Loop::run( function () use ( $streamsUrl, $exchange ) {
    /** @var \Amp\Websocket\Connection $connection */
    $connection = yield Websocket\connect( $streamsUrl );
    while( $message = yield $connection->receive() ) {
        $payload = yield $message->buffer();
// this decode line would be replaced with the CCXT decoder for Binance
        $decode = json_decode( $payload );
// this switch would use the CCXT generic format for a socket packet
        switch( $decode->data->e ) {
            case "kline":
// methods do save my data
            break;
        }
        yield new Delayed( 1000 );
    }
} );

so if you solve the building blocks, then people can implement their systems to use it. and perhaps a generic socket solution that can pass packets to the developers system will emerge over time.

@pursehouse to me the above just boils down to a proper and flexible method decomposition. However, I'd agree that doing async things with PHP can be tricky, might even require a threading solution... But anyways, doing it in a language-specific or exchange-specific way would be even harder to maintain or transpile.

@kroitor right so don't try to solve the socket abilities at all right now, just focus on building blocks:

url generation
packet creation
packet processing

cuz right now it seems a lot of time is going into how to do sockets in general, which isn't going to solve across all languages. And seems to be the most difficult aspect as far as making it a generic solution, and even if the generic sockets are solved for JS, then all those building blocks are still required to be able to function across all exchanges. So why not do building blocks first, and maybe include some example code on how to use them. Then afterwards a generic socket solution can be worked on and most likely have a larger scale of contributors.

@lfern Will do by tomorrow EOD

@lfern Was going through list - you have added Bistamp.  Bitstamp currently uses Pusher.  I thought Pusher was out of scope at this stage

@GoChartingAdmin , yes, you are right. I think we could add all exchanges with websocket and maybe add a column to point out that pusher protocol is not available. What do you think?

@lfern I agree. That will take out Bitstamp and Quionex

Thanks guys for working on WS! Do you need any help to make it happens faster? Also do you have an ETA when this will be ok? My company really needs this feature and we can allocate 1 or 2 developers if necessary.

@GoChartingAdmin Added "Ws Protocol" clumn in readme table.

@backnight in recent posts some changes has been proposed. But I think it would be better other people could write code for other exchanges so we could realize current ws implementation is really a good implementation. I don't know if anyone is trying to do this, or modifying this implemention in some way, but I am waiting for some feedback about it. If this implementation is accepted (with any changes proposed) I think that new exchanges could be added really fast.

@lfern
at branch https://github.com/lfern/ccxt/tree/feature/websockets-multiple
this is like to happen in a slow network when connect to okex.
it is strange, that the socket is still not open when invoke websocketSubscribe.
Error Stack:
Error: not opened I20180905-18:02:28.011(8)? at WebSocket.send (node_modules/ws/lib/WebSocket.js:360:18) I20180905-18:02:28.011(8)? at WebsocketConnection.send (ccxt-websocket/js/base/websocket/websocket_connection.js:77:28) I20180905-18:02:28.012(8)? at WebsocketConnection.sendJson (cxt-websocket/js/base/websocket/websocket_base_connection.js:37:14) I20180905-18:02:28.012(8)? at okex.websocketSendJson (ccxt-websocket/js/base/Exchange.js:1866:31) I20180905-18:02:28.012(8)? at okex._websocketSubscribe (ccxt-websocket/js/okex.js:295:14) I20180905-18:02:28.012(8)? at Promise.asyncApply (ccxt-websocket/js/base/Exchange.js:2092:14)
this is not going to be easily reproduced.

@kroitor  Thanks for the awesome project. Is there any example showing how to get trades via websocket?
And which exchanges are supported till date

@kroitor Thanks for the awesome project. Is there any example showing how to get trades via websocket?
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/examples/js/websocket-orderbook.js

And which exchanges are supported till date
https://github.com/lfern/ccxt/tree/feature/websockets-multiple#supported-cryptocurrency-exchange-markets

Hey that seems to be a lot of exchanges...Are all of them supported via WS ?

@lfern hi，could you please add an example of binance websocket fetch orderbook ?

Hi @djoffrey  , in examples/js/websocket-orderbook.js you can find an example to fetch orderbook. Execute it with these parameters and you could fetch and get events from orderbook info:
node examples/js/websocket-orderbook.js binance 'no-apikey-required' 'no-key-required' 10 ETH/BTC

First parameter is the exchange id, second/third are api-key/secret (not required by binance), fourth is the number of bids/asks to get and fifth and following are the symbol to get.

Hey guys, and how about FIX API ? CoinbasePro supports it and it would open the doors to classical (stock/...) trading exchanges. http://en.wikipedia.org/wiki/Financial_Information_eXchange Dont know which other Crypto-Exchanges support this standard.

I hate to be that guy, but any progress on this? Do not get me wrong, I think it is amazing that you are giving away such an amazing project for free, but the websocket implementation was supposingly almost done over a year ago.
Simply using a HTTP request in a loop as in some examples is not a good solution for my project.
So is there any ETA for this?
Once again, thanks for the great project :)

@bramvbilsen Maybe you need to start implementing it in ccxt. I guess @kroitor and all other developers would be happy if you can draft how you would like it to implement with ccxt (giving callback function to call instead of returning a promise I guess). As soon as I get to the FixAPI (have also lots of other tasks on my list) I will probably start on this. But like the "one year ago" issue, it will also take some time. ;-)

@TechupBusiness Do not get me wrong, I would not mind doing it. But the reason I was looking at ccxt is because I do not have a lot of time at all unfortunately.

@bramvbilsen Thats probably the reason why most of us landed here: Saving time as its the most scarce resource. :)

@TechupBusiness Exactly!

If your project needs this and the exchange(s) have websocket support, then why not write your own wrapper. It's a lot easier than writing something that fits within a unified abstraction layer which the folks at ccxt are trying to do. There are plenty of implementations already for popular exchanges like binance, bitfinex, bitstamp, etc. There's no reason why you should be looping over HTTP requests if you have a websocket api available to you

@soliveira-vouga I am not doing that, I am currently using the websocket implementations of different exchanges. But I thought I saw that that is what the examples in this repo do to get "live" data.

Can the feature/websockets-multiple branch be used from python?

@lfern I am trying to figure out how the python part works. Where is the config for websocket url for each exchange? and the events available?
ask exception was never retrieved
future: <Task finished coro=<main() done, defined at websocket-playground.py:19> exception=ExchangeError('Not valid event orderbook for exchange binance',)>
Traceback (most recent call last):
File "websocket-playground.py", line 59, in main
await exchange.websocket_subscribe(event, symbol, params)
File "/home/user/crypto/ccxt/python/ccxt/async_support/base/exchange.py", line 773, in websocket_subscribe
raise ExchangeError('Not valid event ' + event + ' for exchange ' + self.id)
ccxt.base.errors.ExchangeError: Not valid event orderbook for exchange binance
thanks

@rg687  I am not commiting PHP and Python generated files from transpilation process yet. You must execute npm run transpile in base ccxt folder to get them. You will find a `wsconf' configuration structure with valid events ('ob' for orderbook)

@rg687 maybe you must execute npm run export-exchanges before transpilation to ensure exchange.json is generated.

@rg687 I am not commiting PHP and Python generated files from transpilation process yet. You must execute npm run transpile in base ccxt folder to get them. You will find a `wsconf' configuration structure with valid events ('ob' for orderbook)

@lfern I just tested with python and it worked nice with a few exchanges I tested. Good job man!
I see many major exchanges have been implemented.
Ideas when bitfinex will be implemented?

added websocket orderbook for therock, theocean, bitfinex2

@lfern May I ask when do you plan to merge 'websockets-multiple' to the master?

@AlexeyZelenin I don't really know if this development is mature enough to make a push request to master. If ccxt owners think so, I would add a push request to master. But I think they have to check if code developed and dependencies added could be merged on master branch.

@lfern great job about bitfinex2, I just tested it and it works great.
I also tested cex (authenticated WS).
May I report a few issues I've found? I've tested a few websocket examples in examples/py:
btw I think it may be a good idea to put a few examples with usage (feel free to copy/adapt the following) in the WEBSOCKETS.md file, noob friendly.

websocket-poloniex-orderbook.py, (at line 11 replace import ccxt.async with import ccxt.async_support) usage ./websocket-poloniex-orderbook.py works
websocket-playground.py, usage ./websocket-playground.py bitstamp ob BTC/EUR works like a charm
websocket-orderbook.py, (at line 11 replace import ccxt.async with import ccxt.async_support), usage:

one symbol: ./websocket-orderbook.py bitstamp None None 5 BTC/EUR
multiple symbols ./websocket-orderbook.py bitstamp None None 5 BTC/EUR XRP/EUR

After around 5 few seconds I get this error:
Traceback (most recent call last):
  File "/opt/projects-python/ccxtws/examples/websocket-orderbook.py", line 81, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 473, in run_until_complete
    return future.result()
  File "/opt/projects-python/ccxtws/examples/websocket-orderbook.py", line 74, in main
    await exchange.websocket_unsubscribe('ob', symbol)
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/base/exchange.py", line 825, in websocket_unsubscribe
    self._websocket_unsubscribe(conxid, event, symbol, oid, params)
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/bitstamp.py", line 941, in _websocket_unsubscribe
    self.websocketSendJson(payload)
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/base/exchange.py", line 669, in websocketSendJson
    websocket_conx_info['conx'].sendJson(data)
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/websocket/websocket_base_connection.py", line 31, in sendJson
    self.send(json.dumps(data))
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/websocket/pusher_light_connection.py", line 172, in send
    if json_data['event'] == 'subscribe':
KeyError: 'event'

./websocket-orderbook.py binance None None 5 NANO/BTC BTC/USDT: seems to work well, but after around 30s it fails with KeyError:'limit'.
I think the lines await asyncio.sleep(5) are responsible for this. If I increase e.g. to 20, it will work 20 seconds. Is it possible to modify the example so that it works forever?
Also I was expecting multiple websockets to work simultaneously, which is not really the case.
Please fix all the ccxt.async with ccxt.async_support in the examples if there are other leftovers.
Thanks again for the great work. I was really looking forward to have a good library supporting many exchanges and this does the job.

Thanks to you @lfern for the quick fixes, I appreciate.
I noticed another thing, testing multiple websockets on the same exchange, e.g:
./websocket-orderbook.py bitfinex2 None None 5 BTC/USDT XRP/USDT
I start seeing BTC/USDT updates immediately, but before seeing anything about XRP/USDT I have to wait around 5 seconds.
I think this is caused by the await asyncio.sleep(5) is this really needed?
If I try with 3 pairs:
./websocket-orderbook.py bitfinex2 None None 5 BTC/USDT XRP/USDT
Same: BTC/USDT shows up immediately, after 5 seconds I see XRP/EUR updates, then I have to wait another 5 seconds to see ETH/EUR updates. Like this, imagine you want to monitor 11 pairs, you'll see updates of the last pair only after 50 seconds. Is this a sort of rate limiting for websockets?
Also, in the websocker-orderbook example, it seems that the websocket is unbsubscrbed and re-subscribed each 5 seconds. Something doesn't look correct in this example.
I tried to comment the first await asyncio.sleep(5), but then it seems to only subscribe the first pair.
I tried to put a lower value, 1 second: await asyncio.sleep(1) so the delay now is very low.
I did put a high value for the second sleep: await asyncio.sleep(2000000) like this it's unsubscribed only later. Has this an implication? What happens if the websocket gets disconnected earlier? 5 seconds was a way to make it resubscribe in such an event? Is there an event fired if a subscription to a websocket doesn't work anymore? Would be nice to have an example to handle a websocket disconnection properly, e.g. try to re-subscribe after 2 minutes when the websocket connection is lost.
Thanks for your feedback

added live orderbook market table example:
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/examples/js/websocket-orderbook.js
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/examples/py/websocket-orderbook.py

@firepol, binance uses streams in websocket connection. You have to open websocket with all symbols you want to observe: wss://stream.binance.com:9443/stream?streams=//
First time you try to subscribe websocket library open websocket with only one stream:wss://stream.binance.com:9443/stream?streams=NANOBTC@depth
while waiting for first subscription succeeds, if you try to subscribe to second symbol, binance ws driver need to close previous connection and open websocket with a new URL: wss://stream.binance.com:9443/stream?streams=NANOBTC@depth/ETHBTC@depth, and the first websocket open action is closed before it succeeds.
I don't know how to handle it correctly, maybe before closing binance driver would have to wait to first subscription succeeds. but close function is not async, and we need to be sure previous ws connection is closed before connecting with new URL. For the time being, you must not subscribe in paralell on same exchange

Hi there, after some more testing I actually developer my little own project and open sourced it here: https://github.com/firepol/ccxt-websockets-db-updater
@lfern I have trouble connecting to more than one pair at binance, maybe you can help to fix that? From what I've seen in my other tool I coded (also based on another library, so I'm totally no expert at all in websockets), binance and GDAX need a "product registration" before subscribing to each order book. I don't know what that means, but that may be why binance doesn't work if I try to connect in parallel as I did in my program.
See: firepol/ccxt-websockets-db-updater#1

@lfern thxfor your reply, btw I noticed a small thing with bitfinex2, the timestamp is missing.
For binance to allow multiple pairs, maybe you can get some inspiration by the xchange-stream library, even if it's java, maybe some ideas can be useful for CCXT too? in particular check ProductSubscribtion. See e.g. how I use it (I'm no expert, somebody helped me to make it work) here:
https://github.com/firepol/crypto-websockets/blob/dev/src/main/java/com/github/firepol/cryptows/ExchangeManager.java
See in processWebsockets:
        pairsByExchange.forEach((exchangeName, pairsCollection)->{
            if (Arrays.asList(NEED_PRODUCT_REGISTRATION).contains(exchangeName)) {
                pairsCollection.pairs.forEach(pair->{
                    StreamingExchange exchange = getStreamingExchange(exchangeName);
                    ProductSubscription productSubscription = ProductSubscription.create()
                        .addOrderbook(pair)
                        .build();
                    exchange.connect(productSubscription).blockingAwait();
                    subscribeOrderBook(exchange, pair);
                });
            } else {
                StreamingExchange exchange = getStreamingExchange(exchangeName);
                exchange.connect().blockingAwait();
                pairsCollection.pairs.forEach(pair->subscribeOrderBook(exchange, pair));
            }
        });

Basically the first if checks if the exchange name is in the NEED_PRODUCT_REGISTRATION array (which contains gdax, binance, maybe other exchanges will show up) and if so connects to the websocket passing the product subscription. How this is handled I don't know exactly, but let's have a look at the source code here:
Here the ProductSubscription class
Here how it is used e.g. in BinanceStreamingExchange.java
    private BinanceStreamingService createStreamingService(ProductSubscription subscription) {
        String path = API_BASE_URI + "stream?streams=" + buildSubscriptionStreams(subscription);
        return new BinanceStreamingService(path, subscription);
    }

So far so good as you already mentioned, how this is handled in xchange-stream I'm not sure, I see that the message is handled in the super class, maybe you can have a look and have better luck in understanding it?

Hi @lfern during more testing (tryigm to keep many connections alive for a longer period) I found more issues:

bitstamp, ws for btc/usd seems to subscribe, but no values are returned. See bitstamp websocket documentation, for any other pair than btc/usd, the channel name is order_book_{currency_pair}, but for btc/usd it's not order_book_btcusd, but simply order_book.
bitstamp: datetime is always: 1970-01-18T20:58:45.806Z
bitfinex: datetime & timestamp are both null
bitfinex: volume values for asks are returned with negative values, e.g. -240, expected: 240
cex seems to be quite unstable: I tested many connections and cex is the one that disconnect the most. Cex websockets need to be authenticated. If you want to test this and don't have an account, you can register at cex.io I know it may not be an issue of your implementation, but it would be nice to have an example showing how to reconnect. I'm working on it on python and could not manage yet to find a solution for this.

In general, double check timestamp & datetime, so far really good work!
Thanks again and cheers

Thanks @firepol for feedback.

bitstamp datetime and btc/usd subscription problems are fixed
I can't find timestamp info in bitfinex orderbook messages. So I prefer not to fill these values with local time.
bitfinex negative values is fixed too.
I will add a new example with reconnection example. Maybe we have to change error reporting with more detailed error messages.

Hi @lfern ,
I checked your last commit but the commit message is misleading as it states "bitfinex", but the file changed it bitstamp.js. I think you forgot to include bitfinex changes or confused bitstamp with bitfinex changes you were about to commit next?
About handling disconnections, I tried to attempt something in my project (which btw saves data in a database already, I use Postgres, but you can test also with SQLite). I created a custom exception (WsError), which I raise in: https://github.com/firepol/ccxt-websockets-db-updater/blob/master/utils.py:
    @exchange.on('err')
    async def websocket_error(err, conxid):  # pylint: disable=W0612
        error_message = type(err).__name__ + ":" + str(err)
        error_stack = traceback.extract_stack()
        logging.error(f'{exchange.id}: {error_message}')
        logging.error(error_stack)
        print(f'{exchange.id}, {datetime.datetime.now()}, {error_stack}')
        await exchange.close()

And catch it here: https://github.com/firepol/ccxt-websockets-db-updater/blob/master/ob_updater.py
    except WsError as wse:
        print(f'Canceling: {wse}')
        ob_subscriptions[wse].cancel()

I'm sure I'm doing something wrong, I'm still a noob in async programming in python.
Ideally I'd like in the ob_updater.py program to catch the exception per exchange (that's why I added the tasks in a dictionary, as follows:
            # make a list of tasks by exchange id
            ob_subscriptions[exchange.id] = asyncio.ensure_future(utils.subscribe_ws('ob', exchange, symbols, limit,
                                                     pp, args.debug, args.verbose, session))

The idea was then to properly cancel the task (which I think is not being canceled correctly now as it complains it had to be awaited) and then create a new one of the same exchange. That would be the best.
If you can include that in your repo examples then I'll try to learn from there and apply to my separate project. Cheers

added new examples for recovering from websocket error.

websocket-recover-connection.js
websocket-recover-connection.py

@firepol, what do you think about recovering process? Maybe it could be easier.

Awesome work & branch @lfern, many thanks.
Regarding 'ob', is it possible to get only the first snapshot and then the updates as they come, not everything (e.g. with an option)? I see in the code (at least, when mid-way printing the received messages from the exchange) that the data patching is done by the ccxt code.
Many thanks

Thanks @maayank, I think we need to keep a copy of orderbook inside ccxt, so you can get a live orderbook copy when invoking websocketFetchOrdebook method. It could be posible to add an adidtional attribute to 'ob' event object with the last changes received (like 'info' ccxt attribute with original response received from exchange in CCXT REST methods). But some exchanges always sends a snapshot  (e.g. bitstamp). In these cases ccxt only replaces internal orderbook copy with new one.
It is not difficult to add this new parameter 'info' with last asks/bids received (price/amount), or we could define a new event 'rawob' that send only snapshot and updates, But before adding this I would like to know what other people think about it.

Regarding 'ob', is it possible to get only the first snapshot and then the updates as they come, not everything (e.g. with an option)? I see in the code (at least, when mid-way printing the received messages from the exchange) that the data patching is done by the ccxt code.
Many thanks

@lfern is your script here (https://github.com/lfern/ccxt/blob/feature/websockets-multiple/examples/py/websocket-orderbook.py) can be executed standalone ?

added cobinhood websocket support: orderbook/trade/ticker/ohlcv

@mishaker  Yes, you can execute this example to print simple live orderbook bids/asks table. It is a console script, and I have only tested in windows/cygwin, but I think it would have to work on linux console.

@lfern is your script here (https://github.com/lfern/ccxt/blob/feature/websockets-multiple/examples/py/websocket-orderbook.py) can be executed standalone ?

@lfern maybe a noob question but I have this error while trying to execute your script:
Task exception was never retrieved
future: <Task finished coro=<subscribe() done, defined at example.py:67> exception=AttributeError("'binance' object has no attribute 'on'",) created at example.py:64>
source_traceback: Object created at (most recent call last):
  File "example.py", line 194, in <module>
    main()
  File "example.py", line 64, in main
    asyncio.ensure_future(subscribe (ex, exchange['symbols'], config['symbolDefaults']), loop=loop)
Traceback (most recent call last):
  File "example.py", line 69, in subscribe
    @exchange.on('err')
AttributeError: 'binance' object has no attribute 'on'

Maybe I should include something before running the script ?

added upbit websocket support: orderbook/trade/ticker

@Fcl69  please check if exchanges.json file has been generated in the ccxt folder. If so, try again
npm run transpile

It's totally awesome to see what you guys achieve here - thank you for that.
How do you all (especially @lfern and @kroitor) think about going bigger with that project? It's a bit exhausting to see this all handled in one issue thread. Maybe a wiki page with what's working already and what isn't and maybe a rough howto or something could be helpful.
Again thanks for your hard work!

I did, plus transpile again, but same stuff:

v:ccxt fcl$  python3 examples/py/websocket-playground.py kraken ob ETH/XTC limit:1
Task exception was never retrieved
future: <Task finished coro=<main() done, defined at examples/py/websocket-playground.py:19> exception=AttributeError("'kraken' object has no attribute 'on'",)>
Traceback (most recent call last):
File "examples/py/websocket-playground.py", line 43, in main
@exchange.on('err')
AttributeError: 'kraken' object has no attribute 'on'
kraken requires to release all resources with an explicit call to the .close() coroutine. If you are creating the exchange instance from within your async coroutine, add exchange.close() to your code into a place when you're done with the exchange and don't need the exchange instance anymore (at the end of your async coroutine).
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10a3163c8>

=(

Sorry  @Fcl69 . I have downloaded this branch on a clean linux distro and with these steps I have configured my development enviroment:
git clone https://github.com/lfern/ccxt.git
cd ccxt/
git checkout -t origin/feature/websockets-multiple
npm install
pip3 install -e python/
npm run transpile
npm run export-exchanges
npm run transpile
python3 examples/py/websocket-playground.py kraken ob ETH/XBT

I have had to upgrade from python3.5 to python3.6, and install pip3 to this python version, but if python3.6 or above is installed on your system, it should work

@Fcl69 I just checked my project (you linked my other project in java btw, the correct link is https://github.com/firepol/ccxt-websockets-db-updater ), fixed a few issues affecting ob_tester.py and made a demo video that shows it in action https://www.youtube.com/watch?v=0UIvIBUiqQQ
Maybe in the readme it was not clear, but if you specify no --verbose and no --debug option, there is no output. Sorry if that was not clear. So, just try to set it up as in the video, from scratch. If you use linux as I do, you can just copy/paste the installation code block.
Also note, ob_tester.py doesn't populate the DB: it just connects to the websockets and does nothing.
ob_updater.py populates the DB.
Btw, in case you have trouble using my ccxt-websockets-db-updater project, feel free to write an issue in my repository https://github.com/firepol/ccxt-websockets-db-updater and I'll try to help you to get things up and running.
Also, I keep seeing the same issues, that people forget to transpile etc. and in fact I created already a transpiled repository a long time ago, to make it easier to get started (I use it as dependency in my ccxt-websockets-db-updater program): https://github.com/firepol/ccxt-websockets note that this is not suitable if you plan modifying the CCXT project, it's meant "read only" just to "consume" lfern's work ;)

@lfern I tried huobipro using my project (feel free to try it, maybe you can use it to test your changes in python in a more advanced scenario), but it doesn's seem to work. I get no output.
See instructions at https://github.com/firepol/ccxt-websockets-db-updater (see also my video)
Working connection (with output):
./ob_tester.py -e binance -s eth/btc --verbose --debug

Huobipro (it doesn't need api keys, or does it?), not working, maybe can you have a look at it?
./ob_tester.py -e huobipro -s eth/btc --verbose --debug

Expected: see websockets output, as in binance. Result: nothing...
Ideas?

@lfern @firepol I was able to launch script using a different setup.
Thank you very much for your help. I'll keep digging.

@firepol Huobipro has some transpilation erros and exchange.py gunzip method was wrong. It seems now it is working well.

Hey @lfern , two issues with poloniex:

In poloniex.js:1191, there's an error that goes over to python where channelId isn't casted to string and contextId isn't passed to emit(). So, it should be:
1191                 this.emit ('err', new ExchangeError (this.id + '._websocketOnMessage() failed to get symbol for channelId: ' + channelId.toString ()), contextId);

I've encountered the above as part of another issue where I can't subscribe to BTC/USD or BTC/USDT. I get, for example:
[default]ExchangeError ^^^poloniex._websocketOnMessage() failed to get symbol for channelId: 121

Let me know if there's anything else I can check.
Cheers

Thanks so much for the effort that's gone into this branch!  Is there a readme for the origin/feature/websockets-multiple branch?
I have got it working, but was wondering what the API was?  Specifically,

what are the valid values for the event param for websocketSubscribe? (looks like 'ob', is anything else supported?
how to use websocketFetchOrderBook method?
the data in the orderbook callback for bitfinex has the entire order book.  Is this a common format for all exchanges?

Also, I noticed the updateBidAsk method looks a problematic in that the bids/asks are stored in an array.  For busy orderbooks it's going to be doing a lot of searching to make sure the items are stored in the right order.  I think replacing the array with a binary tree  would be a much more efficient solution.

Hi @npomfret all credits go do @lfern and other contributors, I'm just a "consumer" of this branch too. But let me help you since I have some minutes to use ;)
Yes there is a readme and I believe you can find some answers by reading it:
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/WEBSOCKETS.md
Ws Features:

ob: orderbook
ti: tickers
tr: trades
oh: ohlcv

Other questions I can't help, but until then you have something to read. Hope this helps, cheers

Thanks @maayank , let me check out

Hey @lfern , two issues with poloniex:

In poloniex.js:1191, there's an error that goes over to python where channelId isn't casted to string and contextId isn't passed to emit(). So, it should be:
1191                 this.emit ('err', new ExchangeError (this.id + '._websocketOnMessage() failed to get symbol for channelId: ' + channelId.toString ()), contextId);
I've encountered the above as part of another issue where I can't subscribe to BTC/USD or BTC/USDT. I get, for example:
ExchangeError: poloniex._websocketOnMessage() failed to get symbol for channelId: 121

Let me know if there's anything else I can check.
Cheers

@npomfret, websocketFetchOrderBook function returns a copy of the current internal orderbook managed by exchange. I think this function is not the best way to get orderbook info, it is better to listen 'ob' events emited as it receives updates from websocket connection. You can find some examples in examples folder in python, js and php language (files with "websocket-" prefix).
About orderbook format, I think that all of them has a similar format. Some of them let you to choose the depth so you receive updates up to this depth. In some cases you have to get a snapshot using REST API and receive update from websocket connections ... Anyway, each exchange implementation try to manage an internal copy of exchange orderbook and emit 'ob' events to subscriber when any update is received.
And yes, You are right about how to update askd/bids in the ordered array. From the begining this was a task that we had to improve, but we wanted to check if the way we were implementing websockets would work fine.

Thanks so much for the effort that's gone into this branch! Is there a readme for the origin/feature/websockets-multiple branch?
I have got it working, but was wondering what the API was? Specifically,

what are the valid values for the event param for websocketSubscribe? (looks like 'ob', is anything else supported?
how to use websocketFetchOrderBook method?
the data in the orderbook callback for bitfinex has the entire order book.  Is this a common format for all exchanges?

Also, I noticed the updateBidAsk method looks a problematic in that the bids/asks are stored in an array. For busy orderbooks it's going to be doing a lot of searching to make sure the items are stored in the right order. I think replacing the array with a binary tree would be a much more efficient solution.

@maayank I think poloniex is fixed. Please check it out

Hey @lfern , two issues with poloniex:

In poloniex.js:1191, there's an error that goes over to python where channelId isn't casted to string and contextId isn't passed to emit(). So, it should be:
1191                 this.emit ('err', new ExchangeError (this.id + '._websocketOnMessage() failed to get symbol for channelId: ' + channelId.toString ()), contextId);
I've encountered the above as part of another issue where I can't subscribe to BTC/USD or BTC/USDT. I get, for example:
ExchangeError: poloniex._websocketOnMessage() failed to get symbol for channelId: 121

Let me know if there's anything else I can check.
Cheers

Thanks @lfern and @firepol
I've used binary trees to implement orderbooks before, its very simple.  I  think the package I mentioned  is the one that coinbasepro uses on their site.

Works well. As always, many thanks @lfern :)

hi, I'm trying to find some examples of how to connect to the websocket for bitmex in python. I'm happy to help add functionality for exchanges if needed. which branch should I start from for the most up-to-date python examples?

Hi @furryboffin check this out: https://github.com/lfern/ccxt/tree/feature/websockets-multiple
Examples etc. read here: https://github.com/lfern/ccxt/blob/feature/websockets-multiple/WEBSOCKETS.md
Enjoy

Hi,
Encountering 'CloudFlare' error, when letting the code run for several hours. This was reproduced multiple times. I'm running python's example code:

examples/py/websocket-playground.py kraken ob ETH/XBT

Kindly asking if there is an implemented (or planned) mechanism intended to prevent this? (I've generally read about a requirement for some keepalive, e.g. ping?)
When it does occur - What should be the recommended approach to handle this exception? Should an unsubscribe and subscribe again be good enough?

NetworkError:CloudFlare WebSocket proxy restarting
File "examples/py/websocket-playground.py", line 64, in  loop.run_forever()
File "/usr/lib/python3.6/asyncio/base_events.py", line 427, in run_forever self._run_once()
File "/usr/lib/python3.6/asyncio/base_events.py", line 1440, in _run_once handle._run()
File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run self._callback(*self._args)
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/autobahn/asyncio/websocket.py", line 100, in connection_lost self._connectionLost(exc)
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/autobahn/websocket/protocol.py", line 3365, in _connectionLost WebSocketProtocol._connectionLost(self, reason)
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/autobahn/websocket/protocol.py", line 1105, in _connectionLost self._onClose(self.wasClean, self.remoteCloseCode, self.remoteCloseReason)
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/autobahn/asyncio/websocket.py", line 181, in _onClose res = self.onClose(wasClean, code, reason)
File "/home/user/Projects/ccxt/python/ccxt/async_support/websocket/websocket_connection.py", line 43, in onClose self.event_emitter.emit('err', Exception(reason))
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/pyee/init.py", line 151, in emit
result = f(*args, **kwargs)
File "/home/user/Projects/ccxt/python/ccxt/async_support/base/exchange.py", line 782, in websocket_connection_error self.emit('err', NetworkError(error), conxid)
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/pyee/init.py", line 151, in emit result = f(*args, **kwargs)
File "examples/py/websocket-playground.py", line 47, in websocket_error traceback.print_stack()
Closing Loop
after complete

I would like to contribute, either through debugging or if you can hint me towards the general direction of investigating this...
Cheers!

Hi @aviad21 . In theory Kraken sends a message every second (https://www.kraken.com/features/websocket-api#message-heartbeat) and it seems that CloudFlare could close sockets after 100 or 1000 seconds of inactivity, so this error should not take place. Maybe we can include a ping request if we don't receive any packet in a short period of time.
Anyway, you can catch exceptions emited from exchange and decide what to do then:

always try to reconnect inmmediately
wait some period of time if previous error was raised a short time ago.

Take a look at examples/py/websocket-recover-connection.py and websocket-recover-connection2.py. These scripts simulates a NetworkError and wait 5 seconds before recovering websocket connection. Basically:
 @exchange.on('err')
    async def websocket_error(err, conxid):
        # log error
...
        # close websocket if necesary
        exchange.websocketClose(conxid)
       # analyze what to do now
        if isinstance(err, ccxt.NetworkError):
            # wait 5 seconds
            await asyncio.sleep(5)
            try:
                # clean current context data
                exchange.websocketCleanContext(conxid)
                # subscribe all symbols again
                await doSubscribe(exchange, symbols, {
                    'limit': limit,
                })
            except Exception as ex:
                print(ex)
                sys.stdout.flush()
        else:
...

Or you can try a new exchange method that try to reconnect to websocket ans subscribe again to subscribed symbol (it still is not fully tested)
async def websocket_error(err, conxid):
        #log error
        if isinstance(err, ccxt.NetworkError):
            # close if necesary
            exchange.websocketClose(conxid)
            #wait 5 seconds
            await asyncio.sleep(5)
            try:
                await exchange.websocketRecoverConxid(conxid)
            except Exception as ex:
                print(ex)
                sys.stdout.flush()
        else:

@lfern, I appreciate reply, thank you.
Following several hours of trying to get to the roots of the above mentioned issue, I can summarize the following observations:

After some duration at which all is running well, 'websocket_ob()' events stop occurring all together
'websocket_error()' is not received either (i.e. except for the above 'CloudFlare' one, which appears as a symptom of the actual issue...)
I'm subscribed to multiple symbols of a specific exchange - when the issue occurs, all of the symbols stop receiving websocket_ob() updates
This reproduces repeatedly on: poloniex (highest reproduce rate), huobipro, kraken.
This was not yet reproduced over the last 24 hours on: binance, bitstamp, coinbasepro.
(i.e. each exchange is running separately, on a dedicated python process. Processes run simultaneously on same computer. No RAM or high CPU issues...)
This reproduces both on my pc, and on GCP instance, python 3.6.7
I'm running asyincio logging in debug mode, but do not see anything alarming

A further hint of things going wrong, is observed when I try to recover from this situation.
Since there is no 'websocket_error()' - After of timeout of 60seconds that nothing was received, I try to run: websocketClose(), then websocketCleanContext(), and finally doSubscribe(), as you've mentioned above.
A first exception is raised when issuing websocketClose():

2019-03-15 13:09:54,345 - INFO: handle_log: websocketClose conxid: default
2019-03-15 13:09:54,345 - ERROR: handle_log: GENERIC (non ccxt.BaseError) exception while websocketClose: 'NoneType' object has no attribute 'close'
Traceback (most recent call last):
File "examples/py/websocket-generic-orderbook-poloniex.py", line 107, in handle_log
exchange_ctx.websocketClose(conxid)
File "/home/user/Projects/ccxt/python/ccxt/async_support/base/exchange.py", line 728, in websocketClose
websocket_conx_info['conx'].close()
File "/home/user/Projects/ccxt/python/ccxt/async_support/websocket/websocket_connection.py", line 107, in close
self.client._closeConnection(True)
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/autobahn/asyncio/websocket.py", line 128, in _closeConnection
self.transport.close()
AttributeError: 'NoneType' object has no attribute 'close'

And then when reaching doSubscribe() execution, another exception occurs. It is due to exchange.py's call _websocket_ensure_conx_active(), trying again to close the connection:

2019-03-15 13:09:59,388 - ERROR: exception close location 002: 'NoneType' object has no attribute 'close'
Traceback (most recent call last):
File "/home/user/Projects/ccxt/python/ccxt/async_support/base/exchange.py", line 652, in _websocket_ensure_conx_active
conx.close()
File "/home/user/Projects/ccxt/python/ccxt/async_support/websocket/websocket_connection.py", line 107, in close
self.client._closeConnection(True)
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/autobahn/asyncio/websocket.py", line 128, in _closeConnection
self.transport.close()
AttributeError: 'NoneType' object has no attribute 'close'

When I add a try-except on the close() attempts under _websocket_ensure_conx_active(), I am able to recover from the state, and resubscribe.
Alas... the issue keeps occurring once in a while (e.g. on poloniex it could be once in 30 minutes or so, on huobi and kraken observed once in several hours...) - so I need to understand the actual cause and resolve it.
Any hint on how I can further debug this?

Hi @aviad21 when you say you run the processes simultaneously, can I ask you if you do that similarly as I do in my program (based on lfern branch) https://github.com/firepol/ccxt-websockets-db-updater (see ob_updater.py I use one .py file and asyncio to loop all exchanges and all pairs)? Would be interesting to know if you can reproduce the issue also using my ob_updater.py...

Hi @firepol

when you say you run the processes simultaneously, can I ask you if you do that similarly as I do in my
program...

I meant to say that I'm executing multiple python scripts concurrently, each launched independently:
python examples/py/websocket-generic-orderbook-krken.py &
python examples/py/websocket-generic-orderbook-poloniex.py &
python examples/py/websocket-generic-orderbook-huobipro.py &

(each of these is based on the ccxt\examples\py\websocket-playground.py example)

Hey @lfern , I believe there's a bug with poloniex or the python base/exchange class. It currently doesn't update the orderbook upon delete updates, i.e. when there's a (Price = p, Amount = 0) update.
This can be evidenced in 3 ways:

A slight modification of the original poloniex orderbook example: https://github.com/maayank/ccxt_polo_bug/blob/master/websocket-poloniex-orderbook.py
In essence, it subscribes to BTC/USDT with a limit of 100 and quits with a message if one of the top 10 bids/asks prices is deleted (as opposed to have its amount set).
It never quits (e.g. I've tested for ~3 hours)
Putting a print message on the execution branch that deletes the row in the internal orderbook data structure and grepping for it (or similarly, putting a breakpoint) shows the code of that branch is never called. See this copy of async_support/base/exchange.py at https://github.com/maayank/ccxt_polo_bug/blob/fe43543f88cd4079d167700119f910ddfb359d6d/exchange.py#L344
As far as I can tell, the issue is that:
a. _websocket_handle_ob calls (non-async-)exchange.parse_order_book
b. Which calls self.parse_bids_asks which filters items with amount == 0
c. Later in the poloniex class _websocket_handle_ob_delta_cache calls (async-)exchange.mergeOrderBookDelta which calls updateBidAsk, which in turn expects amount==0 items to exist for signalling deletion
d. git blaming the 'js' and 'python' non-async exchange classes shows the former becoming a simple map (without the filtering side-effect) in March 2018.

Unsure how to fix it myself, as it is potentially foundational.
FYI, let me know if I can further assist.

Hi @furryboffin check this out: https://github.com/lfern/ccxt/tree/feature/websockets-multiple
Examples etc. read here: https://github.com/lfern/ccxt/blob/feature/websockets-multiple/WEBSOCKETS.md
Read "Before you test in python or PHP" (you need to transpile to generate the python files).
For your convenience I created a git repository containing (already transpiled) the latest code from lfern's branch: https://github.com/firepol/ccxt-websockets
Enjoy

THANKS!

Hi @kroitor,
Following @maayank's comment, It seems that zero-value-amount order updates are not being removed from the orderbook.
Possibly, the python origin of this was introduced in the following added condition:
https://github.com/ccxt/ccxt/blob/45febed9b792ab33c9454626e468e8e9672dad37/python/ccxt/base/exchange.py#L1233 ...wheres the intention was filtering out 'none' values?
If so, to avoid filtering out valid 'amount==0' entries, kindly consider using:
if bidask[price_key] is not None and bidask[amount_key] is not None:
...
if (price_key in bidask) and (amount_key in bidask) and (bidask[price_key] is not None and bidask[amount_key] is not None):

Where does the timestamp come from in the gdax/coinbase pro order book feed?  I ask because I'm getting updates with very old (more than 10 seconds) timestamps from time to time.

how can i use websocket in huobipro and zb for 'ob' and trade?
can u take a detail steps?
thx

how can i use websocket in huobipro and zb for 'ob' and trade?
can u take a detail steps?
thx

Hi, as I answered in the other issue you opened in the wrong repository, here a link to my repository, where I tried to create a practical example on how to use lfern's work: https://github.com/firepol/ccxt-websockets-db-updater
You can easily see it in action. Follow the instructions in the readme to get started, then run in example this command:
python ob_tester.py -e huobipro -s BTC/USDT --debug --verbose
I just tested it and the tester works. To save the data in a database, see ob_updater.py usage. The error handling is missing, so it's still work in progress but hopefully it can get you started. Cheers.

Hi @lfern, there appears to be a bug in Kraken's transpiled python code: Orderbook 'bids' updates are not always handled.
From time to time, Kraken will send an orderbook update that contains both 'asks' and 'bids', in a single message e.g.:
[0,{"a":[["5035.90000","7.19104284","1554559551.265576"]]},{"b":[["5029.00000","0.00000000","1554559551.278368"],["3601.00000","0.03400000","1554235030.583914"]]}]

The relevant transpiled code in ccxt\async_support\kraken.py seems to handle data only from first item of msg:
    def _websocket_on_message(self, contextId, data):
        msg = json.loads(data)
        event = self.safe_string(msg, 'event')
        status = self.safe_string(msg, 'status')
        if event is None:
            # channel data
            chanId = msg[0]
            data = msg[1]       ### <-- need to take into account a possible msg[2] as well

Please let me know if I should provide any further information about this. Thank you

Hey @lfern, wanted to let you know that I've added a couple events to the Coinbase/GDAX exchange (ticker, trades, and heartbeat). Willing to make a PR once I have it cleaned up. Running into a small issue though and can't seem to figure out the wait4readyEvent in addition how to properly provide the _websocketTimeoutRemoveNonce event.
For the wait4ready:
(node:1558) UnhandledPromiseRejectionWarning: Error: WebSocket is not open: readyState 0 (CONNECTING)
    at WebSocket.send (.../node_modules/ws/lib/websocket.js:314:19)
    at WebsocketConnection.send (.../node_modules/ccxt/js/base/websocket/websocket_connection.js:81:28)
    at WebsocketConnection.sendJson (.../node_modules/ccxt/js/base/websocket/websocket_base_connection.js:37:14)
    at coinbasepro.websocketSendJson (.../node_modules/ccxt/js/base/Exchange.js:2106:35)
    at coinbasepro._websocketSubscribe (.../node_modules/ccxt/js/gdax.js:1032:16)
    at Promise (.../node_modules/ccxt/js/base/Exchange.js:2323:26)
    at process.internalTickCallback (internal/process/next_tick.js:77:7)

And for the _websocketTimeoutRemoveNonce:
(node:1558) UnhandledPromiseRejectionWarning: Error: coinbasepro websocketSubscribe request timed out (10000 ms)
    at timeout.catch.e (.../node_modules/ccxt/js/base/Exchange.js:2168:23)
(node:1558) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)

Although the exchange seems to work fine (with the exception of the first error). Would really like to solve the nonce riddle.

@lfern from what i understand from the comments here, the websocket api is rather stable.
Are there plans to merge this to the main ccxt version soonish?

@lfern from what i understand from the comments here, the websocket api is rather stable.
Are there plans to merge this to the main ccxt version soonish?

Don't know if it is stable enough, see ccxt/ccxt#4335

Hi @bugs181, Someone reported some problems on slow connections, and I think there is a bug that throws this exception when you try to do a second call to subcription method before a first call had returned. If you have a public testing branch for this feature, I could check it out.

Hey @lfern, wanted to let you know that I've added a couple events to the Coinbase/GDAX exchange (ticker, trades, and heartbeat). Willing to make a PR once I have it cleaned up. Running into a small issue though and can't seem to figure out the wait4readyEvent.
For the wait4ready:
(node:1558) UnhandledPromiseRejectionWarning: Error: WebSocket is not open: readyState 0 (CONNECTING)
    at WebSocket.send (.../node_modules/ws/lib/websocket.js:314:19)
    at WebsocketConnection.send (.../node_modules/ccxt/js/base/websocket/websocket_connection.js:81:28)
    at WebsocketConnection.sendJson (.../node_modules/ccxt/js/base/websocket/websocket_base_connection.js:37:14)
    at coinbasepro.websocketSendJson (.../node_modules/ccxt/js/base/Exchange.js:2106:35)
    at coinbasepro._websocketSubscribe (.../node_modules/ccxt/js/gdax.js:1032:16)
    at Promise (.../node_modules/ccxt/js/base/Exchange.js:2323:26)
    at process.internalTickCallback (internal/process/next_tick.js:77:7)

Although the exchange seems to work fine otherwise, would really like to solve the connection riddle.

I’ve tried using the websocket branch, but I haven’t been able to understand the documentation and for some reason the code isn’t self explanatory for me. Would anyone mind just nudging me in the right direction getting by show an example on how I would get e.g the ws ticker for BitMEX up and running in PHP? Pusesdo code would probably work, as long as it use the correct metods. Hope some one can push me in the right direction.  Thanks!

@btc-zz , just a nudge (especially when I never used the php code), but once you git clone the websockets branch you need to transpile (i.e. compile) the js code to php using npm. So:
git clone https://github.com/lfern/ccxt.git
cd ccxt
git checkout remotes/origin/feature/websockets-multiple
npm install opencollective
npm install ws
npm run postinstall
npm run export-exchanges
npm run transpile

It might ask for additional node.js modules - YMMV.

I think there is a bug that throws this exception when you try to do a second call to subcription method before a first call had returned.

This sounds like what I'm doing. I've noticed that await exchange.websocketSubscribe just hangs however. The promise will never resolve. I had assumed the ws implementation would queue the subscriptions. Most times after the connect error, it will retry the subscription and work fine. There have been some times where subscribing to the heartbeat channel fails and node closes after some time.

@bugs181 If websocketSubscribe hangs forever then there is something wrong in js implementation, you should get a timeout exception. Maybe you have a slow connection or you have to wait for some message sent by server before sending any subscription to server. Maybe gdax websocket server change the protocol. I would try to make some tests, Are you using js version?

@bugs181 I have just tested current gdax js version and it seems it is connecting properly. Maybe we can commit to websocket branch and do some tests.

@lfern Just tested the code I had and it's working fine today. Strange indeed. It's possible that it was from a slow connection (we've had storms for about 2 weeks straight here).

@bugs181  Not really, follow ccxt general rules, and take care that js code transpile right to python and PHP.

Edit:
Is there an linting ruleset you'd like me to follow for PR's? I've noticed a few things about your coding style and want to ensure that we can agree.

@lfern Seems like cobinhood has changed their WS API or something. It gives timed out half of the time

feature/websocket-multiple merged to current ccxt/master version.
@xCuri0 how much time to get timeout error? Testing ob event without error for 1 hour.

@lfern around 75% of the time it fails to subscribe with timeout error. Doesn't happen when I manually use wscat to connect

how to use websocket api now? and is there some docs?

@goutie WEBSOCKET.md on the feature/websocket-multiple branch. Also you can look at the examples. Afaik reconnecting on a network error in WS doesn't work so except it to disconnect after a few hours unless you have perfect internet

Dunno why but if I simulate a network error by turning off WiFi for a while it can reconnect. A real network error gives AttributeError: 'NoneType' object has no attribute 'close' when trying to reconnect

@kroitor what are the requirements for the websocket branch to be merged into master ?

I did not found feature/websocket-multiple branch.

@guotie on lfern's CCXT there is

Ok, thanks

Seems like the only problem with the ws branch is the php build now. Hopefully it will be fixed so it can be merged into master

Any update on this ?

@kroitor What's missing to be able to integrate? Lots of people looking forward to the websockets

@kroitor Could you please tell, when websockets will be avaliable in ccxt?

@qwasko1212 I will post an update here within a week or so. Plz hold on. Thx for your patience!

Hello everyone, and a big thank you to all CCXT users and contributors! We appreciate your involvement very much!
First off, we want to thank the community for enabling us to get this far. You are all awesome people, and without you this project would not be in the position that it stands today. Even as CCXT has concentrated on RESTful APIs, we have always appreciated and understood your enthusiasm for WebSockets. The constant stream of comments expressing interest in the status of WebSockets has not gone unnoticed, despite the fact that we have remained mostly silent on the matter.
It is clear that CCXT cannot remain REST-only any longer. The community has waited for a long time and our silence has created frustration and even doubt that we were working on adding this feature. I apologize for not replying to everyone, but I wanted to provide a meaningful answer and not just an empty promise. We're very focused on the actual work of maintaining and improving CCXT, so we hope for your understanding.
Streaming support within CCXT has been in development for some time, and we are very close to having an architecture that satisfies our criteria for ease of use, hot-swapping, unification and portability. Everything has been coded internally from scratch to the same high standards that CCXT is known for having, with extensibility being an additional goal.
Over the past two years CCXT has grown exponentially, and the amount of required maintenance work and support has grown proportionally with it. As a MIT licensed open source project, we have self-funded everything, but at this scale we cannot continue to do that. Without the necessary resources to maintain the library in the face of a rapidly-changing crypto universe, it would soon lose functionality and decline in usefulness. Without continued development, new integrations would stagnate. None of this is good for the library, the community, or the development of cryptocurrency infrastructure to compete with traditional finance.
Which brings us to CCXT Pro. CCXT Pro is a professional add-on to CCXT that will support WebSockets and much, much more. CCXT will continue as an open source project under the MIT license and will be supported by you, the community, in conjunction with a better-funded CCXT Dev Team. CCXT Pro will be hosted from a private repository on a paid basis, but with the opportunity for top CCXT contributors to get free access.
This balances our strongly-held belief in the value and power of community-oriented open source projects while also addressing the reality of limited resources and a growing demand for a higher level of service and responsiveness from professional traders, larger companies, and financial institutions that use CCXT for mission-critical work. These commercial users want instant access to support, compiled languages, low latencies, additional trading protocols, and, of course, WebSocket capability.
Creating a commercial license and charging these CCXT Pro users allows us to continue the important work on CCXT and, we believe, is an ethical way of supporting the community and its contributors. We expect that some people will object to making CCXT Pro a commercial project, and we are interested in your opinions, particularly if you have been an active contributor. We want to reward our contributors with CCXT Pro licenses, and will soon open a channel where inquiries can be made to do this.
CCXT Pro's release is planned for Q3 this year. We will publish more details on the roadmap to CCXT Pro soon.
With Heartfelt Thanks and Appreciation!

Good to hear @kroitor ! This is the right decision. Please make sure that those with a professional license can still build from sources. Personally, I'm not concerned with paying for software, but I can't afford the security risks of using proprietary software.

Please make sure that those with a professional license can still build from sources.

The licensees will have the access to the source code repository of CCXT Pro and will be able to build from sources. Contributors will get a chance for a free license.

Thanks @kroitor. Any idea how much CCXT Pro will cost?

@zanoo we will offer a simple grid of prices for different types of customers. Considering the purpose of the lib which is to make money, the starting plan for individual developers will be affordable – comparable to your monthly cable/sat tv subscription.
With CCXT Pro the user will communicate with exchanges directly without intermediaries. You don't lose the functionality if your license expires, the lib will keep working. However, it's in your best interest to have a working license that will give access you the access to the repository – including the most recent updates, bugfixes, dedicated support, new exchanges, methods and enhancements.

@kroitor from CCXT Pro, sure enterprises that need features and quick adjustments to the library will benefit, so I think you should charge them accordingly. Medium companies that will almost never bother/ask you something but make money thanks to CCXT should pay, but much less.
If you want to make even small fishes (private persons) pay I think it would be unfair (people can potentially use CCXT to trade, so yes "to make money", but ultimately they invest their funds and it's with their risk that they make money, not directly with CCXT). To consider that there are bots like 3commas, cryptohopper, gekko, gunbot etc. that cost very cheap and actually have some AI / algo that make people gain money.
Also if you wanna leave this for free only for "top CCXT contributors" the question will arise: who are top ccxt contributors, sure the ones doing the dirty work. What about people testing this and submitting lot of bug reports (like me) and useful ideas?
I think it would be fair to keep a personal tier for people not having a business and using this as "hobby" under 10 USD/month. Or even better, for free. I mean: for western countries 10 USD is still ok, but I can imagine that some developers in development countries would like to save 10 USD/month...
...ultimately I think you should allow a license for free for people not using it commercially. E.g. if a developer codes his own bot based on CCXT, this is not commercial. Should be free of charge "non-commercial" license.
I coded some stuff based on CCXT and it's open source, would you want to make people like me pay for it? Seriously, this would remind me Apple requiring developers to pay 99 USD/year, even if they plan publishing free apps. Ridiculous!
If a developer created a bot or a software based on CCXT and wants to sell his software, then I agree that he needs to pay a commercial license. Hopefully, there should be a tier for this too. Imagine again a little developer, trying to code a tool based on CCXT and trying to sell it, but nobody will buys it at the beginning, maybe a few copies per year. There are many developers who try to make some little money after investing many many hours on something and not investing in marketing much to make it a super commercial product... I sure don't wanna spend 50-70 usd/month for a "commercial license" (e.g. coinmarketcap charges 70 usd/month, ridiculously high price for a small developer able to sell maybe 2-3 copies of an app at 3 USD each) and rather remove the CCXT dependency and use native APIs instead. It would be a step back and very disappointing one.
I hope you will consider such aspects and make free licenses for non-commercial people or people tying to use it commercially, but also not making big gains out of it... and relying on people honesty for it. E.g. look at jetbrains, syntevo. Until you are non profit using their tools, you pay nothing, the day you start making money you pay the licenses. It's fair enough.
I really hope that CCXT will remain for free for occasional users. Let the enterprises & commercial users pay! Thx

@kroitor from CCXT Pro, sure enterprises that need features and quick adjustments to the library will benefit, so I think you should charge them accordingly. Medium companies that will almost never bother/ask you something but make money thanks to CCXT should pay, but much less.

👍

If you want to make even small fishes (private persons) pay I think it would be unfair (people can potentially use CCXT to trade, so yes "to make money", but ultimately they invest their funds and it's with their risk that they make money, not directly with CCXT). To consider that there are bots like 3commas, cryptohopper, gekko, gunbot etc. that cost very cheap and actually have some AI / algo that make people gain money.

Don't forget that CCXT with all of its functionality will remain under MIT and will still remain free, so, people who use it are not affected.
The paid part will be the professional part, which includes streaming, that means higher speeds to be more competitive. That part (which has not been released yet) will be paid, and it will be an optional add-on. You will be able to either build upon the free CCXT, or seamlessly switch to the commercial version to be slightly faster and more competitive.
That paid part is not based on the work done by @lfern, but is coded internally by the CCXT Dev Team from scratch. However, @lfern undoubtedly deserves a free license. And licensees will have the access to the source code.

Also if you wanna leave this for free only for "top CCXT contributors" the question will arise: who are top ccxt contributors, sure the ones doing the dirty work. What about people testing this and submitting lot of bug reports (like me) and useful ideas?

Top contributors are people who have contributed significantly to the project in this or that way. Either with code, or with testing and reporting (we'd love to incentivize testers as well, by the way ;)), or by other means. Plz, don't worry, we will not miss people who have put their effort into it. Knowing that you have indeed helped with it a lot over the year, we will, of course, account for that.

Actually, top contributors (e.g. like lfern) not only should get a free license, but would rather deserve to get paid too. I mean, else how do you plan to incentive active contributions?

In fact, that is exactly what we'd want to do along with the other things – we are going to reward contributors and maybe even invite contributors to join the project. In other words, the whole purpose of making CCXT Pro a commercial project – is to sustain the living of both CCXT and CCXT Pro. Because... see the answer to the next question...

If you don't pay me a cent, I keep my code for myself, why releasing it for free and you make money with it and maybe even ask me to pay a license for it,

... why would I keep maintaining CCXT? Why the CCXT Dev Team would put the effort into releasing and maintaining the code? ) I've released quite a lot of code done by myself so that you could use it for free and I did not demand a payment from you ) Why would I do that do you think? ))
To me the attitude is the other way around – making it commercial will help us properly incentivize the contributors. And, again, the original CCXT will remain free. CCXT Pro will be paid though. And, ideally, they will be interoperable, means you could switch back and forth without even changing your existing userland code.

I think it would be fair to keep a personal tier for people not having a business and using this as "hobby" under 10 USD/month. Or even better, for free. I mean: for western countries 10 USD is still ok, but I can imagine that some developers in development countries would like to save 10 USD/month...

We will do our best to make it affordable for individuals. Our starting tier will be close to your estimate. However, please, note, that in order provide the access to sources and to remain accessible to GitHub users (to be easy-to-use with the existing infrastructure) we might have to spend $9 per seat per month (https://github.com/pricing), not including other costs and not including the effort of the devs.

...ultimately I think you should allow a license for free for people not using it commercially. E.g. if a developer codes his own bot based on CCXT, this is not commercial. Should be free of charge "non-commercial" license.

If it is a non-commercial use, then (if you think of it, logically) it should fit perfectly into the category of purposes of CCXT which remains free and will be developed further under the MIT license. So, those who build bots based on CCXT can still enjoy everything for free and they are not affected. However, there are also those who want to enjoy slightly more than others – it is ethical to charge those for extra speed.
If you think of any purpose of streaming – that is purely commercial. Can you name at least one non-commercial purpose of using a slightly faster tool when a free version of the same tool exits, all the rest being equal? Taking it further, if it's a bot, then this is a commercial usage for sure, in all senses. Because the pure purpose of a bot is to be commercial, "to make money", obviously. Why would you need faster speeds other than for building a faster bot? All other purposes including historical tracking – don't really have to be instantaneous, in other words, you would only need CCXT Pro if you want "to make slightly more money than average". If you don't need a faster bot, then you can continue using the CCXT under MIT. Absolutely for free ) And that's honest, open and fair.

I coded some stuff based on CCXT and it's open source, would you want to make people like me pay for it?

Of course, not. Because CCXT remains free.

Seriously, this would remind me Apple requiring developers to pay 99 USD/year, even if they plan publishing free apps.

Our starting tiers will be far from that number.

Ridiculous!

Totally agree with you on this example. However, we're not Apple, we're CCXT. And CCXT Pro now. Which is a bit different. That is why:

We will have a "CCXT bRO" license which will be free for non-commercial use and academic use, and will be free for our contributors, who will be able to get a free license by contributing code to the CCXT Pro repo.
Top contributors to the CCXT repository will also get their free bRO licenses for CCXT Pro. We will provide more details on that.
CCXT will remain fully free under MIT and will be maintained even better if contributors get better incentivized.
CCXT will be subsidized by CCXT Pro (otherwise it's not sustainable).
– CCXT Pro will charge for the advantage only, basically, that includes the maintenance, adding new integrations quickly, the development, support and the underlying costs.

If a developer created a bot or a software based on CCXT and wants to sell his software, then I agree that he needs to pay a commercial license. Hopefully, there should be a tier for this too. Imagine again a little developer, trying to code a tool based on CCXT and trying to sell it, but nobody will buys it at the beginning, maybe a few copies per year. There are many developers who try to make some little money after investing many many hours on something and not investing in marketing much to make it a super commercial product...

The license will be mostly addressing bigger fish. So, a poor little developer will have the opportunity to struggle as long as he wants and our plan for beginners will be affordable. The CCXT Pro license will target those developing companies who don't struggle with marketing and will use CCXT Pro to build successful products and services. Which is fair.

I sure don't wanna spend 50-70 usd/month for a "commercial license" (e.g. coinmarketcap charges 70 usd/month,

Our plan for beginners is far from that. No worries.

ridiculously high price for a small developer able to sell maybe 2-3 copies of an app at 3 USD each) and rather remove the CCXT dependency and use native APIs instead. It would be a step back and very disappointing one.

We understand that, therefore we're don't have the intent to shoot the project in the leg by raising the entry threshold too high.

I hope you will consider such aspects and make free licenses for non-commercial people or people tying to use it commercially, but also not making big gains out of it... and relying on people honesty for it.

We think the same.

E.g. look at jetbrains, syntevo. Until you are non profit using their tools, you pay nothing, the day you start making money you pay the licenses. It's fair enough.

Totally with you on this.

I really hope that CCXT will remain for free for occasional users. Let the enterprises & commercial users pay! Thx

CCXT will remain free for occasional users, moreover, it will remain free even for commercial users and for the enterprises.
CCXT Pro will have a commercial license, but will be affordable to beginners, professional traders, businesses and whales.
We will post more details in a week or two. Stay tuned!