Buy grid stuck after last buy order is not removed from mongodb

Version

v0.0.96
Description

Sometimes after buy order is filled it is not removed from trailing-trade-grid-trade-orders which will cause another buy order will create secondary SYMBOL-grid-trade-last-buy-order record. After buy order is filled, then one of those records is deleted and one will remain in database. This will cause next buy order is not executed because of There is a last gird trade buy order. Wait.. After that the symbol trading is stuck. Not even stop-loss is working as there are buy orders not filled. This can cause significant loss in bear market.
To Reproduce

Random error. I tried to log in detail but didn't found the cause why is the SYMBOL-grid-trade-last-buy-order record not removed from mongodb sometimes.
Edit: From what I've seen I think this issue is occuring when buy order is filled very shortly after new buy order is made. In normal slow market this error will not occur as there is plenty of time to process new order before it's filled but in fast market the price can spike so fast it'll trigger buy order filled right after new buy order is created and this will make inconsistent state as websocket is processed while previous job in queue is still running. The probable cause of issue is making changes in database out of order. Changes in database should be part of the job in queue or otherwise we shouldn't use the queue at all.
Expected Behaviours

It should always remove the SYMBOL-grid-trade-last-buy-order record from mongodb when buy order is filled.
Screenshots

Additional context

I tried to log all "deleteOne" transactions which reveals previous SYMBOL-grid-trade-last-buy-order was never asked to be removed from database. I didn't found why ensure-grid-trade-order-executed.js will skip this. I'm currently trying to log in detail how the buy order is processed line by line, but it's hard to debug as I have to wait until the bug will occur again.

Hmm, strange.
Are you able to export logs when it happens again?

It just happend even after buy order cancellation:

Now stuck at Grid Trade #1:

I can't export logs as it is freezing my browser when I click Export logs. I'll try to resolve the issue.

It looks like BUY - FILLED has been executed before BUY - NEW has been finished. Isn't queue concurrency set to 1 to prevent this? How is it possible it was not waiting in the queue? 🤔

I tried to log every job start and finish in queue and it is very strage what I found.

Jobs are running as expected, but logged events are overlapping and running before jobs with corresponding IDs were running in queue.

I thought everything is running in the quere but from what I see there are some processes running in parallel before the job is queued?

Now I see those unknown events are actually inserting jobs into queue and then just first job is processed as buy filled and other one is silently skipped thru trailing trade steps. It's a bit confusing but I think I understand the logic now.
I thought when new buy order is received and then buy order fill is received then in queue will be both events processed. It looks like only the last event is processed in first running job in queue. Second job is practically useless and doing nothing as there is no change since the first queued job has been finished.
So it looks like if server is lagging then it can somehow receive next buy order before the job for buy filled is finished? I'm little bit confused about the logic of queue.

I tried to change max to 100 as the bot was also very laggy missing some spikes in price. Since then I don't see the error yet. I'll keep testing over night.
https://github.com/chrisleekr/binance-trading-bot/blob/209d2e081ec63259c0c0b6fc1eb3606d13bd8786/app/cronjob/trailingTradeHelper/queue.js#L16-L20

Doesn't helped. It's There is a last gird trade buy order. Wait. again.

Interesting thing is it can fix this automatically if bot is restarted by docker restart binance-bot. I'll do restart every 15 minutes with cron as a workaround but it's very annoying bug causing bad losses.

It happened again. Log from bot:

My detailed log about running steps indicated the trailing trade at time of buy order filled on Binance (10:04:41) has been running longer than usual. It looks like a problem is when Binance fill buy order while trailing trade is processing a job. Time on Binance is local, time in log is UTC.

..."level":50,"msg":"381e1d81-0586-4bf9-94a5-a0f66f4ff471 Job started.","time":"2022-12-31T10:04:38.902Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-configuration","time":"2022-12-31T10:04:38.904Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-info","time":"2022-12-31T10:04:38.909Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-manual-order","time":"2022-12-31T10:04:38.913Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-grid-trade-order-executed","time":"2022-12-31T10:04:38.917Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T10:04:38.922Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - get-open-orders","time":"2022-12-31T10:04:38.925Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - get-indicators","time":"2022-12-31T10:04:38.929Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - get-override-action","time":"2022-12-31T10:04:38.937Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - handle-open-orders","time":"2022-12-31T10:04:38.943Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T10:04:38.946Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - determine-action","time":"2022-12-31T10:04:38.950Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - place-manual-order","time":"2022-12-31T10:04:38.953Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - cancel-order","time":"2022-12-31T10:04:38.957Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - place-buy-order","time":"2022-12-31T10:04:38.960Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-order","time":"2022-12-31T10:04:38.963Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-stop-loss-order","time":"2022-12-31T10:04:38.966Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T10:04:38.969Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - remove-last-buy-price","time":"2022-12-31T10:04:38.972Z","v":0}
..."jobName":"trailingTrade","correlationId":"381e1d81-0586-4bf9-94a5-a0f66f4ff471","symbol":"BTCBUSD","level":50,"msg":"Step - save-data-to-cache","time":"2022-12-31T10:04:38.981Z","v":0}
..."level":50,"msg":"381e1d81-0586-4bf9-94a5-a0f66f4ff471 Job finished.","time":"2022-12-31T10:04:38.994Z","v":0}
..."level":50,"msg":"a56c59fc-5559-4601-bdf8-af3da2781733 Job started.","time":"2022-12-31T10:04:39.909Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-configuration","time":"2022-12-31T10:04:39.911Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-info","time":"2022-12-31T10:04:39.916Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-manual-order","time":"2022-12-31T10:04:39.920Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-grid-trade-order-executed","time":"2022-12-31T10:04:39.924Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T10:04:39.929Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - get-open-orders","time":"2022-12-31T10:04:39.932Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - get-indicators","time":"2022-12-31T10:04:39.936Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - get-override-action","time":"2022-12-31T10:04:39.943Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - handle-open-orders","time":"2022-12-31T10:04:39.947Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T10:04:39.950Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - determine-action","time":"2022-12-31T10:04:39.954Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - place-manual-order","time":"2022-12-31T10:04:39.957Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - cancel-order","time":"2022-12-31T10:04:39.960Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - place-buy-order","time":"2022-12-31T10:04:39.963Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-order","time":"2022-12-31T10:04:39.967Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-stop-loss-order","time":"2022-12-31T10:04:39.970Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T10:04:39.973Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - remove-last-buy-price","time":"2022-12-31T10:04:39.976Z","v":0}
..."jobName":"trailingTrade","correlationId":"a56c59fc-5559-4601-bdf8-af3da2781733","symbol":"BTCBUSD","level":50,"msg":"Step - save-data-to-cache","time":"2022-12-31T10:04:39.986Z","v":0}
..."level":50,"msg":"a56c59fc-5559-4601-bdf8-af3da2781733 Job finished.","time":"2022-12-31T10:04:39.997Z","v":0}
..."level":50,"msg":"50c6465d-bfca-456e-8a5a-fdde57a4342a Job started.","time":"2022-12-31T10:04:40.915Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-configuration","time":"2022-12-31T10:04:40.916Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-info","time":"2022-12-31T10:04:40.921Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-manual-order","time":"2022-12-31T10:04:40.925Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-grid-trade-order-executed","time":"2022-12-31T10:04:40.932Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T10:04:40.938Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - get-open-orders","time":"2022-12-31T10:04:40.941Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - get-indicators","time":"2022-12-31T10:04:40.945Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - get-override-action","time":"2022-12-31T10:04:40.952Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - handle-open-orders","time":"2022-12-31T10:04:40.956Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T10:04:41.479Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - determine-action","time":"2022-12-31T10:04:41.482Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - place-manual-order","time":"2022-12-31T10:04:41.486Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - cancel-order","time":"2022-12-31T10:04:41.489Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - place-buy-order","time":"2022-12-31T10:04:41.493Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-order","time":"2022-12-31T10:04:42.226Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-stop-loss-order","time":"2022-12-31T10:04:42.231Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T10:04:42.234Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - remove-last-buy-price","time":"2022-12-31T10:04:42.237Z","v":0}
..."jobName":"trailingTrade","correlationId":"50c6465d-bfca-456e-8a5a-fdde57a4342a","symbol":"BTCBUSD","level":50,"msg":"Step - save-data-to-cache","time":"2022-12-31T10:04:42.240Z","v":0}
..."level":50,"msg":"50c6465d-bfca-456e-8a5a-fdde57a4342a Job finished.","time":"2022-12-31T10:04:42.255Z","v":0}
..."level":50,"msg":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300 Job started.","time":"2022-12-31T10:04:42.256Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-configuration","time":"2022-12-31T10:04:42.258Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-info","time":"2022-12-31T10:04:42.262Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-manual-order","time":"2022-12-31T10:04:42.266Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-grid-trade-order-executed","time":"2022-12-31T10:04:42.270Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T10:04:42.275Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - get-open-orders","time":"2022-12-31T10:04:42.278Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - get-indicators","time":"2022-12-31T10:04:42.282Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - get-override-action","time":"2022-12-31T10:04:42.289Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - handle-open-orders","time":"2022-12-31T10:04:42.293Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T10:04:42.296Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - determine-action","time":"2022-12-31T10:04:42.300Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - place-manual-order","time":"2022-12-31T10:04:42.303Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - cancel-order","time":"2022-12-31T10:04:42.307Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - place-buy-order","time":"2022-12-31T10:04:42.310Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-order","time":"2022-12-31T10:04:42.313Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-stop-loss-order","time":"2022-12-31T10:04:42.319Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T10:04:42.322Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - remove-last-buy-price","time":"2022-12-31T10:04:42.325Z","v":0}
..."jobName":"trailingTrade","correlationId":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300","symbol":"BTCBUSD","level":50,"msg":"Step - save-data-to-cache","time":"2022-12-31T10:04:42.336Z","v":0}
..."level":50,"msg":"ed29884f-f4f1-46f5-ae8a-0b59e6c0b300 Job finished.","time":"2022-12-31T10:04:42.346Z","v":0}

I think the problem is we are doing changes in mongodb while another job in queue is still running. This can cause some unexpected results.
For example if new buy order is still processing in queue when websocket received buy order fill it can make inconsistent state in mongodb removing order before the order has been processed yet in queue.
https://github.com/chrisleekr/binance-trading-bot/blob/209d2e081ec63259c0c0b6fc1eb3606d13bd8786/app/frontend/websocket/handlers/symbol-update-last-buy-price.js#L17-L24
We should do all changes at once in queue or don't use queue at all. Or as a workaround we should not make any changes in mongodb for symbol while symbol is processed in queue. Any changes should be made after the current job in queue is finished to prevent intermediate changes in database state during the job in queue is still running. Or am I wrong?

@chrisleekr What do you think about it? Can changes in mongodb made by websocket while previous job in queue is still running make an inconsistent state which will leave buy order stuck in database if buy order is filled very shortly after new buy order is made?

Current websocket process is:

Make changes to mongodb.
Add job in queue.

I think we should implement those steps for websocket:

Mark queue as paused.
If any job is still running, wait until the job is finished.
Make changes to mongodb.
Add job in queue.
Mark queue as resumed.

This will prevent any changes to mongodb while job is running like you can see in this log:
..."helper":"queue","level":50,"symbol":"BTCBUSD","msg":"Added job for BTCBUSD","time":"2022-12-31T13:00:16.490Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Start process...","time":"2022-12-31T13:00:16.510Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-configuration","time":"2022-12-31T13:00:16.513Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-info","time":"2022-12-31T13:00:16.520Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-manual-order","time":"2022-12-31T13:00:16.526Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-grid-trade-order-executed","time":"2022-12-31T13:00:16.533Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T13:00:17.119Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - get-open-orders","time":"2022-12-31T13:00:17.124Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - get-indicators","time":"2022-12-31T13:00:17.131Z","v":0}
..."helper":"queue","level":50,"symbol":"BTCBUSD","msg":"Added job for BTCBUSD","time":"2022-12-31T13:00:17.139Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - get-override-action","time":"2022-12-31T13:00:17.159Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - handle-open-orders","time":"2022-12-31T13:00:17.166Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T13:00:17.170Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - determine-action","time":"2022-12-31T13:00:17.175Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - place-manual-order","time":"2022-12-31T13:00:17.180Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - cancel-order","time":"2022-12-31T13:00:17.185Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - place-buy-order","time":"2022-12-31T13:00:17.190Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-order","time":"2022-12-31T13:00:17.194Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-stop-loss-order","time":"2022-12-31T13:00:17.199Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T13:00:17.203Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - remove-last-buy-price","time":"2022-12-31T13:00:17.207Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"Step - save-data-to-cache","time":"2022-12-31T13:00:17.212Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"9df0ab6d-a004-4ac3-b219-76fd7a699bfd","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Finish process...","time":"2022-12-31T13:00:17.247Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Start process...","time":"2022-12-31T13:00:17.258Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-configuration","time":"2022-12-31T13:00:17.259Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-info","time":"2022-12-31T13:00:17.265Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-manual-order","time":"2022-12-31T13:00:17.272Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-grid-trade-order-executed","time":"2022-12-31T13:00:17.282Z","v":0}
..."correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","helper":"queue","level":50,"msg":"Added job for BTCBUSD","time":"2022-12-31T13:00:17.622Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T13:00:17.820Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - get-open-orders","time":"2022-12-31T13:00:17.824Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - get-indicators","time":"2022-12-31T13:00:17.829Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - get-override-action","time":"2022-12-31T13:00:17.842Z","v":0}
..."helper":"queue","level":50,"symbol":"BTCBUSD","msg":"Added job for BTCBUSD","time":"2022-12-31T13:00:17.844Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - handle-open-orders","time":"2022-12-31T13:00:17.849Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T13:00:17.853Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - determine-action","time":"2022-12-31T13:00:17.857Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - place-manual-order","time":"2022-12-31T13:00:17.868Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - cancel-order","time":"2022-12-31T13:00:17.872Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - place-buy-order","time":"2022-12-31T13:00:17.876Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-order","time":"2022-12-31T13:00:17.880Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-stop-loss-order","time":"2022-12-31T13:00:17.884Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T13:00:17.888Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - remove-last-buy-price","time":"2022-12-31T13:00:17.893Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"Step - save-data-to-cache","time":"2022-12-31T13:00:17.898Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"6b546ec2-eef1-49d7-b61e-7332a46e4bbb","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Finish process...","time":"2022-12-31T13:00:17.922Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Start process...","time":"2022-12-31T13:00:17.924Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-configuration","time":"2022-12-31T13:00:17.925Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-info","time":"2022-12-31T13:00:17.969Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-manual-order","time":"2022-12-31T13:00:17.976Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-grid-trade-order-executed","time":"2022-12-31T13:00:17.980Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T13:00:17.986Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - get-open-orders","time":"2022-12-31T13:00:17.990Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - get-indicators","time":"2022-12-31T13:00:17.996Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - get-override-action","time":"2022-12-31T13:00:18.006Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - handle-open-orders","time":"2022-12-31T13:00:18.015Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T13:00:18.020Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - determine-action","time":"2022-12-31T13:00:18.024Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - place-manual-order","time":"2022-12-31T13:00:18.032Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - cancel-order","time":"2022-12-31T13:00:18.036Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - place-buy-order","time":"2022-12-31T13:00:18.041Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-order","time":"2022-12-31T13:00:18.045Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-stop-loss-order","time":"2022-12-31T13:00:18.049Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T13:00:18.053Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - remove-last-buy-price","time":"2022-12-31T13:00:18.057Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"Step - save-data-to-cache","time":"2022-12-31T13:00:18.061Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"da3f69b7-6fd3-415b-868a-548d35bcd08c","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Finish process...","time":"2022-12-31T13:00:18.074Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Start process...","time":"2022-12-31T13:00:18.217Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-configuration","time":"2022-12-31T13:00:18.218Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-info","time":"2022-12-31T13:00:18.224Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-manual-order","time":"2022-12-31T13:00:18.229Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-grid-trade-order-executed","time":"2022-12-31T13:00:18.234Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T13:00:18.240Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - get-open-orders","time":"2022-12-31T13:00:18.244Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - get-indicators","time":"2022-12-31T13:00:18.248Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - get-override-action","time":"2022-12-31T13:00:18.256Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - handle-open-orders","time":"2022-12-31T13:00:18.261Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T13:00:18.266Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - determine-action","time":"2022-12-31T13:00:18.270Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - place-manual-order","time":"2022-12-31T13:00:18.277Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - cancel-order","time":"2022-12-31T13:00:18.282Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - place-buy-order","time":"2022-12-31T13:00:18.286Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-order","time":"2022-12-31T13:00:18.290Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-stop-loss-order","time":"2022-12-31T13:00:18.294Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T13:00:18.298Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - remove-last-buy-price","time":"2022-12-31T13:00:18.302Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"Step - save-data-to-cache","time":"2022-12-31T13:00:18.306Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"7d47a04a-de0f-4326-8a74-be7cfaf21ec8","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Finish process...","time":"2022-12-31T13:00:18.318Z","v":0}

For example:
const deleteLastBuyPrice = async (logger, ws, symbol) => {
  await mongo.deleteOne(logger, 'trailing-trade-symbols', {
    key: `${symbol}-last-buy-price`
  });

  queue.executeFor(logger, symbol, {
    correlationId: _.get(logger, 'fields.correlationId', '')
  });

Should be:
const deleteLastBuyPrice = async (logger, ws, symbol) => {
  await queue.pauseQueue(logger, symbol);
  await queue.waitForCurrentJobToFinish(logger, symbol);

  await mongo.deleteOne(logger, 'trailing-trade-symbols', {
    key: `${symbol}-last-buy-price`
  });

  queue.executeFor(logger, symbol, {
    correlationId: _.get(logger, 'fields.correlationId', '')
  });

  queue.resumeQueue(logger, symbol);

We also should limit number of jobs queued since there is nothing to do after actual state is processed. We should have only one actually running job and one queued job. All other jobs in queue are useless if there is no new change in data to process then.

I added pause, resume and waitForJob functions to queue.js to test if my idea will be working. I'll test over night.

const pause = async (funcLogger, symbol) => {
  const logger = funcLogger.child({ helper: 'queue' });

  await queues[symbol].pause();

  logger.info({ symbol }, `Queue ${symbol} paused`);
};

const resume = async (funcLogger, symbol) => {
  const logger = funcLogger.child({ helper: 'queue' });

  await queues[symbol].resume();

  logger.info({ symbol }, `Queue ${symbol} resumed`);
};

const waitForJob = async (funcLogger, symbol) => {
  const logger = funcLogger.child({ helper: 'queue' });

  while (await queue[symbol].getActiveCount() > 0) {
    await new Promise(resolve => setTimeout(resolve, 10));
  }

  logger.info({ symbol }, `Queue ${symbol} inactive`);
};

I changed the code little bit and now it's working as expected.

..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Start process...","time":"2022-12-31T21:55:02.353Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-configuration","time":"2022-12-31T21:55:02.354Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - get-symbol-info","time":"2022-12-31T21:55:02.360Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-manual-order","time":"2022-12-31T21:55:02.366Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - ensure-grid-trade-order-executed","time":"2022-12-31T21:55:02.371Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T21:55:02.375Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - get-open-orders","time":"2022-12-31T21:55:02.379Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - get-indicators","time":"2022-12-31T21:55:02.383Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - get-override-action","time":"2022-12-31T21:55:02.390Z","v":0}
..."correlationId":"f2847e24-3474-4fc1-8de7-8aeba92185cf","symbol":"BTCBUSD","helper":"queue","level":50,"msg":"Queue BTCBUSD paused","time":"2022-12-31T21:55:02.391Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - handle-open-orders","time":"2022-12-31T21:55:02.395Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T21:55:02.399Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - determine-action","time":"2022-12-31T21:55:02.403Z","v":0}
..."correlationId":"f2847e24-3474-4fc1-8de7-8aeba92185cf","symbol":"BTCBUSD","helper":"queue","level":50,"msg":"Queue BTCBUSD active","time":"2022-12-31T21:55:02.404Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - place-manual-order","time":"2022-12-31T21:55:02.409Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - cancel-order","time":"2022-12-31T21:55:02.414Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - place-buy-order","time":"2022-12-31T21:55:02.418Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-order","time":"2022-12-31T21:55:02.421Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - place-sell-stop-loss-order","time":"2022-12-31T21:55:02.425Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - get-balances","time":"2022-12-31T21:55:02.428Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - remove-last-buy-price","time":"2022-12-31T21:55:02.432Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"Step - save-data-to-cache","time":"2022-12-31T21:55:02.435Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"13555c69-71a9-4e92-b548-a856529d26bc","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Finish process...","time":"2022-12-31T21:55:02.445Z","v":0}
..."helper":"queue","level":50,"symbol":"BTCBUSD","msg":"Queue BTCBUSD inactive","time":"2022-12-31T21:55:02.451Z","v":0}
..."correlationId":"f2847e24-3474-4fc1-8de7-8aeba92185cf","symbol":"BTCBUSD","helper":"queue","level":50,"msg":"Queue BTCBUSD inactive","time":"2022-12-31T21:55:02.451Z","v":0}
..."correlationId":"f2847e24-3474-4fc1-8de7-8aeba92185cf","symbol":"BTCBUSD","helper":"queue","level":50,"msg":"Added job for BTCBUSD","time":"2022-12-31T21:55:02.452Z","v":0}
..."correlationId":"f2847e24-3474-4fc1-8de7-8aeba92185cf","symbol":"BTCBUSD","helper":"queue","level":50,"msg":"Queue BTCBUSD resumed","time":"2022-12-31T21:55:02.452Z","v":0}
..."helper":"queue","jobName":"trailingTrade","correlationId":"c6dd6639-c741-4b45-b468-85a0ec2d3bad","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Start process...","time":"2022-12-31T21:55:02.454Z","v":0}

I catched strange error:
{"name":"binance-api","version":"0.0.96","hostname":"d60863f30718","pid":46,"gitHash":"unspecified","server":"binance","level":50,"err":{"message":"Redis is already connecting/connected","name":"Error","stack":"Error: Redis is already connecting/connected\n    at /srv/node_modules/bull/node_modules/ioredis/built/redis/index.js:269:20\n    at new Promise (<anonymous>)\n    at Redis.connect (/srv/node_modules/bull/node_modules/ioredis/built/redis/index.js:265:21)\n    at /s>
{"name":"binance-api","version":"0.0.96","hostname":"d60863f30718","pid":46,"gitHash":"unspecified","level":50,"err":{"message":"Redis is already connecting/connected","name":"Error","stack":"Error: Redis is already connecting/connected\n    at /srv/node_modules/bull/node_modules/ioredis/built/redis/index.js:269:20\n    at new Promise (<anonymous>)\n    at Redis.connect (/srv/node_modules/bull/node_modules/ioredis/built/redis/index.js:265:21)\n    at /srv/node_modules/bul>
{

It looks finally stable. No locks or missing buy/sell orders since queue pause/resume implemented and duplicated queued jobs removed. I'll prepare PR today. Happy new year 2023. 🥳

It looks finally stable. No locks or missing buy/sell orders since queue pause/resume implemented and duplicated queued jobs removed. I'll prepare PR today. Happy new year 2023. 🥳

@uhliksk Sorry for not getting back to you sooner. I couldn't make time to be in front of the computer.
If you push the PR, I will take a look.

While debugging I found there is also error when Stop price would trigger immediately. which is causing trailing trade is not finished correctly. See 5th line of this log. Then there is no finish process log as it crashed. I can include fix into this PR also.
{..."helper":"queue","jobName":"trailingTrade","correlationId":"e0703def-1b15-432c-bcd2-274aeb5990f6","symbol":"BTCBUSD","level":50,"msg":"▶ TrailingTrade: Start process...","time":"2023-01-01T19:21:51.346Z","v":0}
{..."helper":"queue","jobName":"trailingTrade","correlationId":"e0703def-1b15-432c-bcd2-274aeb5990f6","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Finish process...","time":"2023-01-01T19:21:51.396Z","v":0}
{..."correlationId":"4a9aea3d-37d3-482b-8537-88b142cb4639","symbol":"BTCBUSD","helper":"queue","level":50,"msg":"Added job for BTCBUSD","time":"2023-01-01T19:21:52.352Z","v":0}
{..."helper":"queue","jobName":"trailingTrade","correlationId":"4a9aea3d-37d3-482b-8537-88b142cb4639","symbol":"BTCBUSD","level":50,"msg":"▶ TrailingTrade: Start process...","time":"2023-01-01T19:21:52.354Z","v":0}
{..."helper":"queue","jobName":"trailingTrade","correlationId":"4a9aea3d-37d3-482b-8537-88b142cb4639","symbol":"BTCBUSD","level":50,"err":{"message":"Stop price would trigger immediately.","name":"Error","stack":"Error: Stop price would trigger immediately.\n    at /srv/node_modules/binance-api-node/dist/http-client.js:100:17\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at async execute (/srv/dist/server.js:1:50586)\n    at async /srv/dist/server.js:1:21389\n    at async errorHandlerWrapper (/srv/dist/server.js:1:102373)\n    at async execute (/srv/dist/server.js:1:20060)\n    at async Queue.<anonymous> (/srv/dist/server.js:1:88417)","code":-2010},"errorCode":-2010,"debug":true,"saveLog":true,"msg":"⚠ Execution failed.","time":"2023-01-01T19:21:54.412Z","v":0}
{..."correlationId":"310983ec-8e80-4ae8-978a-5b1395d62fbb","symbol":"BTCBUSD","helper":"queue","level":50,"msg":"Added job for BTCBUSD","time":"2023-01-01T19:21:54.421Z","v":0}
{..."correlationId":"62a0eaa9-c48e-4e1a-9b35-5b616535fc17","symbol":"BTCBUSD","helper":"queue","level":50,"msg":"Already waiting 1 job(s) in BTCBUSD","time":"2023-01-01T19:21:54.425Z","v":0}
{..."correlationId":"716b4da5-4c9b-4df0-b33f-a003221ffc7a","symbol":"BTCBUSD","helper":"queue","level":50,"msg":"Already waiting 1 job(s) in BTCBUSD","time":"2023-01-01T19:21:54.435Z","v":0}
{..."helper":"queue","jobName":"trailingTrade","correlationId":"310983ec-8e80-4ae8-978a-5b1395d62fbb","symbol":"BTCBUSD","level":50,"msg":"▶ TrailingTrade: Start process...","time":"2023-01-01T19:21:54.438Z","v":0}
{..."helper":"queue","jobName":"trailingTrade","correlationId":"310983ec-8e80-4ae8-978a-5b1395d62fbb","symbol":"BTCBUSD","level":50,"msg":"TrailingTrade: Finish process...","time":"2023-01-01T19:21:56.365Z","v":0}

There is another glitch when buy order is rapidly filled by multiple partial buys. The order of processing is as expected so I think there is another problem, but I have to research. It looks like fill order has been processed faster than new order so we should apply the hold much sooner than what I applied in fix to prevent out of order processing. If that's true then holdAndExecute() will not be applicable.

@chrisleekr  This whole block should be inside hold() and resume() pattern? Maybe holdAndResume() function can be created and put that block inside. Same for sale or other long running operations.
https://github.com/chrisleekr/binance-trading-bot/blob/b12aa5c2c9279d752dd642d04f3225897ad5a1a6/app/cronjob/trailingTrade/step/place-buy-order.js#L416-L466)

@chrisleekr Should I create another issue for ^^^ or will we solve everything at once in #562?

Hello @habibalkhabbaz
thank you, it looks very promising. It would certainly cover all types of interference we are dealing with when they will be processed in queue.

@habibalkhabbaz
Sorry, I was wrong. I misunderstood the log because event is logged in bot before it is actually processed. In screenshot of docker log you can see event *652 has been processed after trailing trade has finished so my fix is working as expected. Maybe mongodb itself is not processing writes out of order? Should we require acknowledgement of mongodb writes?

There is a new update is when event is received from Binance and The last order has been update is when it was actually processed which show queue is working well, but I'm confused why there is 3f3758380bf3 as The last order has been updated only. And also 3a106d8b45dc and some others are missing completely. It looks like even logging is missing writes into database. There is clearly something wrong with mongodb when too many writes are done in short period of time. This is not a queue issue anymore after I applied pause/resume solution.

@uhliksk

This whole block should be inside hold() and resume() pattern? Maybe holdAndResume() function can be created and put that block inside. Same for sale or other long running operations.

Interesting. So those block should trigger hold/resume? Then should put inside of the holdAndResume() except the return part.

Should I create another issue for ^^^ or will we solve everything at once in https://github.com/chrisleekr/binance-trading-bot/pull/562?

Update #562 to apply the changes.

Should we require acknowledgement of mongodb writes?

Two scenarios.

Like order related CRUD should be acked MongoDB writes.
Like log related CRUD can be ignored acked.

There is clearly something wrong with mongodb when too many writes are done in short period of time. This is not a queue issue anymore after I applied pause/resume solution.

Yeah, I think you are right. Within 1 second, there were too many writes.
I think we should think about separating the log database out of MongoDB. It's too much pressure for MongoDB.

@chrisleekr

Interesting. So those block should trigger hold/resume? Then should put inside of the holdAndResume() except the return part.

Sorry, I misinterpretted log. This is not an issue. Please ignore. This block is already in queue so there is no need to hold() and resume().

Two scenarios.

Like order related CRUD should be acked MongoDB writes.
Like log related CRUD can be ignored acked.

I agree. Order related CRUD is important and should be acked.


How to configure interpolation?

The documentation of Chartist states that the interpolation of a chart can be configured.
I found the Interface IChartOptions which defines different types/constants of the Interpolation and is exported as ChartOptions in the chartist.component file.
This knowledge has not been enough for me to implement it, I tried to set it with:
      lineSmooth: {         Interpolation :"step",       }
via the options.
This causes the error "can't assign to properties of (new Boolean(true)): not an object" and I guess I don't set it correctly.  But I think I'm close to the solution, because  lineSmooth seems to be a keyword.
Can someone provide me the correct way to do this?

The example shows that option as being passed with the options object. Is that where you are including it?
Also, according to the docs, Interpolation is a function call:
Chartist.Interpolation.none({
    fillHoles: false
 })

So are you importing Chartist and calling the appropriate function? The snippet you provide makes me think not. If you wanted step interpolation, it would like:
Chartist.Interpolation.step({
    postpone: true,
    fillHoles: false
  })

First, thanks for the very fast reply!
Yes, I'm trying to set it via the option object like it is shown in the demo and It works for "simple" options likeaxisX: {         showLabel: false       },
As I now understands this has to be done via a function call. When I import chartist via
import * as Chartist from 'chartist'; and set interpolation to none with your code snippet, nothing changes? Am I doing something wrong, again?

Well, I am not sure what you might expect to see when you set interpolation like that, I guess it depends on the data you provide? But I would expect that to be correct.

I just did it wrong:
It now works if I import Chartist and add it to the options like :
 lineSmooth: Chartist.Interpolation.step({     postpone: true,     fillHoles: false })
Thanks for your help!

Good to hear. No problem.


Question about data transformer

Hi,
Thank you for sharing the great work.
I have a question about the data transformer.
self.transforms_x = Compose([
RandomStretch(),
CenterCrop(instance_sz - 8),
RandomCrop(instance_sz - 2 * 8),
ToTensor()])
Why the size applied on the search image is "instance_sz - 2 * 8" which would be 255- 16 ??
Why not is just 255?
If you know the answer please share with me.
Thank you in advance.
Best Regards,
Lai

The same question, original paper uses 255x255 resolution

"instance_sz - 2 * 8" is a random crop on 255x255 images, used in the training phase, thereby reducing the overfitting on training data.

Is it okay to use random crop if for training we expect image to be centered on the object?

Hi,
Thank you for your reply.
For my understanding, the smaller input size is okay.
But we have to compute the corresponding embedding size and target size during training.
For example, the original paper input size is 255, so the embedding size would be 22x22.
And in the repo is 255-16, so the embedding size would be 20x20.
Am i right? Please correct me if there is any misunderstanding.
The other question is like @ferumchrome , random crop and center crop is not mentioned in the paper, this should be the features in this repo?
How about the performance with and without the augmentation?
Thank you in advance.
Best Regards,
Lai

@ferumchrome The offsets

@ferumchrome The offsets can be used to simulate the target's movement during test. (During test, the targets are not always stay at the center)
@JyunYuLai It does not matter so much that the embedding sizes in training and test are different. The AlexNet has no padding, so cropping on the image approximately has the same effect as cropping on the features (regardless of the differences caused by 3x3 max-poolings), as well as on the score map.
In effect, cropping a 239x239 image on a 255x255 one, is (approximately) the same as cropping a 15x15 score map on the 17x17 one.

hi i could not understand in transforms.py and in _crop function, what the variable size stand for ? what's its meaning? Help, everyone who sees it.

def call(self, z, x, box_z, box_x):
z = self._crop(z, box_z, self.instance_sz)
x = self._crop(x, box_x, self.instance_sz)
z = self.transforms_z(z)
x = self.transforms_x(x)
return z, x
这是什么意思啊？？

请问z = self._crop(z, box_z, self.instance_sz)
x = self._crop(x, box_x, self.instance_sz)    这个_crop()是什么

def call(self, z, x, box_z, box_x):
z = self._crop(z, box_z, self.instance_sz)
x = self._crop(x, box_x, self.instance_sz)
z = self.transforms_z(z)
x = self.transforms_x(x)
return z, x
请问，这是什么意思啊？？

@xzy123456611 The _crop method extracts a squared area centered at the box_z/box_x, with side length of self.instance_sz.

@xzy123456611 The _crop method extracts a squared area centered at the box_z/box_x, with reasonable context. The side length of the output image is self.instance_sz.

谢谢您的回复

z = self._crop(z, box_z, self.instance_sz)
x = self._crop(x, box_x, self.instance_sz)您好，为什么两行的代码，第三个参数用的都是self.instance_sz呢？？

We do this for further data augmentation. The transforms_z further crops image z around its center to size (127, 127).

We do this for further data augmentation. The transforms_z further crops image z around its center to size (127, 127).

thank you


Handle incomplete cert chain

As of right now, the jail server does not serve up a complete certificate chain for TLS (SSL Report). While browsers and CLI tools like curl will deal with this, node does not and the REST client will throw an error about an incomplete cert chain.
To workaround this issue, we set the environment variable NODE_TLS_REJECT_UNAUTHORIZED to 0. While this does make the code chirp along, it opens up the possibility for MITM attacks since it'll accept ANY certificate it is presented with.
We're going to assume that the government won't actually configure their servers properly and that it is unacceptable for this app to have a hardcoded MITM vulnerability. So, this is an us problem, even though it's a them problem.

Without committing the certificate as an artifact of the app, there does not seem to be a nice way to go about solving this.
In commit 5b6abc58876b7f95928323fc49eec033dce2738b I have removed the global override and at least restricted the validation ignore to the jail client itself. While not what I would really like to see, it's slightly less hamfisted.
Maybe someone upstream will figure out how to make a proper cert chain and this can be revisited.
Closing this issue.


Update CONTRIBUTORS.rst

Correction to contributor's name per request by email
Thanks for contributing to Wagtail! 🎉
Before submitting, please review the contributor guidelines http://docs.wagtail.io/en/latest/contributing/index.html and check the following:

Do the tests still pass? (http://docs.wagtail.io/en/latest/contributing/developing.html#testing)
Does the code comply with the style guide? (Run make lint from the Wagtail root)
For Python changes: Have you added tests to cover the new/fixed behaviour?
For front-end changes: Did you test on all of Wagtail’s supported browsers? Please list the exact versions you tested.
For new features: Has the documentation been updated accordingly?

Per email received by email to wagtail.io inbox


Substreams Sink MongoDB tutorial

This is exactly like https://github.com/streamingfast/substreams/issues/107 but targeting MongoDB

The final outline and approach used for the files-sink tutorial should be followed for this documentation. Refer to the following link to find content and the process for creating that page/doc.
https://github.com/streamingfast/substreams/pull/116


Fixed #21454 -- Added support for DB generated fields

Fixes https://code.djangoproject.com/ticket/21454
Field Options
delegated=True|False
A shortcut that automatically sets all of the following options to True when no value is provided for them.
delegated_on_insert=True|False
If True, the value of this field will not be skipped from the INSERT statement completely. This automatically sets return_on_insert to True if not value is given. Supported on all backends.
delegated_on_update=True|False
If True, the value of this field will not be skipped from the UPDATE statement completely. This automatically sets return_on_update to True if not value is given. Supported on all backends.
return_on_insert=True|False
If True, the value for field is loaded and assigned from the database along with the INSERT statement. Supported on PostgreSQL using the RETURNING clause and Oracle using the RETURNING INTO clause.
return_on_update=True|False
If True, the value for field is loaded and assigned from the database along with the UPDATE statement. Supported on PostgreSQL using the RETURNING clause and Oracle using the RETURNING INTO clause.
Model API Changes
save() method now takes an optional argument called ignore_delegated which can be a list of delegated field names. This negates the behavior of delegated_on_insert and delegated_on_update.
Queryset/Manager API Changes
Querysets have a new method call ignore_delegated which takes names of delegated fields as arguments and negates the behavior of delegated_on_insert and delegated_on_update for any subsequent operations. For example, if we have a delegated field called created_at on a model, we can temporarily disable delegation to DB for a single query.
    class MyModel(models.Model):
         pytthon_ts = fields.DatetimeField()
         db_ts = fields.DatetimeField(delegated=True)

This will only update python_ts as db_ts is delegated to database
MyModel.objects.update(db_ts=now(), python_ts=now())
However, this will update both
MyModel.objects.ignore_delegated('db_ts').update(db_ts=now(), python_on=now())
The above example also applies to Model.objects.create().

@timgraham Could you please take a look and point me in the right direction? I've squashed all commits and fixed lint issues and broken tests. I tested manually on PostgreSQL, MySQL and Oracle. Also wrote test cases to cover all backends. Please leave your comments here or on the django ticket. I'll resume work on this next week.

I'd suggest to write to the django-developers mailing list to get a design review of the proposed API.

Thanks Tim. Done: https://groups.google.com/forum/#!msg/django-developers/BDAlTyJwQeY/BOuTv5AHEgAJ

Closing due to inactivity. The mailing list discussion suggests we should head in a slightly different direction.

This remains a very interesting feature request. It seems to have completely lost traction though. @owais , how far did you get? What are we missing in order to bring this into the main branch?
I'd love to help out, but I'd need a bit of direction

@eldamir I did end up with a working implementation but the API design was not very good. This needs a lot more discussion on how the API should look like before someone implements a POC like I did.

@eldamir General consensus on the mailing list was to create a more simpler even if less flexible API and I agree with that. I however didn't follow up on it due to lack to time

Alright. Guess we'll wait and see for now. I managed to solve most of my specific issues by creating an abstract base Model and overriding builtin methods like so:
class MyModel(models.Model):
    ignored_fields = [...]

    class Meta:
        abstract = True

    def _do_insert(self, manager, using, fields, update_pk, raw):
        for i in reversed(xrange(len(fields))):
            if fields[i].name in self.ignored_fields:
                fields.pop(i)
        return super(TemporalModel, self)._do_insert(
            manager, using, fields, update_pk, raw
        )

    def _do_update(self, base_qs, using, pk_val, values, update_fields, forced_update):
        for i in reversed(xrange(len(values))):
            if values[i][0].name in self.ignored_fields:
                values.pop(i)
        return super(TemporalModel, self)._do_update(
            base_qs, using, pk_val, values, update_fields, forced_update
        )


char is not defined

Hello,
I came across a weird error message and I hope this is the right place to report it. I am not sure if it is 'just' a weird message or some kind of bug.
The following code
import Graphics.Element exposing (show)
import String

detect char chars lim idx acc =
  if lim == 0
    then Nothing
    else
      case chars of
        (x::xs) ->
          if char == x
            then
              Just (idx, List.reverse acc ++ xs)
            else
              detect char xs (lim - 1) (idx + 1) (x::xs)
        [] ->
          Nothing

main = show <| detect "h" ["e","h","l"] 1 0 []

leads to a runtime error which says:
char is not defined

Open the developer console for more details.

The console shows:
Uncaught ReferenceError: char is not defined

The Problem is solved by replacing 'char' with something else, for example 'c'. I use the name 'char' in other places without problems. The problem seems to be related to (JavaScript) reserved words: If I replace 'char' with 'byte' the same error message appears.

I guess it's not a problem of error message because it causes a runtime error instead of compile-time.
Looks related to https://github.com/elm-lang/elm-compiler/issues/214 .
So reporting to elm-compiler would be good.
And remember to show your environment (Elm, your browser, and OS version) and http://sscce.org/ when submitting.
Perhaps your example should (can) be much smaller.

The code above gets compiled using tail recursion elimination. So almost certainly the root cause here is the same issue as https://github.com/elm-lang/elm-compiler/issues/1226. Somewhere in the tail recursion elimination code, a call to Generate.JavaScript.Variable.safe is missing. That function is responsible both for replacing ' in variables by $ and for prefixing JavaScript keywords (when used as variables) with $. The error in https://github.com/elm-lang/elm-compiler/issues/1226 was caused by a variable containing ', your error is caused by a variable being a JavaScript keyword. Both errors would be fixed by a call to the safe function during code generation. @evancz or @JoeyEremondi should know where that call needs to happen.

Oh, sorry for misdiagnosing.

Oh, @igrep, you didn't misdiagnose at all. It really was essentially the same issue like in https://github.com/elm-lang/elm-compiler/issues/214, just in a part of the compiler that was written only later. (And the makeSafe and jsReserveds stuff referenced there was refactored/moved since then. But your pointing to https://github.com/elm-lang/elm-compiler/issues/214 helped me see what is amiss here.)

@ggb, I think the issue can be closed here in the error-message-catalog. You may want to add your example to https://github.com/elm-lang/elm-compiler/issues/1226, but actually I think that is not absolutely necessary. Because ' and keywords as variable names are handled by the same safe function anyway. So when https://github.com/elm-lang/elm-compiler/issues/1226 gets fixed, it should fix your example automatically as well.

@jvoigtlaender Thank you! Because it is already addressed/solved, I'll just close this issue :)


Unexpected token when using more varaibles in  php echo statement

PHPMD version: 2.12.0
PHP Version: 8.0, 8.1
Installation type: composer, in docker image
Operating System / Distribution & Version: Alpine, ubuntu 20.04

Current Behavior
Unexpected token: ,, line: 6, col: 10, file:: /test.php
Expected Behavior
Analysis should work
Steps To Reproduce:

Install PHPMD composer require --dev phpmd/phpmd
create test.php file with code example below
run php vendor/bin/phpmd test.php ansi naming

Example code
<?php
$var1 = 'foo';
$var2 = 'bar';
?>

<?= $var1,$var2 ?>

Checks before submitting

[x] Be sure that there isn't already an issue about this. See: Issues list
[x] Be sure that there isn't already a pull request about this. See: Pull requests
[x] I have added every step to reproduce the bug.
[x] If possible I added relevant code examples.
[x] This issue is about 1 bug and nothing more.
[x] The issue has a descriptive title. For example:  "JSON rendering failed on Windows for filenames with space".

Bug confirmed, meanwhile you can use:
<?php echo $var1,$var2; ?>

Or:
<?= $var1.$var2 ?>

I'm working on a fix, you can try composer require pdepend/pdepend:dev-fix/short-tags-multiple-variables (if you use PHPMD from composer) to preview it and support comas in <?= ?> tags.


Simple Anti-analysis IsDebuggerPresent POC

A simple POC that includes a naive anti-analysis technique. Leverages winim to perform the IsDebuggerPresent API call and convert the result to a Nim navite bool value. Then splits the flow of the program based on the outcome of that call.
Contains messagebox payloads but can easily be outfitted for weaponization.

Cheers


owncloud 1.8.3 => 1.8.4

Thank you for the contribution. It was merged directly as 3c91414fdf2648b7dc4422d5bb3d29f0a7bbf858 to keep commit history cleaner. Your contribution is still credited to you.


recent development

is there still some development going on on this project?

There's no development atm. Providing support if I'm able to.


Add Tools section

Need a home for docs on using a dev server, build tools. Currently mentions are scattered around: Getting Started mentions the need for a dev server (recommends polymer-cli); Use a component also mentions it (recommends open-wc dev server).
Building for production is buried under "Use a component," which seems sub-optimal.

tools section is added. closing issue


python3Packages.pydantic: fix doctests race

Description of changes
As submitted upstream: https://github.com/pydantic/pydantic/pull/6103
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@ofborg build python3Packages.pydantic


Add deploy directory to the MEF probe path

Running into MEF cache building errors after some new reference to the Linux workload were added.

@paulmaybee In case you don't know - Pierson is currently out. His wife just had a baby.


Pin version of python to 3.11

Pin version of python to 3.11 since python 3.12 has removed support for distutils

What this PR does / why we need it:
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Type of changes
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)

Feature/Issue validation/testing:
Special notes for your reviewer:
Changes only in workflow
Release note:

NONE

/lgtm
/approve


Update to Spring  5.2.3.RELEASE_1

5.2.3.RELEASE_1 has been released to maven central. The optional spring integration should update to that version.

Done!


commit the MoveFilesToTrashCommand code

Hi Austin,
I'm sending the updated code with MoveFilesToTrashCommand class. I'm a beginner dev and would be very happy to have your feedback.
Best Regards,
Sorena

Thanks for contributing. I will take a look at it tonight and provide any feedback.

Hi Austin, I'm sorry I couldn't respond sooner. I decided to adopt the easier solution as I had to deal with other problems. I used send2trash module and it works. I removed dir_a and dir_b and the class MoveFilesToTrashCommand takes no argument. I have tested it on my pc, and it worked.

Hi @Sorena1994 excellent work on the new solution and it does work! I'll have this pull request merged soon.


[CU-8685qz11x] Allows users to hide article comments

Task linked: CU-8685qz11x Buzz Launch

Task linked: CU-8685qz11x Buzz Launch


Add details about invalid options for k8s backend

A couple of comments then LGTM.

LGTM, thanks @robbrockbank!


Reproducing Results

@brandontrabucco Thank you for the great research and open-sourcing your code. I was trying to reproduce the plots in your paper (Figure 5). I am using the tokens you have provided in Google Drive. I am making a mistake somewhere and can not reproduce the exact plots. If it is possible can you provide the exact arguments for the train_classifier.py script so that I can obtain the results in the paper. Thank you.

Have you tried these arguments?
python train_classifier.py --logdir pascal-baselines/textual-inversion-1.0-0.75-0.5-0.25 \
--synthetic-dir "/projects/rsalakhugroup/btrabucc/aug/\
textual-inversion-1.0-0.75-0.5-0.25/{dataset}-{seed}-{examples_per_class}" \
--dataset pascal --prompt "a photo of a {name}" \
--aug textual-inversion textual-inversion textual-inversion textual-inversion \
--guidance-scale 7.5 7.5 7.5 7.5 \
--strength 1.0 0.75 0.5 0.25 \
--mask 0 0 0 0 \
--inverted 0 0 0 0 \
--probs 0.25 0.25 0.25 0.25 \
--compose parallel --num-synthetic 10 --synthetic-probability 0.5 \
--num-trials 8 --examples-per-class 1 2 4 8 16

@tanriverdiege where you ever able recreate the results + what drive link?


tracing: complete the node status ADT documentation for the Daedalus team

We want to give the Daedalus team something to use for interpretation of the JSON values they'll get for the node status.
The consensus team has already produced a document that is a good basis.

Closing this. If this is still relevant please reopen.


Don't move patients when they have draft records

Currently if a patient has a draft gillick assessment or vaccination record, when they are moved between schools an error is triggered. This is because the existing draft records cause a foreign key constraint error.
https://good-machine.sentry.io/issues/6028433066/
The fix in this PR is to treat draft versions of these associations the same as the recorded versions. This appears to be align with the spirit of PR #2018 which introduces the change to not move patients who have gillick assessments or vaccination records.
It's also possible the right fix is to remove these draft records, but that seems unlikely since as the #2018 states, it's unclear what actual state the patient record is in.

Will sort tomorrow.


Pushing folder and its content to Student Repos

Suppose that the initial content I would like to push to the student repos is organized in subfolders such as:
index.html
css/styles.ccs
imgs/img1.png
imgs/img2.png
I have tried to use push_files action to place the above folders hierarchy into the repository for each student but it did not work.
What is the best way to achieve the above?
Thanks for your appreciated help.

@erradi can you copy and paste the error here, it might give me a better idea of how to help you.


Add support for custom price fields - resolve price refence

Fix for https://github.com/sphereio/sphere-product-import/issues/87

@Siilwyn and @hisabimbola can you please review too

@junajan can you first ensure the tests are green on travis.

@hajoeichler  - Project used for travis tests (importertest-42) has expired. I already asked someone to activate it.

The project is active.

@junajan please add the new option of providing the error call back to the documentation. if possible with a reference to the test example.

@panshul007 @butenkor please check this commit 789326a
I updated readme and the way how logging works.

@butenkor Check this update please 580441e
I added @logger service as another parameter to the errorCallback function - I think it won't harm anything.
I also added better validation of errorCallback variable.


Fix folder casing

The casing here is inconsistent with everything else and possibly causing problems.

Maybe we should add a unix-based Fast check to avoid this kind of thing? Seems to happen fairly often, but then again this is probably the most churn any of our repos will ever see.


Code blocks too close to text

Hello, my code blocks are too close to the text.  Could you send me your document with the command you are using to render The PDF? What version of pandoc are you using?

I use identical repo but using xelatex, can reproduce this issue.

This markdown reproduces the issue
XXXXX YYYYYYYYYY  `OOOOOO` OVERFLOWOVERFLOWOVERFLOWOVERFLOWOVERFLOWOVERFLOWOVERFLOWOVERFLOWOVERFLOWOVERFLOW.

    LISTING BLOCK
    ----------------
    LISTING BLOCK

Compiling with
pandoc bug.md -o bug.pdf --template eisvogel.tex  --listing

This commit introduced the issue: https://github.com/Wandmalfarbe/pandoc-latex-template/commit/5046ee08cdfd7b037b4f4281ffe77f0ad266ced1.
The problem is the negative aboveskip:
aboveskip        = -0.7em,
Changing it back to
aboveskip        = 1.0em,
solves the issue.

I can confirm that this solves the issue. Thanks @oripka

That is strange because I still can't reproduce the issue.

@oripka, @tschinz can you please post the output of latex --version. Maybe you have an old version installed. Are you using MiKTeX or TeX Life? On Windows, macOS or Linux?

This is on macOs:
latex --version
pdfTeX 3.14159265-2.6-1.40.19 (TeX Live 2018)
kpathsea version 6.3.0
Copyright 2018 Han The Thanh (pdfTeX) et al.
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Han The Thanh (pdfTeX) et al.
Compiled with libpng 1.6.34; using libpng 1.6.34
Compiled with zlib 1.2.11; using zlib 1.2.11
Compiled with xpdf version 4.00

@oripka Thank you for your help! Could you please update all your installed packages? Open TeX Live Utility, click on the tab Updates, Select all the packages, right-click them and click update.

@Wandmalfarbe Yes that fixes it as well.

Phew, and I thought I was going crazy :) Thank you very much for your help!

By the way: I recently installed via brew cask install basictex and and the other dependencies via tlmgr. Which seemed to me like the common way to get Latex running on macOs.

Hi,
I face the same issue on Windows using MiKTeX with all packages updated (via MiKTeX Console).
λ latex --version
MiKTeX-pdfTeX 2.9.7029 (1.40.20) (MiKTeX 2.9.7050 64-bit)
Copyright (C) 1982 D. E. Knuth, (C) 1996-2018 Han The Thanh
TeX is a trademark of the American Mathematical Society.
using bzip2 version 1.0.6, 6-Sept-2010
compiled with curl version 7.61.1; using libcurl/7.61.1 WinSSL
compiled with expat version 2.2.6; using expat_2.2.6
compiled with jpeg version 9.3
compiled with liblzma version 50020042; using 50020042
compiled with libpng version 1.6.35; using 1.6.35
compiled with libressl version LibreSSL 2.8.2; using LibreSSL 2.8.2
compiled with MiKTeX Application Framework version 4.6961; using 4.6961
compiled with MiKTeX Core version 14.7079; using 14.7079
compiled with MiKTeX Archive Extractor version 1.6882; using 1.6882
compiled with MiKTeX Package Manager version 6.7072; using 6.7072
compiled with poppler version 0.60.1
compiled with uriparser version 0.9.2
compiled with zlib version 1.2.11; using 1.2.11

To resolve this issue, please follow these steps:

Update all your LaTeX packages to the latest version.

If the problem still persists, you may want to uninstall your LaTeX distribution and install the latest one.

If the problem still persists you can copy the following code and replace the four lines here.
aboveskip        = 1.0em,
belowskip        = 0.1em,
abovecaptionskip = 0em,
belowcaptionskip = 1.0em,


Add network swap functions + checks before create/mint/send asset/collection

closes #79

applied all suggestions but merge is not ready yet. Will test build and function after lunch.


Spellcheck

Fixes Issue #1545

Changes proposed

Screenshots

Note to reviewers

@rupali-codes @CBID2 please review and merge this


Notification of abandoned players

You promised to make notification of abandoned players.

I hope it works, since I can not test it on my own.

You reverted it after bug
https://github.com/Roofkiller/LegionTD-Reborn/issues/29

I know, but I hope it is fixed now


Qs try 3

Codecov Report

Merging #212 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #212      +/-   ##
==========================================
+ Coverage   68.34%   68.34%   +<.01%
==========================================
  Files          82       82
  Lines       15698    15699       +1
==========================================
+ Hits        10729    10730       +1
  Misses       4143     4143
  Partials      826      826

Impacted Files
Coverage Δ

expr/builtins/builtins.go
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b032cc4...b8d6c01. Read the comment docs.


Should use Share icon for Share button

in CommandBar sample, re-ordered buttons so Share in corner.

Description
Should use the OS icon for Share when we labeled the button 'Share', moved it over too so it's near the menu.
Screenshots (if appropriate):

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

It's the little things. Thanks @michael-hawker!


Duplicate lines sometimes wrongs

I'm seeing this occasionally where even with one cursor, the duplicate lines command duplicates more than one line. This GIF shows it happening twice in a row:

I havent seen this in the last several months with many many duplications. @kevinsawicki reopen if you wanna


Upgrading from version 2.4.0 to 2.5.0 on PostgreSQL results in a database error when accessing the console

Describe the bug
When upgrading an established installation from version 2.4.0 to version 2.5.0, an HTTP request to the route oauth/v2/keys returns the HTTP code 500 with the following body:
{
    "Parent": {
        "Severity": "ERROR",
        "Code": "42P01",
        "Message": "relation \"projections.keys3\" does not exist",
        "Detail": "",
        "Hint": "",
        "Position": 293,
        "InternalPosition": 0,
        "InternalQuery": "",
        "Where": "",
        "SchemaName": "",
        "TableName": "",
        "ColumnName": "",
        "DataTypeName": "",
        "ConstraintName": "",
        "File": "parse_relation.c",
        "Line": 1360,
        "Routine": "parserOpenTable"
    },
    "Message": "Errors.Internal",
    "ID": "QUERY-Sghn4"
}

The user sees a "blank" screen. Blank in quotes because the page template is rendered without the page body.
To Reproduce
Steps to reproduce the behavior:

Install version 2.4.0, the configuration is setup to use PostgreSQL
Execute the initial setup, you can login into the service as the admin if you wish with /ui/console
Install version 2.5.0 to update your executable
Restart the service
Access the console via /ui/console
Web page with the template rendered but no content

Expected behavior
I expect that the web console is rendered properly, either a redirection to the login steps or the user interface rendered.
Screenshots
While logging in debug mode, I saw this line in systemd's journal:
Sep 17 15:48:04 zitadel zitadel[7444]: time="2022-09-17T15:48:04Z" level=debug msg="executing check" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/init.go:199" execute=0 projection=projections.keys3

I would assume the migration is not executed ?
Desktop (please complete the following information):
Not relevant; issue is on the server side
Smartphone (please complete the following information):
Not relevant; issue is on the server side
Additional context
The PostgreSQL server is on version 14, as per the requirements in the README. The LXC container running ZITADEL is on Debian bullseye.
While working on other parts of my Ansible roles and playbooks, I may have accidentally downgraded the installation, although it's highly unlikely according to the git history on the file and the merge request opened by Renovatebot.
I haven't found any upgrade steps in the documentation ─ unless I really need some glasses ─ so I assumed that upgrading was a matter of replacing the binary and restarting the service.

Hm this should work. Can you show us the commands and args you use to start zitadel with each run.
CC @adlerhurst

The command is the following: (relative to /opt/zitadel to keep the command short)
bin/zitadel start-from-init --masterkeyFile config/masterkey.txt --config config/config.yml --steps config/first_step.yml

If it should work, I assume that I misused the start-from-init command to make sure that the migrations are always applied ?

Hi @lapin-b thanks for your report. I think the problem is, that an index name of the keys table is reused. Postgresql only allows the same index name per schema, cockroach allows per table. We will rename the index tomorrow.

Hi @lapin-b sorry for the late reply. https://github.com/zitadel/zitadel/releases/tag/v2.6.0 fixes the problem


Artifacts when using playOnce

If I download one of the explosion emitters from the editor, and start it with playOnce:
const particles = new PIXI.ParticleContainer(1500, {rotation: true, tint: true, uvs: true, vertices: true})
const config = PIXI.particles.upgradeConfig(await PIXI.Assets.load('emitter.json'), 'smoke.png')
const emitter = new PIXI.particles.Emitter(particles, {...config, autoUpdate: true, emit: false})

addEventListener('pointerup', _ => {
    emitter.updateSpawnPos(_.x, _.y)
    emitter.playOnce()
})

It consistently happens that a few particles will randomly spawn in between the old position and the new one. I tried calling init and cleanup before I called playOnce but that didn't fix it. Should I be creating a new Emitter every time I want a new explosion?
All the examples I looked at on https://pixijs.io/particle-emitter/examples/ use a manual update loop instead of Pixi's ticker. What's the right way to use the modern features to spawn an emitter where the mouse clicks like the explosion examples do?
One more question that didn't seem like it deserved it's own issue: Why doesn't playOnce set autoUpdate itself like playOnceAndDestroy does?
Thanks 👍

As much as the names might lead you to believe otherwise, the way ParticleContainer works and the way Emitter work don't really work well together - I've found that PIXI.particles.LinkedListContainer gets better performance, due to the constant adding and removing of children (plus not having to disable most of the speed boosts of ParticleContainer). ParticleContainer is big on batching sprites and not changing them, so I am not too surprised by weird behavior like that. It may also be that the options you are using are incomplete - here is what I set up several versions of PixiJS ago, I am not sure if any of them have changed since.
For playOnce vs playOnceAndDestroy: playOnce was set up as an easy way to attach a completion callback, for both users of autoUpdate and manual updaters. playOnceAndDestroy only makes sense if you are using autoUpdate, since otherwise you'd have to clean out your own references when complete anyway.

Thanks for the quick reply.
Going by https://pixijs.download/dev/docs/PIXI.ParticleContainer.html, it looks like a one-to-one matchup from your old initialization to mine. scale becomes vertices, position is enabled by default, rotation and uvs stay the same, and alpha becomes tint.
I'm not attached to particle ParticleContainer though. I'll use LinkedListContainer from now on. But when I swap it out I'm still getting a couple rogue particles. As far as I can tell, it's always exactly two: One at the old spawn and one at the midpoint between the old and new one.
https://codesandbox.io/s/eloquent-scooby-h7n62i

Oh, in that case it's intended behavior of the emitter to interpolate its own position. When reusing an emitter, call emitter.resetPositionTracking() each time you restart it.

Perfect  👍


testing: enable nodeExpansion for k8s-storage rbd-rwo verification

RBD volumes support NodeExpansion, so we should enable that in the tests.
Updates: #2015

Show available bot commands
These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)
/retest all: run this in case the CentOS CI failed to start/report any test
progress or results

/test ci/centos/k8s-e2e-external-storage

/retest ci/centos/k8s-e2e-external-storage

/retest ci/centos/k8s-e2e-external-storage

This succeeded, so nothing prevents this from getting merged now :partying_face:


False positive of incompatible override

Language Server version: 2022.2.5-pre-1
OS and version: Windows 7
Python version (& distribution if applicable, e.g. Anaconda): 3.8.12 Anaconda

Sample code
from __future__ import annotations

from typing import Any

class Base:
    def f(self, *, kwarg0: Any) -> None:
        ...

class Derived(Base):
    def f(self, arg0: Any = ..., *, kwarg0: Any) -> None:
        ...

Expected behavior
No error/warning.
Actual behavior

Thanks for the bug report. I've created a tracking bug in the pyright repo.

This will be fixed in the next release.

This issue has been fixed in version 2022.3.1, which we've just released. You can find the changelog here: CHANGELOG.md


Cocoapods issue

Cocoapods isn't downloading the latest version with doneActionBlock functionality.
I think it is a tag issue in repo.

Yep, just pushed it in release 1.0.5. Cheers!


Compatibility issu with keras 3.2

Hi, I am using keras 3.2 and when using the ntm layer I had an
error:
574
575     def step(self, x, states):
--> 576         assert len(states) == 4  # 2 states and 2 constants
577         h_tm1 = states[0]
578         c_tm1 = states[1]
AssertionError:
This is related to the fact that the states in the LSTM in the new version of keras is now
a list with four components, the hidden state, the cell states and two "constants".
I made it work by changing your update controller function:
def _update_controller(self, inp, h_tm1, M):
    """We have to update the inner RNN inside the NTM, this
    is the function to do it. Pretty much copy+pasta from Keras
    """
    x = T.concatenate([inp, M], axis=-1)
    # update state
    if len(h_tm1) == 2:
        BW,BU = self.rnn.get_constants(x)
        h_tm1 += (BW,BU)

    _, h = self.rnn.step(x, h_tm1)

    return h

But is is probably dependant of the keras version and I am not sure when it changed.
Another question. I saw that in the tutorial you reach the accuracy of 97.
How long did you train it? Did you change the learning rate? Because right now
I am kind of stuck to 90 % of accuracy and a loss of 0.35.

This implementation is really unstable after some recent changes in Keras.
If it doesn't work for you with commit I mentioned in the notebook, then we got a problem.
I'll probably have to review my code from scratch since a lot changed in Keras...
In the meantime if you need a NTM I'd recommend you to take at look at https://github.com/snipsco/ntm-lasagne
In my next approach I'll probably just translate their code to keras

It is ok, with the modification that I mentionned for update_controler, it is working.
The only strange thing is that the accuracy I get is around 90 % while you get around 98 %.

I mean, my accuracy was not problematic. When it converged it would just go up and up with a total of 10000 I would be with >95% close to 98%
You made it work with new Keras? Cool! Would you consider making a PR so I can review it carefully? This way I could go straight to focusing in making the code faster.

Well sorry if I gave you false hope, because I just updated keras, and in the new  LSTM,
they are doing some preprocessing on the input that they move out of the step, which make it more
difficult to adapt to your code, because for the neural turing machine the input change at each step.
In the paper from graves with neural turing machines they achieve the best results with feedforward
controler. What do you thing if temporaly I just put a dense layer as the controler. Have you tried it ?

But it worked with the pip 0.3 version right? We could just make it a note for other people trying to just use it.
I tried with FF controllers, that wouldn't be hard to add to the code. Sometimes it doesn't converge for me... I don't know why. But check this mod with FF controllers:
https://gist.github.com/EderSantana/fb66f36ab8577672ba3c
Let me know what you think

I just did the pull request.
Apparently the guy from lasagne also have this divergence problem and they say that it is bad luck
with the inital values: (Note: unlucky initialisation of the parameters might lead to a diverging solution witness by NaNs.)

that is not bad look, that is division by zero. NaN only appears if you initialize somebody with zeros, since we have a cosine divergence and we got vectors in the denominator. I think I told him that.
My problem is simply that the method does not learn to use its memory. I saw a presentation by Zaremba and he confirmed my intuitions. If the controller network is too strong it will simply overfit the training set. We have to make it slightly weak so it learns to rely on the memory and generalize to larger inputs. Right now I'm more inclined to believe there are better things than neural turing machines, and so does DeepMind given that they didn't try hard to follow up with it and instead proposed several alternative architectures.
Thanks for the pull request! I'll check it out and play with it. Btw, would you have time to look into "Learning to Transduce with unbounded Memory" http://arxiv.org/abs/1506.02516
It shouldn't require a lot of modification from the implementation we have on NTMs. And now that you have the intuition on how whole thing works maybe you could tackle this problem.  I believe this method has a few architectural choices that makes it much easier to get working than NTM for some specific tasks. For example, instead of writing a weaker controller and hope that it will learn to use the memory, this other paper simply force the controller to ALWAYS read and get everything written before that.

Ok,
I took a look at the Transduce article and indeed it looks that the architecture is similar at the turing
machine.
I will try to implement it next week.

I believe this one is solved right?

Yes. I am planning to make the library compatible with keras 0.3.3
but the internal of the lstm is changing quite fast.
I am waiting for a version a little bit more stable .
2016-04-13 15:38 GMT+02:00 Eder Santana<EMAIL_ADDRESS>
I believe this one is solved right?
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/EderSantana/seya/issues/23#issuecomment-209444362

keras-1 was just released. Maybe you want to support that.
Also, wait a little bit for a Theano and Tensorflow wrapper for cuDNN5.
That one will come with optimized RNNs (LSTMs, GRU, vanilla)

I vaguely heard about that, do you ave more info about this optimization?
I will take a look at keras-1.
2016-04-13 18:56 GMT+02:00 Eder Santana<EMAIL_ADDRESS>
keras-1 was just released. Maybe you want to support that.
Also, wait a little bit for a Theano and Tensorflow wrapper for cuDNN5.
That one will come with optimized RNNs (LSTMs, GRU, vanilla)
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/EderSantana/seya/issues/23#issuecomment-209544668

yeah they wrote a really nice blog post about it https://devblogs.nvidia.com/parallelforall/optimizing-recurrent-neural-networks-cudnn-5/


Update Pypi

I believe version 1.0.15 can't be the latest and should be updated on python-binance
Since lots of new feature added in this repo.
Could anyone with permission help?

Please 🙏🏻