cache.Putter duplicates RefCnt

Context
1.18.2
Summary
Rapidly downloading a non-local reference can result in chunkStore RefCnt inconsistencies relative to the cache records.   This happens because there is no mutex protection in the Cache.Putter method.
https://github.com/ethersphere/bee/blob/1a9e1be8a86fbb1725de464c621f4201772bfff7/pkg/storer/internal/cache/cache.go#L69
Expected behavior
If a chunk is retrieved multiple times in quick succession, I'd still only expect it to be inserted into the cache with a corresponding chunkStore RefCnt increment ONCE.
Actual behavior
I've been analyzing the RefCnts in the ChunkStore and added extensive logging (called chunkTrace) to my bee node to track whenever the RefCnt is incremented or decremented as well as when actual chunks are initially stored to and finally removed from sharky.  As an extension of this logging, I've recently started working on a RefCnt analyzer/fixer and much to my surprise, I have a bunch of chunks with incorrect RefCnts vs what is represented by the cache, reserve, and pins (ignoring upload for the moment).
I db nuked a node in testnet and started it up with even more extensive logging that reveals why the ChunkStore.Put() and .Delete() were invoked.  In just a few hours of running, I have RefCnts that are higher than is justified.   And in all cases, it was a Download Retrieval CachePutter that incorrectly incremented the RefCnt for a chunk while storing a single entry in the actual cache store.
I requested a download of a specific reference, maybe two multiple times in quick succession   I don't remember why (I suspect it was the OSM map browser), but that's what the logs show.   Since the 2 chunks making up that reference were not local, they were fetched from the swarm, concurrently, for each of my SIX download requests.
When these six retrievals (times 2 chunks) completed, the netstore.Download method invoked the cache putter to save the chunks.  As you can see in the logs below, this all occurred in less than 1 millisecond for each of the two chunks.
https://github.com/ethersphere/bee/blob/1a9e1be8a86fbb1725de464c621f4201772bfff7/pkg/storer/netstore.go#L94
First the API resolving the reference:

bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.407428" "level"="debug" "logger"="node/api" "v"=1 "msg"="resolve name: parsing bzz address successful" "string"="2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011" "address"="2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011"
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.407428" "level"="debug" "logger"="node/api" "v"=1 "msg"="resolve name: parsing bzz address successful" "string"="2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011" "address"="2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011"
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.407428" "level"="debug" "logger"="node/api" "v"=1 "msg"="resolve name: parsing bzz address successful" "string"="2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011" "address"="2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011"
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.409145" "level"="debug" "logger"="node/api" "v"=1 "msg"="resolve name: parsing bzz address successful" "string"="2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011" "address"="2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011"
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.409163" "level"="debug" "logger"="node/api" "v"=1 "msg"="resolve name: parsing bzz address successful" "string"="2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011" "address"="2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011"
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.409147" "level"="debug" "logger"="node/api" "v"=1 "msg"="resolve name: parsing bzz address successful" "string"="2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011" "address"="2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011"

Then the successful retrieval of the first chunk:

bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.674035" "level"="debug" "logger"="node/retrieval" "msg"="retrieved chunk" "chunk_address"="714655aa19478dadfdb869f7f4def0c0409fecdc41107915b7a017bb7ed94cff" "peer_address"="675d2d8f31ba851a5a3eaad5cbbeeca94c02b0fe666dcb2acef95d9bf4c73e3f" "peer_proximity"=3
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.674618" "level"="debug" "logger"="node/storer" "msg"="chunkTrace: chunkStore.Put new chunk NOstamp!" "why"="CachePutter(Cache(Download))" "address"="714655aa19478dadfdb869f7f4def0c0409fecdc41107915b7a017bb7ed94cff" "loc"="14:242(1013)"

And the redundant increments on that chunk:

bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.674677" "level"="debug" "logger"="node/storer" "msg"="chunkTrace: chunkStore.Put increment chunk NOstamp!" "why"="CachePutter(Cache(Download))" "address"="714655aa19478dadfdb869f7f4def0c0409fecdc41107915b7a017bb7ed94cff" "loc"="14:242(1013)" "refCnt"=2
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.674721" "level"="debug" "logger"="node/storer" "msg"="chunkTrace: chunkStore.Put increment chunk NOstamp!" "why"="CachePutter(Cache(Download))" "address"="714655aa19478dadfdb869f7f4def0c0409fecdc41107915b7a017bb7ed94cff" "loc"="14:242(1013)" "refCnt"=3
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.674773" "level"="debug" "logger"="node/storer" "msg"="chunkTrace: chunkStore.Put increment chunk NOstamp!" "why"="CachePutter(Cache(Download))" "address"="714655aa19478dadfdb869f7f4def0c0409fecdc41107915b7a017bb7ed94cff" "loc"="14:242(1013)" "refCnt"=4
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.674822" "level"="debug" "logger"="node/storer" "msg"="chunkTrace: chunkStore.Put increment chunk NOstamp!" "why"="CachePutter(Cache(Download))" "address"="714655aa19478dadfdb869f7f4def0c0409fecdc41107915b7a017bb7ed94cff" "loc"="14:242(1013)" "refCnt"=5
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.674876" "level"="debug" "logger"="node/storer" "msg"="chunkTrace: chunkStore.Put increment chunk NOstamp!" "why"="CachePutter(Cache(Download))" "address"="714655aa19478dadfdb869f7f4def0c0409fecdc41107915b7a017bb7ed94cff" "loc"="14:242(1013)" "refCnt"=6

Then the second chunk was successfully retrieved:

bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.932027" "level"="debug" "logger"="node/retrieval" "msg"="retrieved chunk" "chunk_address"="eed32a1fc98f4e6dc980a022b8c429fafcea04ffb71ff74b85b5aa9e3ce73561" "peer_address"="f320fb80e8c3872fd7605eb44dba93ae192b23ab0a78d6e8c0523565072e8e18" "peer_proximity"=3
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.932539" "level"="debug" "logger"="node/storer" "msg"="chunkTrace: chunkStore.Put new chunk NOstamp!" "why"="CachePutter(Cache(Download))" "address"="eed32a1fc98f4e6dc980a022b8c429fafcea04ffb71ff74b85b5aa9e3ce73561" "loc"="1:246(840)"

And also redundantly incremented:

bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.932628" "level"="debug" "logger"="node/storer" "msg"="chunkTrace: chunkStore.Put increment chunk NOstamp!" "why"="CachePutter(Cache(Download))" "address"="eed32a1fc98f4e6dc980a022b8c429fafcea04ffb71ff74b85b5aa9e3ce73561" "loc"="1:246(840)" "refCnt"=2
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.932722" "level"="debug" "logger"="node/storer" "msg"="chunkTrace: chunkStore.Put increment chunk NOstamp!" "why"="CachePutter(Cache(Download))" "address"="eed32a1fc98f4e6dc980a022b8c429fafcea04ffb71ff74b85b5aa9e3ce73561" "loc"="1:246(840)" "refCnt"=3
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.932775" "level"="debug" "logger"="node/storer" "msg"="chunkTrace: chunkStore.Put increment chunk NOstamp!" "why"="CachePutter(Cache(Download))" "address"="eed32a1fc98f4e6dc980a022b8c429fafcea04ffb71ff74b85b5aa9e3ce73561" "loc"="1:246(840)" "refCnt"=4
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.932842" "level"="debug" "logger"="node/storer" "msg"="chunkTrace: chunkStore.Put increment chunk NOstamp!" "why"="CachePutter(Cache(Download))" "address"="eed32a1fc98f4e6dc980a022b8c429fafcea04ffb71ff74b85b5aa9e3ce73561" "loc"="1:246(840)" "refCnt"=5
bee-1.18.2-B.out:"time"="2023-12-18 23:09:54.932921" "level"="debug" "logger"="node/storer" "msg"="chunkTrace: chunkStore.Put increment chunk NOstamp!" "why"="CachePutter(Cache(Download))" "address"="eed32a1fc98f4e6dc980a022b8c429fafcea04ffb71ff74b85b5aa9e3ce73561" "loc"="1:246(840)" "refCnt"=6

In actual fact, it wasn't redundant fetches of a single reference from the API, but the internal manifest resolution fetching one of the nodes along the /8/12... path as seen in these api completion logs:
bee-1.18.2-B.out:"time"="2023-12-18 23:09:56.610923" "level"="info" "logger"="node/api" "msg"="api access" "ip"="<IP_ADDRESS>" "method"="GET" "host"="<IP_ADDRESS>:1633" "uri"="/bzz/2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011/8/128/85.png" "proto"="HTTP/1.1" "status"=200 "size"=38560 "duration"="2.201763364s" "referrer"="http://<IP_ADDRESS>:1633/bzz/2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011/" "user-agent"="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"
bee-1.18.2-B.out:"time"="2023-12-18 23:09:56.639080" "level"="info" "logger"="node/api" "msg"="api access" "ip"="<IP_ADDRESS>" "method"="GET" "host"="<IP_ADDRESS>:1633" "uri"="/bzz/2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011/8/127/83.png" "proto"="HTTP/1.1" "status"=200 "size"=47064 "duration"="2.229934697s" "referrer"="http://<IP_ADDRESS>:1633/bzz/2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011/" "user-agent"="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"
bee-1.18.2-B.out:"time"="2023-12-18 23:09:56.754622" "level"="info" "logger"="node/api" "msg"="api access" "ip"="<IP_ADDRESS>" "method"="GET" "host"="<IP_ADDRESS>:1633" "uri"="/bzz/2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011/8/128/83.png" "proto"="HTTP/1.1" "status"=200 "size"=28438 "duration"="2.345477204s" "referrer"="http://<IP_ADDRESS>:1633/bzz/2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011/" "user-agent"="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"
bee-1.18.2-B.out:"time"="2023-12-18 23:09:56.755814" "level"="info" "logger"="node/api" "msg"="api access" "ip"="<IP_ADDRESS>" "method"="GET" "host"="<IP_ADDRESS>:1633" "uri"="/bzz/2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011/8/127/84.png" "proto"="HTTP/1.1" "status"=200 "size"=48705 "duration"="2.348396845s" "referrer"="http://<IP_ADDRESS>:1633/bzz/2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011/" "user-agent"="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"
bee-1.18.2-B.out:"time"="2023-12-18 23:09:56.756660" "level"="info" "logger"="node/api" "msg"="api access" "ip"="<IP_ADDRESS>" "method"="GET" "host"="<IP_ADDRESS>:1633" "uri"="/bzz/2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011/8/128/84.png" "proto"="HTTP/1.1" "status"=200 "size"=42980 "duration"="2.349252757s" "referrer"="http://<IP_ADDRESS>:1633/bzz/2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011/" "user-agent"="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"
bee-1.18.2-B.out:"time"="2023-12-18 23:09:57.041609" "level"="info" "logger"="node/api" "msg"="api access" "ip"="<IP_ADDRESS>" "method"="GET" "host"="<IP_ADDRESS>:1633" "uri"="/bzz/2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011/8/127/85.png" "proto"="HTTP/1.1" "status"=200 "size"=50954 "duration"="2.634205123s" "referrer"="http://<IP_ADDRESS>:1633/bzz/2f0d6bd9afb1d95f37b241f2b79f258c0af577288cb320a226ab46fecfd66011/" "user-agent"="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"

Steps to reproduce
Find a reference not already local in your node and fire quick concurrent /bytes (or it may have been /bzz) retrievals of that reference.   If the timing is correct and quick enough, you'll end up with a chunk that will never leave your sharky even after it has long since been flushed from the cache.
But there are NO tools nor logging in the stock bee node that will ever let you see this effect other than your sharky will slowly grow larger and larger over time with no apparently explanation.
Possible solution
Protect the Cache.Putter from concurrent execution between the Has and actually storing the chunk in the cache.  This will ensure that concurrent invocations will not all blow through the Has not being present and them all redundantly putting a single cacheEntry record (keyed) but invoking ChunkStore().Put multiple times causing the RefCnt to increment multiple times with but a single decrement when the cacheEntry is finally flushed.
https://github.com/ethersphere/bee/blob/1a9e1be8a86fbb1725de464c621f4201772bfff7/pkg/storer/internal/cache/cache.go#L69

I just also realized that the Retrieval.RetrieveChunk singleflight keyed on the chunk reference (with an optional _origin suffix as would be in this case) will practically guarantee that multiple concurrent downloads of a single reference chunk will all complete within microseconds of each other, which is probably why I'm seeing so many of these in such a short time after browsing the OSM map manifest through this newly nuked node.
https://github.com/ethersphere/bee/blob/1a9e1be8a86fbb1725de464c621f4201772bfff7/pkg/retrieval/retrieval.go#L155


Hiring stream id in vacancy model should be Int

Right now it is string, that conflicts with hiring stream id in hire
https://github.com/taskany-inc/crew/blob/cd00e1e72a14958df1663e6fdeebbf39e67729a3/prisma/schema.prisma#L261C22-L262C1
https://github.com/taskany-inc/crew/blob/c19b97fa560644c802eb414fa8f14d325828601b/src/modules/vacancyMethods.ts#L59

It should probably be fixed in hire because string is a wider type


html comment parse problem

Type: Bug
There is a bug where the web browser displays "-->" when running the following HTML code for debugging using Live Server. When the second occurrence of  is changed to another tag, the "-->" no longer appears. Therefore, it is believed to be a bug caused by the second . This has been confirmed in the latest versions of Chrome (non-Chromium) and Firefox. Also, this issue does not occur when the HTML is displayed directly in these web browsers without going through the debugger. In other words, it is a issue specific to debugging execution in VSCode.

VS Code version: Code 1.78.2 (b3e4e68a0bc097f0ae7907b217c1119af9e03435, 2023-05-10T14:39:26.248Z)
OS version: Windows_NT x64 10.0.19045
Modes:
Sandboxed: Yes
Extensions (3)

Extension
Author (truncated)
Version

Bookmarks
ale
13.3.1

code-runner
for
0.12.0

LiveServer
rit
5.7.9

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30445986
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593cf:30376535
pythonvs932:30410667
cppdebug:30492333
vsclangdf:30486550
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
282f8724:30602487
pyind779:30671433
f6dab269:30613381
pythonsymbol12:30671437
a9j8j154:30646983
showlangstatbar:30737416
vsctsb:30705553
pythonms35:30701012
pythonfmttext:30731395
pythoncmv:30739700
fixhidewlkth:30730051
hidesbindicator:30730055
pythongtdpathcf:30739705

Additional note:
The cause of this behavior has been found to be in Live Server. VSCode seems to be unrelated because the same problem occurs even when running on a standalone Live Server (installed with the npm install -g live-server command).

Please file an issue on the repo for the Live Server extension, thanks

Got it. Thank you very much as well.


Removed WFI test from automated tests

Removes WFI test from automated tests

We might want to remove it completely? :)

I think having the test is still useful, especially if this question pops again.


[TST] Import pandas test functions in uniform pattern

PR Description
Please describe the changes proposed in the pull request:

This imports assert_frame_equal directly from pandas.testing instead of using pd.testing.assert_frame_equal
Also changes imports from private pandas._testing to pandas.testing

This PR resolves #674.

PR Checklist

Please ensure that you have done the following:

[X] PR in from a fork off your branch. Do not PR from <your_username>:dev, but rather from <your_username>:<feature-branch_name>.

[X] If you're not on the contributors list, add yourself to AUTHORS.rst.

[X] Add a line to CHANGELOG.rst under the latest version header (i.e. the one that is "on deck") describing the contribution.

Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

Quick Check
To do a very quick check that everything is correct, follow these steps below:

[X] Run the command make check from pyjanitor's top-level directory. This will automatically run:

black formatting
flake8 checking
running the test suite
docs build

Once done, please check off the check-box above.
If make check does not work for you, you can execute the commands listed in the Makefile individually.
Relevant Reviewers

Please tag maintainers to review.

@ericmjl

@hectormz in case it helps, I pasted here the errors from Azure. They all pertain to Flake8 only.
./tests/spark/functions/test_update_where_spark.py:1:1: F401 'pandas as pd' imported but unused
./tests/spark/functions/test_update_where_spark.py:18:5: F821 undefined name 'assert_frame_equal'
./tests/spark/functions/test_update_where_spark.py:41:5: F821 undefined name 'assert_frame_equal'
./tests/spark/functions/test_update_where_spark.py:64:5: F821 undefined name 'assert_frame_equal'
./tests/spark/functions/test_update_where_spark.py:87:5: F821 undefined name 'assert_frame_equal'
./tests/functions/test_add_column.py:6:1: F401 'pandas.testing.assert_frame_equal' imported but unused
./tests/functions/test_collapse_levels.py:1:1: F401 'pandas as pd' imported but unused
./tests/functions/test_add_columns.py:6:1: F401 'pandas.testing.assert_frame_equal' imported but unused
./tests/functions/test_select_columns.py:1:1: F401 'pandas as pd' imported but unused

Thanks @ericmjl ! I'm not sure how I missed these issues when running make check. I also want to add something in the documentation as you mentioned in the issue.

Let's get one more pair of eyes on here. @szuckerman?

Will there be a comment/guide somewhere in the docs, advising code contributors to import assert_frame_equal instead of pd.testing.assert_frame_equal

I will keep a note of this for future contributions, as a reminder to myself. All that is left is to include this change somewhere in the docs for contributors. Well done @hectormz

Thanks @samukweku . I have not yet added this usage pattern to the docs. I thought I would make a quick and dirty addition to our docs, but have since gone down a rabbit hole:

Our guide on unit tests doesn't use assert_frame_equal etc, which would be helpful to demonstrate
Our current guide is very nicely done and I don't want to mangle it by stuffing in this item
I know we did some docs rearranging in the past, but I think it might be worthwhile to consider another round of it to make some material more accessible (which would definitely be another issue)

@ericmjl perhaps we could add another section to/like 2. How should the test look like? that uses one of these pandas assert functions. Or find a test that uses both.

@ericmjl sorry I've left this open for so long. For simplicity sake, what do you think about demonstrating writing unit tests with test_add_column_scalar, which uses assert_series_equal, and a normal assert:
@pytest.mark.functions
@given(df=df_strategy())
def test_add_column_scalar(df):
    # column appears in DataFrame
    df = df.add_column("fortytwo", 42)
    assert "fortytwo" in df.columns

    # values are correct in dataframe for scalar
    series = pd.Series([42] * len(df))
    series.name = "fortytwo"
    assert_series_equal(df["fortytwo"], series)

    # scalar values are correct for strings
    # also, verify sanity check excludes strings, which have a length:

If we went this route, I would add a test docstring as part of the commit.
But curious if anyone things this test is better/worse for teaching tests

@hectormz looks like a not bad one for teaching the practice of testing, and providing a good starting point. Let's go with it!

@hectormz, I'd like to propose merging, as the PR has been open for quite a while. If you're cool with that, please ping back. We can do the docs piece in a separate PR, unless you're close to finishing -- in which case, let's sneak in that piece here :smile:.
Others who haven't been so involved in the code review can chime in - don't feel shy! :smile:

Yes, sorry! I've been a bit distracted. Please merge and I'll add the documentation separately

OK, it's been a few days without movement here. I'm going to hit the merge button. Thanks @hectomz!


Context for some ticks scriptable options

I noticed that the following properties in the ticks options for radial linear scale are scriptable

count
format
maxTicksLimit
precision
stepSize

and the invoked callbacks are receiving a chart context as argument.
Maybe this is implemented by design but it could make more sense to pass a scale context due to the ticks options is a sub element of scale options.

Yes, those should likely use the scale context. I'd support updating this


Swift & Moved SDWebImage to subspec

Swift rewrite
Refactored retrieveImageFromURL to a protocol

Moved SDWebImage to new subspec (pod 'BFRImageViewer/SDWebImage').

Renamed BFR... classes to Buffer...
Updated docs to markdown format
Added fade animation (#14) (considering containerView.y <~> gesture.y distance)
Removed BFRImageViewerDemo/Pods/ subdir from the repo

TODO: Apply API Design guidelines
Closes #35
Closes #14
Update: Maintain ObjC and Swift versions of the pod ? It's not that hard to remove AnyObject in favor of a type-safe approach (like an enum). The issue is objc compatibility.

Hey hey @bre7 - awesome stuff 😄 Sorry for the delay on reviewing this - we've been pushing to get some larger projects ready for the world! @ay8s or I will have a look asap 👍

Awesome 👍

Hey hey @bre7 - so sorry for the delay here and thanks for all your hard work! At Buffer - we aren't quite moving to Swift yet (likely won't until ABI compatibility hits), I was wondering if you'd be up for moving this PR over to the 'release/swift' branch as we'd love to have a Swift rewrite in another location 👍

Hey Jordan, I'll resend it ;)


As of version 3, the pre-built Lambda functionality does not work

Version 3.0.0 renamed the main executable file from main to bootstrap, as this is the executable the AL2023 Lambda runtime expects.
However as per https://github.com/cdklabs/cdk-ecr-deployment/pull/424#issuecomment-1863828565 (thanks @yito-dev-tokyo), a required update to the scripting for using the prebuilt Lambda [*] was missed. Unfortunately by it's nature, the updates to this scripting were not testable without releasing a new version of the cdk-ecr-deployment package.
https://github.com/cdklabs/cdk-ecr-deployment/blob/0debc2341e029584794e46240bf2e5a729c58273/lambda/install.js#L48
Should be
const bin = path.join(dir, 'bootstrap');

As a workaround for this bug, set the NO_PREBUILT_LAMBDA environment variable as per https://github.com/cdklabs/cdk-ecr-deployment#environment-variables to avoid the prebuilt functionality entirely.
Part of the release criteria for this fix from this should be a successful deploy of the Lambda with FORCE_PREBUILT_LAMBDA enabled, to ensure there are no other bugs in the prebuilt Lambda functionality that were missed.
[*] By default, the prebuilt Go executable from the Github release is downloaded and used, to avoid having to do your own build.

@scanlonp it might be worth updating the README to explain this in the meantime, or perhaps to make this a sticky issue to make it easier for users to discover the workaround.


fix: record the external memory summary inside of the context viewer

@mattzh72 we should (in a later PR) update the get context viewer route to not call load_agent


"Could not be read the video [video name]" when running "extract_features.p"

Hello, I'm running extract_features.py using the command KERAS_BACKEND=theano python -u extract_features.py -d ~/Videos/videos. Sometimes I see the program printing messages of Could not be read the video [video name], for example Could not be read the video Af_0zV79zsk, while Af_0zV79zsk.mp4 does exist under the path where I store all the downloaded videos of the ActivityNet dataset.
So my question is are those videos with this kind of messages printed still be extracted the C3D features?

As you have downloaded Af_0zV79zsk.mp4 video, it should still be there, but what shouldn't be is the Af_0zV79zsk.npy file at the extracted features directory.
At this line it shows that when the video can not be read, it jumps to the next one without saving the extracted features.


Added Generative AI & LLM>Copilot LLM ext for VS Code IDE files

Description
This extension was developed to enhance coding efficiency by integrating advanced language models like Llama 3.1, OpenAI’s GPT-3.5-turbo, and Gemini 1.5. These models offer powerful code generation, suggestion capabilities, and the ability to solve complex coding problems. The extension is designed with a user-friendly interface and can be easily integrated into your existing workflow.
Fixes #1502
Type of change

[x] Added a new machine learning frameworks, libraries or software.
[x] Documentation update

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings

@sanjay-kv Sir, I think no labels were assigned when this got merged . Please review once as i didnt receive any points


Paddle ernie layout

软件环境
- paddlepaddle:
- paddlepaddle-gpu:
- paddlenlp:

重复问题

[X] I have searched the existing issues

错误描述
Why the paddle ernie layout base model work badly than the service in huggingface.co/paddlepaddle/ernie-layout

稳定复现步骤 & 代码
Why the paddle ernie layout base model work badly than the service in huggingface.co/paddlepaddle/ernie-layout

Hi @ocrifydotnet
The model DocPrompt provided in HuggingFace has undergone secondary pre-training on a large-scale extraction &  question-answering dataset, building upon the ernie-layout pre-trained model.


ModuleNotFoundError: No module named 'locomotion'

When I want to use gym.
env = gym.make('motion_imitation:A1GymEnv-v1', render=True)
Error Occurred. I dont' know what's wrong with this, and wonder if someone could help fix it.
Traceback (most recent call last):
File "", line 1, in
File "/home/robot/anaconda2/envs/stone_legged/lib/python3.6/site-packages/gym/envs/registration.py", line 142, in make
return registry.make(id, **kwargs)
File "/home/robot/anaconda2/envs/stone_legged/lib/python3.6/site-packages/gym/envs/registration.py", line 87, in make
env = spec.make(**kwargs)
File "/home/robot/anaconda2/envs/stone_legged/lib/python3.6/site-packages/gym/envs/registration.py", line 58, in make
cls = load(self.entry_point)
File "/home/robot/anaconda2/envs/stone_legged/lib/python3.6/site-packages/gym/envs/registration.py", line 17, in load
mod = importlib.import_module(mod_name)
File "/home/robot/anaconda2/envs/stone_legged/lib/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 941, in _find_and_load_unlocked
File "", line 219, in _call_with_frames_removed
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 941, in _find_and_load_unlocked
File "", line 219, in _call_with_frames_removed
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'locomotion'

I have solved it, just modify the /motion_imitation/init.py as
register(
env_id='A1GymEnv-v1',
entry_point='motion_imitation.envs.gym_envs:A1GymEnv',
max_episode_steps=2000,
reward_threshold=2000.0,
)


Create state for live execution

In backtest and Quantopian real trading the algo's initialize method called only once: at the beginning of the execution.
It is required that live trading does not diverge from backtest with regards of execution (i.e. initialize is called only once).
It is also important to implement a robust live trading base which survives network outages & restarts.
Therefore state must be introduced to live-trading code path which is able to store the actual state ({{context}} variable, visible in initialize() & handle_data()) in disk. The logic to implement is simple:

During startup check if state file (pickle?) exists for the algo

If not: call algo's initialize() and store context to state file
Else: load context from state file

Save state file after every call to handle_data()

https://docs.python.org/3.4/library/pickle.html

I can take this, it is a simple task but it requires a number of classes to be added.

Thanks for taking this!
Besides algo's 'context' field we need to consider to include portfolio related items in this state
as not all the required fields could be derived from IB's API:

zipline.protocol.Positions: last_sale_price and last_sale_date
zipline.protocol.Portfolio: starting_cash, start_date
The generic stats (perf result of TradingAlgorithm) might be also helpful to store & extend each day.

What do you think?

I have the logic for this working, however not everything can be serialized with Pickle.  When I try to pickle the context I get the following error:
TypeError: can't pickle LRU objects
LRU refers to this external data structure.

TradingCalendar's _minute_to_session_label_cache attribute is of type LRU:
https://github.com/zipline-live/zipline/blob/live/zipline/utils/calendars/trading_calendar.py#L110

@pbharrin : I would not pickle cache objects, logic should be there to (re-)load the missing values.

I have the first version of this working, just need a few details figured out.
I think this should be a command line option, so a user can specify where to store the file.
What should be the default behavior if no filename is specified?  Not store the state?  Store it in a default filename?

I'd opt for always storing the state to a file, whose filename is generated using application's daily the start time.
That way we will accumulate more files over time but it would help debugging features.
What do you think?

Can we make this feature optional?

How do you imagine this being optional?
We cannot call initialize on every start (i.e. daily), because that's not how Q's live trading works....
Something I miss here?

Frankly speaking I consider saving state as an unneeded complication. We already have state - it's broker account state (available cash, open orders, etc). Inventing another one will just complicate implementation. You seem to have another opinion, hence I decided to at least make this optional.

I don't think state (i.e. context variable) can be derived from broker data.
What a broker provides is positions (in their representation) and account info, whereas
context stores all kind of user specified variables. Even the simplest algorithms (buy_and_hold) makes use of context.
As I would like to run algorithms (which are running on Quantopian Backtest and Live) without any modification this is a necessary change.

Hello all!
In regards to the "saving state" conversation, before using Zipline I used QuantConnect (a competitor). The servers were upgraded regularly, and when they were, the system shutdown any running algos. Depending on how the algos were programmed, some needed their "state" to be saved before shutting the algo down. Otherwise they needed to go through a "warm up" period before they could live trade again.
The particular algorithm I'm referring to, coded by myself and my colleague, used a Markov model to do pattern recognition. The Markov model needs to see a few days of data before it can begin to make accurate predictions, thus requiring its state to be saved.
QuantConnect didn't provide any local storage, so we weren't able to save its state. It created problems for the algo, so much so that we jumped ship to Zipline (and we're very happy we did!).
I'm just thinking out loud here, but in my opinion having the ability for an algo to save its state could be important, depending on how the algorithm is coded and what technology it uses.  I have some code I already have well tested using the Peewee ORM library to serialize data to a MySQL database. Peewee can also serialize to local disk into a SQLite file.
If you think it would make the development effort easier, I'd be happy to contribute my Peewee wrapper library. Or maybe I could lend with some code to serialize to a pickle file, if that's the preferred method.
Either way, my intention isn't to make things more complicated. Please let me know how I can help.

Given Quantopian's announcement yesterday regarding live trading, I came here. I'm glad there will be a mechanism for continuing to live trade.
This particular issue of state and the context variable(s) has become an area of great interest to me over the last week as I worked through some of Q's live paper trading nuances.  Specifically the divergence between what is displayed as "Positions" and the actual orders/fills.
I agree that state should be kept and that the context variables are a place to do that. However, I also believe that @bartosh makes an interesting point about the Broker (whomever they are) being the source of 'truth' regarding certain components of state. Perhaps the correct optionallity is that cash, positions, and trades(order/fills) can(I would argue should) be optionally populated from the broker data at start of day.

@rtntdeck : Most of the broker provided info is already populated through context.portfolio:
https://github.com/zipline-live/zipline/blob/d7f0cff4fa322a72cb4a80208585739d2d2d73a2/zipline/gens/brokers/ib_broker.py#L377-L443
Unfortunately you cannot load all the trades using IB API, only the active positions are presented.

@tibkiss that is good news. Thank you for the follow up and contributions.

@tibkiss I'd like to at least float the idea of a logical data separation. While instantiating Portfolio from the broker data simplifies state management, it also binds the brokerage account to a single algorithm. This may be a better approach overall. However, it forces users to create an account partition per algorithm. I believe that this is hard to do with some brokers (e.g. RobinHood).
Here is a proposed logical breakdown for consideration:

Algorithm: Positions, capital used, portfolio value, pnl, returns, start date
Broker: Account, Starting cash, cash
Performance tracker: Orders, transactions, performance metrics

Essentially, this allows multiple algorithms to run against the same brokerage account without mangling positions and performance metrics. Since cash and margin data still comes from the brokerage account, algorithms only spend what they can afford. Pickling up the portfolio and perf_tracker objects after every bar would be a crude way to preserve a full state.
I'm simply suggesting an alternative. I'm not sure which approach is best for most users.

@fredfortier : I appreciate the idea, but I have two concerns:

It deviates from Q's original API.  Q does not support multiple algos running agains the same account. Something what we don't enforce at the moment.
If you persist your position state and you make manual adjustments on your position size will result inconsistency. Handling such cases would mean that you manually adjust your pickled state (which sounds very error-prone).

As this idea has little to do with the persistence of state variable I'd suggest to move this discussion out of this thread.

Quick update on this issue:
I have extended @pbharrin 's state persistence branch a while ago and managed to load / store the context successfully by introducing black-list of variables which shall not be pickled.
Last challenge was to persist & load the scheduled functions. That's something which is difficult due to the complexity of the scheduled functions. My current workaround is to extend the API with a define_schedule() function which will be called every day. This approach is not the nicest (deviates from Q's API) but it works reasonably well.
Our friends at Quantopian hinted that it is worth-while to try to do the scheduling at before_trading_start() function, something I'll be experimenting with next week.
Once this problem is worked out I need to implement state matching to algo (if algo changes state loading shall not happen).
@pbharrin : As I have invested a reasonable amount of time here I'd be happy to take over this task from you and bring to completion. What do you think?

@tibkiss yes, please take this over.

I totally agree. I merely wanted to initiate the debate as I have a feeling
that it will be a concern as adoption picks up.
IMHO - Concern #2 is more of a feature than a bug. I don't believe that
users should be allowed to play god with their algorithm. Especially when
it's done by accident. For example, I once made a cash deposit to my
RobinHood which drastically skewed the performance statics of the
associated algorithm. If they want to change the parameters of an
algorithm, users should relaunch to keep performance statistic sane and
accurate.
More on this in a separate thread when this issue enters the spotlight.
On Fri, Aug 25, 2017 at 10:20 AM Peter Harrington<EMAIL_ADDRESS>wrote:

@tibkiss https://github.com/tibkiss yes, please take this over.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/zipline-live/zipline/issues/6#issuecomment-324934255,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABZ-Qvp4AlPRdYBfjT5He07gK1C1E1ejks5sbtgsgaJpZM4Nb-RH
.

First public version is here: https://github.com/zipline-live/zipline/pull/53
Reviews are welcome!

Delivered with: https://github.com/zipline-live/zipline/pull/53


Code cleanup: Breackout drop-out filter into util function

Currently the bike driver for flywheel uses a drop-out filter for sudden power-drops to 0W. The bike driver for keiser uses a similar drop-out filter, covering both power and cadence drop.
These filters could be moved into /util/ as a shared utility function. There should not be an issue with the flywheel bike also using the cadence drop filter part.

This is a good opportunity to wrap some tests around it too :-)


aws-c-common: 0.9.27 -> 0.9.28

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/awslabs/aws-c-common/releases.
meta.description for aws-c-common is: AWS SDK for C common core
meta.homepage for aws-c-common is: https://github.com/awslabs/aws-c-common
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS

[!CAUTION]
A test defined in passthru.tests did not pass.

found 0.9.28 in filename of file in /nix/store/2qynn33mcxjlii6py5bf4h76ad74c54h-aws-c-common-0.9.28

Rebuild report (if merged into master) (click to expand)

526 total rebuild path(s)

526 package rebuild(s)

First fifty rebuilds by attrpath

aider-chat
apacheHttpdPackages.mod_tile
apacheHttpdPackages_2_4.mod_tile
appvm
arrow-cpp
arrow-glib
attic-client
attic-server
aws-c-auth
aws-c-cal
aws-c-common
aws-c-compression
aws-c-event-stream
aws-c-http
aws-c-io
aws-c-mqtt
aws-c-s3
aws-c-sdkutils
aws-checksums
aws-crt-cpp
aws-sdk-cpp
browsr
bundix
cabal2nix
cached-nix-shell
cachix
ceph
ceph-client
ceph-csi
ceph-dev
chatgpt-retrieval-plugin
cloudcompare
colmena
common-updater-scripts
crate2nix
crystal2nix
databricks-sql-cli
datatrove
dbx
devenv
disko
dub-to-nix
dydisnix
entwine
fusionInventory
gdal
gmt
gplates
grass

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/2qynn33mcxjlii6py5bf4h76ad74c54h-aws-c-common-0.9.28 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A aws-c-common https://github.com/r-ryantm/nixpkgs/archive/90e144b71747adb104df097664e328dff2bf5c09.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/90e144b71747adb104df097664e328dff2bf5c09#aws-c-common

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/2qynn33mcxjlii6py5bf4h76ad74c54h-aws-c-common-0.9.28
ls -la /nix/store/2qynn33mcxjlii6py5bf4h76ad74c54h-aws-c-common-0.9.28/bin

Pre-merge build results
NixPkgs review skipped

Maintainer pings
cc @orivej @r-burns for testing.

[!TIP]
As a maintainer, if your package is located under pkgs/by-name/*, you can comment @NixOS/nixpkgs-merge-bot merge to automatically merge this update using the nixpkgs-merge-bot.

Add a :+1: reaction to pull requests you find important.

Merge conflict, please open a new PR.

r-ryantm shouldn't even attempt to update the AWS packages IMO. They usually are updated en masse because individual AWS packages often introduce breaking changes that require them all to be updated in lockstep.

Then I guess we can do that?

Updates can be disabled by adding a comment to the package:
# nixpkgs-update: no auto update

(From https://github.com/nix-community/nixpkgs-update/blob/main/doc/nixpkgs-maintainer-faq.md#disabling-package-updates)

Thanks! Didn't know it's that easy. Let me go ahead and issue a new mass update, and I'll add that comment to all the relevant packages.


cause of '500 "Internal Server Error"' at /api/backtesting

Note: this is the technical bug tracker, please use other platforms for getting support and starting a (non technical) discussion. See the getting help page for details.
I'm submitting a ...
[x] bug report
[ ] question about the decisions made in the repository
Action taken (what you did)
backtesting with gekkoga with correctly data range , and surely all data was imported
Expected result (what you hoped would happen)
finish the test and give out result
Actual result (unexpected outcome)
500 "Internal Server Error" throw at /api/backtest, and cause gekkoga throw this error too.
Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, etc)
@askmike @generalectric
I have search all the related issues on gekko & gekkoga, and all the solution not help.
so after print trace some debug information, i found this issue is cause by the problem of implement of [stream] (https://nodejs.org/api/stream.html#stream_buffering) in gekko
api/backtest will randomly stop on this line
this.push(c);

I guess the buffer increase to the [highWaterMark] (https://nodejs.org/api/stream.html#stream_readable_readablehighwatermark) limit of stream buffer. and make it stop with error.
because consumer's process speed is lower then reader push() speed. this stream implementing is little complex , I have no idea how to fix it yet.
@askmike  in the implement, I saw you use readable.pipe(writeable) to flowing data to gekkostream.  in theory it will automatically handle the read/write consumer speed problem.  it's wired.
ref: 500 error on gekkoga

Is there any output in the terminal you ran gekko?

If gekko actually executes the strategy there will be errors in the terminal in which you're running server.js. If you fail to capture errors there then the strat isn't being executed by the api. This is always because of stratname to filename mismatch or dataset not available. Are you using Mongo pgsql or sqlite? Have you edited baseconfig for the api to make sure it's using correct db/credentials?

@askmike @generalectric yes, there is error output in terminal:
  xxx POST /api/backtest 500 281ms -

  Error: non-error thrown: Child process has died.
      at Object.onerror (/home/projects/gekko/node_modules/koa/lib/context.js:105:40)
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:118:7)

i add some code to trace it:
Market.prototype.processCandles = function(err, candles) {
  this.pushing = true;
  var amount = _.size(candles);
......
  console.log(`amount: ${amount}`)
  _.each(candles, function(c, i) {
    c.start = moment.unix(c.start);
    console.log('trying to push');
    console.log(i)
    this.push(c);
    console.log('pushed');
  }, this);
....

here are the outputs:
amount: 50
trying to push
0
pushed
trying to push
1
pushed
trying to push
2
pushed
trying to push
3
pushed
trying to push
4
pushed
trying to push
5
pushed
trying to push
6
pushed
trying to push
7
pushed
trying to push
8
pushed
trying to push
9
pushed
trying to push
10
pushed
trying to push
11
pushed
trying to push
12
pushed
trying to push
13
pushed
trying to push
14
pushed
trying to push
15
pushed
trying to push
16
pushed
trying to push
17
pushed
trying to push
18
pushed
trying to push
19
pushed
trying to push
20
pushed
trying to push
21
pushed
trying to push
22
pushed
trying to push
23
pushed
trying to push
24
pushed
trying to push
25
pushed
trying to push
26
pushed
trying to push
27
pushed
trying to push
28
pushed
trying to push
29
  xxx POST /api/backtest 500 207ms -

  Error: non-error thrown: Child process has died.
      at Object.onerror (/home/projects/gekko/node_modules/koa/lib/context.js:105:40)
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:118:7)

I wouldn't look that much into that piece of code: when a candle is pushed
down the stream a ton of code runs (the gekko stream, all plugins, your
strategy maybe). Not seeing the push log means that "something" failed.
I will have another look at capturing stderr messages from a childprocess
and making sure they go to the parent process.
On Wed, 21 Mar 2018, 08:44 Jas<EMAIL_ADDRESS>wrote:

i add some code to trace it:
Market.prototype.processCandles = function(err, candles) {
this.pushing = true;
var amount = _.size(candles);......
console.log(amount: ${amount})
_.each(candles, function(c, i) {
c.start = moment.unix(c.start);
console.log('trying to push');
console.log(i)
this.push(c);
console.log('pushed');
}, this);....
here are the outputs:
amount: 50
trying to push
0
pushed
trying to push
1
pushed
trying to push
2
pushed
trying to push
3
pushed
trying to push
4
pushed
trying to push
5
pushed
trying to push
6
pushed
trying to push
7
pushed
trying to push
8
pushed
trying to push
9
pushed
trying to push
10
pushed
trying to push
11
pushed
trying to push
12
pushed
trying to push
13
pushed
trying to push
14
pushed
trying to push
15
pushed
trying to push
16
pushed
trying to push
17
pushed
trying to push
18
pushed
trying to push
19
pushed
trying to push
20
pushed
trying to push
21
pushed
trying to push
22
pushed
trying to push
23
pushed
trying to push
24
pushed
trying to push
25
pushed
trying to push
26
pushed
trying to push
27
pushed
trying to push
28
pushed
trying to push
29
xxx POST /api/backtest 500 207ms -
Error: non-error thrown: Child process has died.
at Object.onerror (/home/projects/gekko/node_modules/koa/lib/context.js:105:40)
at
at process._tickCallback (internal/process/next_tick.js:118:7)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/askmike/gekko/issues/2040#issuecomment-374813057, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7MD9yGXVlbEhzC6CAzPUmhtw-D_T4qks5tgbB1gaJpZM4Sxzfm
.

@askmike thanks for help.
there are 3 plugin running at my backtest:
 _.each(candleConsumers, function(c){
        console.log(`c.meta.name: ${c.meta.name}`);
      });

c.meta.name: Trading Advisor
c.meta.name: Paper Trader
c.meta.name: Performance Analyzer

so , i tried to try catch all their processCandle method, like this:
...
PaperTrader.prototype.processCandle = function(candle, done) {
  try {
    this.price = candle.close;

    if(!this.portfolio.balance)
      this.setStartBalance();

    done();
  } catch (error) {
    console.log(error);
    throw error;
  }
}

module.exports = PaperTrader;

and try catch all the executed methods (init,log,check,end) in my backtesting strategy (RSI_BULL_BEAR_ADX.js)
there are no any exceptions throw at all.

in my backtesting strategy (RSI_BULL_BEAR_ADX.js)

Quick question: does this strategy use async indicators (either talib or tulip)?

yes, it used tulip, here is the RSI_BULL_BEAR_ADX strategy

@JasLin ah I edited my comment but I guess too late. Can you try logging the uncaught exception (see above)?

thanks mike.
i  tried that, added the following code:
process.on('uncaughtException', (err) => {
    console.log(err);
  });

to the following file:
	modified:   core/gekkoStream.js
	modified:   core/markets/backtest.js
	modified:   core/pipeline.js
	modified:   core/workers/pipeline/child.js
	modified:   core/workers/pipeline/parent.js
	modified:   plugins/paperTrader/paperTrader.js
	modified:   plugins/performanceAnalyzer/performanceAnalyzer.js
	modified:   plugins/tradingAdvisor/tradingAdvisor.js
	modified:   strategies/RSI_BULL_BEAR_ADX.js
	modified:   web/routes/backtest.js

and still the same output without any other helpful information.

@JasLin I finally figured out how to properly log the error happening in the a child process. You can fix it locally by applying the changes from this commit: 6ae42a21e314901690d7695f07adee8327168149

@askmike  thanks bro. i cherry-picked  this commit and applyed it to develp branch. now we got this :
 ERROR: TULIP ERROR:

  xxx POST /api/backtest 500 222ms -

  Error: non-error thrown: Child process has died.

the error is from TULIP
it's there any way to print more information?

it looks like the actual tulip error is not properly logged in the first place. Could you add some stuff over here:
https://github.com/askmike/gekko/blob/2a07c609f3da1629c3e585f702e743f4488dd7ca/plugins/tradingAdvisor/baseTradingMethod.js#L185
You can use log.error for example to log more info. It seems like the err prop was passed to util.die as a second param, which I think is ignored.

Gekko bundles the candle properties together with your indicator
parameters, are they configured correctly?
On Mon, Mar 26, 2018 at 5:58 PM, Jas<EMAIL_ADDRESS>wrote:

got more detail from tulip: Invalid options. and print the input option
with following code:
...
log.info(basectx.candleProps);
_.each(
this.tulipIndicators,
indicator => indicator.run(
basectx.candleProps,
tulindResultHander.bind(indicator)
)
);
}...
here we got the option:
2018-03-26 18:52:53 (INFO):	{ open: [ 6385.02 ],
high: [ 6398 ],
low: [ 6385.02 ],
close: [ 6398 ],
volume: [ 38.28753715999999 ],
vwp: [ 6393.491076272645 ],
trades: [ 200 ] }
2018-03-26 18:52:53 (ERROR):	Invalid options
ERROR: TULIP ERROR:
xxx POST /api/backtest 500 263ms -
Error: non-error thrown: Child process has died.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/askmike/gekko/issues/2040#issuecomment-376127268, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7MD3zUnz6d0DBoootrQlzZXmS233CEks5tiMndgaJpZM4Sxzfm
.

--
PGP key at keybase.io/mikevanrossum
https://keybase.io/mikevanrossum/key.asc

@askmike  thanks for your reply.
the config it's send from gekkoga.
i guess that maybe i missed something here.  i will try to dig into it.  @generalectric  can you give me some tips here?
here are the ga configuratins:
const randomExt = require('random-ext');

const config = {
  stratName: 'RSI_BULL_BEAR_ADX',
  gekkoConfig: {
    watch: {
      exchange: 'gdax',
      currency: 'USD',
      asset: 'BTC'
    },
    //daterange: 'scan',
    daterange: {
      from: '2017-11-01 01:00',
      to: '2017-11-13 15:00'
    },

    simulationBalance: {
      'asset': 1,
      'currency': 100
    },
    slippage: 0.05,
    feeTaker: 0.25,
    feeMaker: 0.15,
    feeUsing: 'maker', // maker || taker
    backtest: {
      batchSize: 20
    }
  },
  apiUrl: 'http://localhost:3000',
  // Population size, better reduce this for larger data
  populationAmt: 2,
  // How many completely new units will be added to the population (Population * variation must be a whole number!!)
  variation: 0.5,
  // How many components maximum to mutate at once
  mutateElements: 8,

  // How many parallel queries to run at once
  parallelqueries: 1,
  // profit || score
  // score = ideas? feedback?
  // profit = recommended!
  mainObjective: 'profit',

  // optionally recieve and archive new all time high every new all time high
  notifications: {
    email: {
      enabled: false,
      receiver<EMAIL_ADDRESS>      senderservice: 'gmail',
      sender<EMAIL_ADDRESS>      senderpass: 'password',
    },
  },

  /*
   When the algorithm reaches this value it will stop,
   but you can stop it any time you wish since the last max parameters are outputted every epoch
   */
  targetValue:<PHONE_NUMBER>,

  candleValues: [5,15,30,60,120,240],
  getProperties: () => ({
    // Here add the indicators and the ranges you want to handle
    // In this case my strategy wants to test RSI and MACD ranges
    historySize: randomExt.integer(100, 10),
    candleSize: randomExt.pick(config.candleValues),
    SMA_long: randomExt.integer(1000,  200),
    SMA_short: randomExt.integer(120, 12),
    BULL_RSI: randomExt.integer(50, 10),
    BULL_RSI_high: randomExt.integer(120, 50),
    BULL_RSI_low: randomExt.integer(60, 10),
    BEAR_RSI: randomExt.integer(15, 5),
    BEAR_RSI_high: randomExt.integer(80, 30),
    BEAR_RSI_low: randomExt.integer(50, 20),
    BULL_MOD_high: randomExt.integer(10, 3),
    BULL_MOD_low: randomExt.integer(-3, -10),
    BEAR_MOD_high: randomExt.integer(20, 10),
    BEAR_MOD_low: randomExt.integer(-3, -10),
    ADX: randomExt.integer(10, 1),
    ADX_high: randomExt.integer(80, 50),
    ADX_low: randomExt.integer(50, 3)
  })
};

module.exports = config;

@JasLin you are using an old version of gekkoga. Pull the newest version from the repo and start with the example config

I am assuming this is fixed in either gekkoga or gekko 0.6.

I have just encountered it in 0.6.1, with gekko ui 0.2.2. Not using any talib or tulip indicators, and running in docker.

Are there any errors in the commandline prompt?
On Wed, Jul 11, 2018 at 10:56 PM, tanevanwifferen<EMAIL_ADDRESS>wrote:

I have just encountered it in 0.6.1, with gekko ui 0.2.2. Not using any
talib or tulip indicators, and running in docker.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/askmike/gekko/issues/2040#issuecomment-404200040, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7MD_Uw1uZ47XK4tQ4Dk4PgSaOONO7gks5uFhIegaJpZM4Sxzfm
.

--
PGP key at keybase.io/mikevanrossum
https://keybase.io/mikevanrossum/key.asc

xxx POST /api/backtest 500 127ms -
Error: non-error thrown: [object Object]     at Object.onerror (/usr/src/app/node-modules/koa/lib/context.js:105:40)     at <anonymous>     at process._tickCallback (internal/pprocess/next_tick.js:188:7)
It does work with some strategies, CCI works for example, and I would think that it's my own fault running a custom strategy. I have written a simple turtle trading strategy, and I think it should work..
`
var log = require('../core/log.js');
var method = {};
method.init = function() {
log.debug("init");
this.name = 'Turtle trading';
//this.requiredHistory = this.settings.totalTicks;
this.ticksLong = this.settings.ticksLong;
this.ticksShort = this.settings.ticksShort;
this.maxRisk = this.settings.maxRisk;
this.position = 0;
this.ticks = [];
};
method.update = function(candle) {
log.debug(candle);
this.ticks.push(candle);
if(this.ticks.length > this.requiredHistory){
this.ticks.shift();
}
};
method.check = function() {
log.debug("check");
var lastTick = this.ticks.slice(-1);
log.debug(lastTick.close);
if(this.position != 0){
if(this.position >= lastTick.close){
this.position = 0;
this.advice('short');
}
} else if (lastTick.close >= Math.max(...this.ticks.map((el)=>el.close).slice(ticksLong * -1))){
this.position = lastTick.close * (1 - this.maxRisk);
this.advice('long');
return;
} else if(lastTick.close <= Math.min(...this.ticks.map((el)=>el.close).slice(ticksShort * -1)){
this.position = 0;
this.advice('short');
} else {
this.advice();
}
};
module.exports = method;
`

Just a guess: Maybe the spread operator in Math.max() / Math.min() causes the problem?

Edited the code to work, it was all my fault. you can see the changes I did in the history of the comment

Hello,
i have this problem using a fresh install of gekko either the stable branch and the develop branch.
same output, error 500 as well on all the tulip strategies and talib.


Tokens with no expiration are not allowed #470

I am implementing personal access tokens (JWT) as a replacement for Api Key. The token is issued by a separate endpoint (not openiddict).
I have kept the structure similar to the jwt tokens issued by the openiddidct token endpoint.
Everything works fine, but if I create a personal access token with no expiry (no exp claim), openiddict middleware rejects the request:
"www-authenticate": "Bearer error="invalid_token", error_description="The token has no expiration"",
How can I prevent that from happening? Is there some config to allow that.
Please note I am allowing users to revoke the personal access token by storing jti in the database for personal access tokens.
https://github.com/openiddict/openiddict-core/issues/470

https://github.com/openiddict/openiddict-core/issues/470

Setting RequireExpirationTime = false in TokenValidationParameters fixed the issue