Fails to Make Deal with Error Not Enough Miners from Reputation Module to Satisfy The Constraints Error

I am running WSL Ubuntu 20.04 (Version 2 with Docker Desktop Support) within Windows 10 Pro Version 21H1
The steps are as follows:

git clone https://github.com/textileio/powergate.git
cd powergate/
cd docker/
nano docker-compose.yaml where I added "["lotus", "daemon", "--import-snapshot", "https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car"]" between lines 32 and 33.
make up
Waited for the node to finish importing and then syncing.
^C then make down then deleted the line "["lotus", "daemon", "--import-snapshot", "https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car"]" from docker-compose.yaml
make up
Now that the node was running I typed cd .. so I was in the repo's root directory, then make install-pow
with the pow command in my GOPATH I typed pow to make sure pow was linked fine to powd. It was.
pow admin users create
copied the token and ran export POW_TOKEN=<token copied to here>
Then pow wallet addrs and funded the address
I went to the directory behind the folder of my static website which is about 5GB in size.
I typed pow data stage <my-static-site-folder>
After it was finished staging and printed out the CID I typed pow config apply --watch <CID waited a long time while it said the job was executing and then I got...

---------------------------------------+--------------------------------+-------+-------+--------------
             <job id here>             | JOB_STATUS_FAILED executing    |       |       |
                                       | cold-storage config: making    |       |       |
                                       | deal configs: getting miners   |       |       |
                                       | from minerselector: getting    |       |       |
                                       | miners from reputation         |       |       |
                                       | module: not enough miners from |       |       |
                                       | reputation module to satisfy   |       |       |
                                       | the constraints                |       |       |

I don't understand what the problem is. I repeated the pow config apply --watch <CID command each time adding the --override flag with several different modifications to a custom config file. The content did appear briefly on IPFS (not Filecoin), but after I continued running the config apply command the site went down from IPFS.
Here are my powergate logs.
<myusername>@<mycomputername>:~/dev/powergate_config_files$ docker logs mainnet_powergate_1
2021-07-26T09:04:20.618Z        INFO    powd    powd/main.go:57 starting powd:
        version:        git
        build date:     2021-07-26T07:45:07Z
        git summary:    v2.6.2-dirty
        git branch:     master
        git commit:     34dc0fe
        git state:      dirty
2021-07-26T09:04:20.619Z        INFO    powd    powd/main.go:72 {
  "RepoPath": "/root/powergate/.powergate",
  "MaxMindDBFolder": ".",
  "Devnet": false,
  "IpfsAPIAddr": "/dns4/ipfs/tcp/5001",
  "LotusAddress": "/dns4/lotus/tcp/1234",
  "LotusAuthToken": "<LotusAuthTokenHere>",
  "LotusMasterAddr": "",
  "LotusConnectionRetries": 180,
  "GrpcHostNetwork": "tcp",
  "GrpcHostAddress": "/ip4/<IP_ADDRESS>/tcp/5002",
  "GrpcServerOpts": null,
  "GrpcWebProxyAddress": "<IP_ADDRESS>:6002",
  "GatewayBasePath": "/",
  "GatewayHostAddr": "<IP_ADDRESS>:7000",
  "IndexRawJSONHostAddr": "<IP_ADDRESS>:8889",
  "MongoURI": "",
  "MongoDB": "",
  "FFSAdminToken": "",
  "FFSUseMasterAddr": false,
  "FFSDealFinalityTimeout":<PHONE_NUMBER>00000,
  "FFSMinimumPieceSize": 67108864,
  "FFSRetrievalNextEventTimeout":<PHONE_NUMBER>000,
  "FFSMaxParallelDealPreparing": 2,
  "FFSGCAutomaticGCInterval":<PHONE_NUMBER>000,
  "FFSGCStageGracePeriod":<PHONE_NUMBER>000,
  "SchedMaxParallel": 1000,
  "MinerSelector": "reputation",
  "MinerSelectorParams": "",
  "DealWatchPollDuration":<PHONE_NUMBER>00,
  "AutocreateMasterAddr": false,
  "WalletInitialFunds": {},
  "AskIndexQueryAskTimeout":<PHONE_NUMBER>0,
  "AskindexMaxParallel": 3,
  "AskIndexRefreshInterval":<PHONE_NUMBER>0000,
  "AskIndexRefreshOnStart": false,
  "IndexMinersRefreshOnStart": false,
  "IndexMinersOnChainMaxParallel": 20,
  "IndexMinersOnChainFrequency":<PHONE_NUMBER>0000,
  "DisableIndices": false,
  "DisableNonCompliantAPIs": false
}
2021-07-26T09:04:20.619Z        INFO    powd    powd/main.go:75 starting server...
2021-07-26T09:04:20.620Z        WARN    lotus-client    lotus/client.go:50      failed to connect to Lotus client dial tcp <IP_ADDRESS>:1234: connect: connection refused, retrying...
2021-07-26T09:04:30.621Z        WARN    lotus-client    lotus/client.go:50      failed to connect to Lotus client dial tcp <IP_ADDRESS>:1234: connect: connection refused, retrying...
2021-07-26T09:04:40.622Z        WARN    lotus-client    lotus/client.go:50      failed to connect to Lotus client dial tcp <IP_ADDRESS>:1234: connect: connection refused, retrying...
2021-07-26T09:04:50.626Z        INFO    server  server/server.go:196    Detected Lotus node connected to network: mainnet
2021-07-26T09:05:03.233Z        INFO    fchost  fchost/fchost.go:116    connected to 11 out of 15 bootstrap peers
2021-07-26T09:05:03.233Z        INFO    fchost  fchost/fchost.go:60     bootstraping libp2p host dht
2021-07-26T09:05:03.233Z        INFO    fchost  fchost/fchost.go:64     dht bootstraped!
2021-07-26T09:05:03.233Z        INFO    server  server/server.go:684    Ensuring migrations...
2021-07-26T09:05:03.233Z        INFO    server  server/server.go:595    Opening badger database...
2021-07-26T09:05:03.237Z        INFO    migrations      migration/migration.go:67       Current datastore version is 5, target version 5
2021-07-26T09:05:03.237Z        INFO    server  server/server.go:699    Migrations ensured
2021-07-26T09:05:03.239Z        INFO    server  server/server.go:595    Opening badger database...
2021-07-26T09:05:03.247Z        INFO    server  server/server.go:217    Wiring internal components...
2021-07-26T09:05:03.247Z        INFO    server  server/server.go:229    Starting ask index...
2021-07-26T09:05:03.247Z        INFO    index-ask       runner/runner.go:65     loaded persisted index with 0 entries
2021-07-26T09:05:03.247Z        INFO    server  server/server.go:235    Starting miner index...
2021-07-26T09:05:03.247Z        INFO    server  server/server.go:247    Starting faults index...
2021-07-26T09:05:03.247Z        INFO    server  server/server.go:256    Starting deals module...
2021-07-26T09:05:03.247Z        INFO    deals   module/module.go:51     creating deal watcher
2021-07-26T09:05:03.247Z        INFO    deals   module/module.go:66     resuming pending records
2021-07-26T09:05:03.247Z        INFO    deals   module/records.go:170   resuming 0 pending records
2021-07-26T09:05:03.247Z        INFO    deals   module/records.go:179   resumed watching 0 pending storage deal records
2021-07-26T09:05:03.247Z        INFO    server  server/server.go:262    Starting wallet module...
2021-07-26T09:05:03.247Z        INFO    server  server/server.go:291    Starting FFS scheduler...
2021-07-26T09:05:03.247Z        INFO    reputation      reputation/reputation.go:200    rebuilding index
2021-07-26T09:05:03.247Z        INFO    server  server/server.go:307    Starting gRPC, gateway and index HTTP servers...
2021-07-26T09:05:03.247Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 0, total executing: 0
2021-07-26T09:05:03.247Z        INFO    reputation      reputation/reputation.go:227    built scores for 0 miners in 0ms
2021-07-26T09:05:03.247Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 0, total executing: 0
2021-07-26T09:05:03.247Z        INFO    server  server/server.go:366    Starting IPFS reverse proxy...
2021-07-26T09:05:03.248Z        INFO    deals-watcher   dealwatcher/dealwatcher.go:144  deal watcher created
2021-07-26T09:05:03.248Z        INFO    server  server/server.go:355    Starting finished, serving requests
2021-07-26T09:05:03.248Z        INFO    powd    powd/main.go:80 server started.
2021-07-26T09:06:50.631Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 367
2021-07-26T09:08:50.638Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 352
2021-07-26T09:10:50.642Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 325
2021-07-26T09:12:50.645Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 277
2021-07-26T09:14:50.650Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 239
2021-07-26T09:16:50.654Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 191
2021-07-26T09:18:50.662Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 142
2021-07-26T09:20:50.668Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 103
2021-07-26T09:22:50.671Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 54
2021-07-26T09:24:50.676Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 370, todo: 8
2021-07-26T09:26:50.681Z        WARN    lotus-client    lotus/metrics.go:90     Louts behind in syncing with height diff 39, todo: 7
2021-07-26T10:05:03.251Z        INFO    ffs-scheduler   scheduler/scheduler.go:208      running scheduler gc...
2021-07-26T10:05:03.259Z        INFO    ffs-scheduler   scheduler/scheduler.go:236      scheduler gc ran with 0 excluded cids, unpinning 0 staged cids
2021-07-26T10:35:18.496Z        INFO    ffs-manager     manager/manager.go:109  creating instance
2021-07-26T10:35:18.528Z        INFO    ffs-auth        auth/auth.go:38 generating auth-token for instance <instanceIdHere>
2021-07-26T11:00:21.977Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Pushing new configuration...
2021-07-26T11:00:21.995Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Configuration saved successfully
2021-07-26T11:00:21.995Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 1, total executing: 0
2021-07-26T11:00:22.005Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 0, total executing: 1
2021-07-26T11:00:22.006Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Executing job <JobIdHere>...
2021-07-26T11:00:22.009Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Automatically staging Cid from the IPFS network...
2021-07-26T11:00:22.019Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Executing Cold-Storage configuration...
2021-07-26T11:00:22.022Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Current replication factor is lower than desired, making 1 new deals...
2021-07-26T11:00:22.024Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Entering deal preprocessing queue...
2021-07-26T11:00:22.026Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Calculating piece size...
2021-07-26T11:01:36.270Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       The payload size is 5.1 GiB, and the calculated piece size is 8.0 GiB
2021-07-26T11:01:36.278Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Starting deals failed, with cause: making deal configs: getting miners from minerselector: getting miners from reputation module: not enough miners from reputation module to satisfy the constraints
2021-07-26T11:01:36.280Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Cold-Storage execution failed.
2021-07-26T11:01:36.282Z        ERROR   ffs-scheduler   scheduler/scheduler.go:531      executing job <JobIdHere>: executing cold-storage config: making deal configs: getting miners from minerselector: getting miners from reputation module: not enough miners from reputation module to satisfy the constraints
2021-07-26T11:01:36.285Z        INFO    ffs-sched-sjstore       sjstore/sjstore.go:702  notifying watcher
2021-07-26T11:01:36.285Z        INFO    ffs-cidlogger   joblogger/joblogger.go:53       Job <JobIdHere> execution failed: executing cold-storage config: making deal configs: getting miners from minerselector: getting miners from reputation module: not enough miners from reputation module to satisfy the constraints
2021-07-26T11:01:36.287Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 0, total executing: 0
2021-07-26T11:01:36.288Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 0, total executing: 0
2021-07-26T11:01:36.288Z        INFO    ffs-scheduler   scheduler/scheduler.go:353      storage job total queued: 0, total executing: 0
<myusername>@<mycomputername>:~/dev/powergate_config_files$

Here are my IPFS logs:
<myusername>@<mycomputername>:~/dev/powergate_config_files$ docker logs mainnet_ipfs_1
Changing user to ipfs
ipfs version 0.7.0
Found IPFS fs-repo at /data/ipfs
Initializing daemon...
go-ipfs version: 0.7.0-ea77213
Repo version: 10
System version: amd64/linux
Golang version: go1.14.4
Swarm listening on /ip4/<IP_ADDRESS>/tcp/4001
Swarm listening on /ip4/<IP_ADDRESS>/udp/4001/quic
Swarm listening on /ip4/<IP_ADDRESS>/tcp/4001
Swarm listening on /ip4/<IP_ADDRESS>/udp/4001/quic
Swarm listening on /p2p-circuit
Swarm announcing /ip4/<IP_ADDRESS>/tcp/4001
Swarm announcing /ip4/<IP_ADDRESS>/udp/4001/quic
Swarm announcing /ip4/<IP_ADDRESS>/tcp/4001
Swarm announcing /ip4/<IP_ADDRESS>/udp/4001/quic
Swarm announcing /ip4/<IP_ADDRESS>/udp/55230/quic
API server listening on /ip4/<IP_ADDRESS>/tcp/5001
WebUI: http://<IP_ADDRESS>:5001/webui
Gateway (readonly) server listening on /ip4/<IP_ADDRESS>/tcp/8080
Daemon is ready
<myusername>@<mycomputername>:~/dev/powergate_config_files$

Please let me know what the problem is, I really love Filecoin and Powergate and want to take part in the ecosystem.

The problem here is that you're relying on the default behavior of auto-miner selection, which is brittle if your miner-index part of the system hasn't built the index or the safe values used for deal making (e.g: price) doesn't match whatever is available.
Miner selection is a hard problem in general, but the only way to avoid this problem is that you provide the miners you want to store the data with using the TrustedMiner (https://docs.textile.io/powergate/storageconfig). That will try first with the miners you specified there. You could pow config default > def.config, then edit def.config, and then pow config set-default to set it as the default, so your next pow apply can use it and try targeting better miners.
From where you would get miners to set that field? Is a tricky question. You can use Filfox to try finding some miners, or joining #powergate-users in the Lotus slack channel to see if anyone from the community can share miners that they're using now and are working.

Thank you so much! This is excellent it looks like the deal is going through. I looked up miners with a high reputation score in North America on https://filrep.io/ and added them to the trusted miners section of the config json file. It looks like it made a deal but the process appears to be hanging on "StorageDealTransferring" I'm assuming it's uploading my directory to the miner right?

It's still not working... After all this lol. It is stuck on StorageDealCheckForAcceptance. It's just hangs there. I killed the process because I used the --watch flag when I did the deal command. Here is the output that was hanging for the deal creation command:
                 JOB ID                |        STATUS        |  MINER   |    PRICE     |          DEAL STATUS
---------------------------------------+----------------------+----------+--------------+--------------------------------
              <Job Id Here>            | JOB_STATUS_EXECUTING |          |              |
                                       |                      | <miner>  |<PHONE_NUMBER>00 | StorageDealCheckForAcceptance

When I ran pow storage-job watch <CID of Directory>, I got this:
                            JOB ID                            |     STATUS     | MINER | PRICE | DEAL STATUS
--------------------------------------------------------------+----------------+-------+-------+--------------
               <CID of The Actual Directory>                  | awaiting state |       |       |
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x<hex string here> pc=0x<hexstringhere>]

goroutine 1 [running]:
github.com/textileio/powergate/v2/cmd/pow/common.WatchJobIds(0x<Hex string here>, 0x1, 0x1)
        /home/warpanomalythree/dev/powergate/cmd/pow/common/common.go:110 +0x41a
github.com/textileio/powergate/v2/cmd/pow/cmd/storagejobs/watch.glob..func2(0x<Hex string here>, 0x<Hex string here>, 0x1, 0x1)
        /home/warpanomalythree/dev/powergate/cmd/pow/cmd/storagejobs/watch/watch.go:23 +0x<Hex string here>
github.com/spf13/cobra.(*Command).execute(0x<Hex string here>, 0x<Hex string here>, 0x1, 0x1, 0x<Hex string here>, 0x<Hex string here>)
        /home/warpanomalythree/golibs/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856 +0x<Hex string here>
github.com/spf13/cobra.(*Command).ExecuteC(0x<Hex string here>, 0x<Hex string here>, 0x<Hex string here>, 0x<Hex string here>)
        /home/warpanomalythree/golibs/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x<Hex string here>
github.com/spf13/cobra.(*Command).Execute(...)
        /home/warpanomalythree/golibs/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
main.main()
        /home/warpanomalythree/dev/powergate/cmd/pow/main.go:11 +0x<Hex string here>

Also, this is the output of pow storage-job list for the relevant job:
"storageJobs":  [
    {
      "id":  "<This Job's Id>",
      "apiId":  "<Id string I don't recoignize>",
      "cid":  "<File Directory's CID>",
      "status":  "JOB_STATUS_EXECUTING",
      "errorCause":  "",
      "dealInfo":  [
        {
          "proposalCid":  "<CID I don't recognize>",
          "stateId":  "29",
          "stateName":  "StorageDealAwaitingPreCommit",
          "miner":  "<Miner Id>",
          "pieceCid":  "<Hex String I Don't Recognize>",
          "size":  "8522825728",
          "pricePerEpoch":  "800000000000",
          "startEpoch":  "0",
          "duration":  "523192",
          "dealId":  "<Number Here>",
          "activationEpoch":  "0",
          "message":  ""
        }
      ],
      "dealErrors":  [],
      "createdAt":  "<Number Here>"
    },

One more thought... Could my problem be that the directory to be stored is too big? I have one file in the directory that's about 5GB.

Actually, things are going pretty well for you (apart from that CLI panic, but looks like hasn't stopped you from moving further).
Both StorageDealCheckForAcceptance and  StorageDealAwaitingPreCommit are expected states of deals that can take a long time to go through, in the order of hours. A Filecoin deal can take one or even two days depending on the miner to be totally confirmed.
So, it isn't really stuck but those states are really the expected ones and the transitions take hours. That fact doesn't depend on Powergate (or any Filecoin client), is just the way that the network works since miners have to do many things from receiving your proposal until the data gets fully confirmed (packing, publishing, sealing, proving).

Actually, things are going pretty well for you (apart from that CLI panic, but looks like hasn't stopped you from moving further).
Both StorageDealCheckForAcceptance and StorageDealAwaitingPreCommit are expected states of deals that can take a long time to go through, in the order of hours. A Filecoin deal can take one or even two days depending on the miner to be totally confirmed.
So, it isn't really stuck but those states are really the expected ones and the transitions take hours. That fact doesn't depend on Powergate (or any Filecoin client), is just the way that the network works since miners have to do many things from receiving your proposal until the data gets fully confirmed (packing, publishing, sealing, proving).

Oh that's good to know. I'm glad it's actually going well. So if I just leave my powergate docker stack running for a few more days it will upload the files to the miner and the deal will go through? And the powergate stack will automatically stream upload the files to the miner? Is there any command to check if the file has been uploaded to the miner yet?
Btw one more thing, how do I back up my wallet's private key? I can't seem to find the .powergate directory to get the file that contains my wallet's private key. I am running this as the docker mainnet version.

Oh that's good to know. I'm glad it's actually going well. So if I just leave my powergate docker stack running for a few more days it will upload the files to the miner and the deal will go through? And the powergate stack will automatically stream upload the files to the miner? Is there any command to check if the file has been uploaded to the miner yet?

Yes, everything needed is done by Powergate to store the data in the miner. Things can fail in general mostly due to failed transfers, or the miner having some problem for some reason. The way you know how things are going are mostly checking the status of the Job that will switch from executing to successful (if all went well), or pow data log <cid> to see the log stream too.

Btw one more thing, how do I back up my wallet's private key? I can't seem to find the .powergate directory to get the file that contains my wallet's private key. I am running this as the docker mainnet version.

The private keys only live in the Lotus node, so if you want to export it, you need to docker run inside the Lotus container and lotus wallet export it.

I'm glad it appears to be working. This is the output of pow data log <CID of The Actual Directory> :
> 2021-07-26T07:00:21 - Pushing new configuration...
> 2021-07-26T07:00:21 - Configuration saved successfully
> 2021-07-26T07:00:22 - Executing job <CID of Previous Failed Job>...
> 2021-07-26T07:00:22 - Automatically staging Cid from the IPFS network...
> 2021-07-26T07:00:22 - Executing Cold-Storage configuration...
> 2021-07-26T07:00:22 - Current replication factor is lower than desired, making 1 new deals...
> 2021-07-26T07:00:22 - Entering deal preprocessing queue...
> 2021-07-26T07:00:22 - Calculating piece size...
> 2021-07-26T07:01:36 - The payload size is 5.1 GiB, and the calculated piece size is 8.0 GiB
> 2021-07-26T07:01:36 - Starting deals failed, with cause: making deal configs: getting miners from minerselector: getting miners from reputation module: not enough miners from reputation module to satisfy the constraints
> 2021-07-26T07:01:36 - Cold-Storage execution failed.
> 2021-07-26T07:01:36 - Job <CID of Previous Failed Job> execution failed: executing cold-storage config: making deal configs: getting miners from minerselector: getting miners from reputation module: not enough miners from reputation module to satisfy the constraints
> 2021-07-26T22:56:03 - Pushing new configuration...
> 2021-07-26T22:56:03 - Configuration saved successfully
> 2021-07-26T22:56:03 - Executing job <CID of Second Previous Failed Job>...
> 2021-07-26T22:56:03 - Automatically staging Cid from the IPFS network...
> 2021-07-26T22:56:07 - Executing Cold-Storage configuration...
> 2021-07-26T22:56:07 - Current replication factor is lower than desired, making 1 new deals...
> 2021-07-26T22:56:07 - Entering deal preprocessing queue...
> 2021-07-26T22:56:07 - Calculating piece size...
> 2021-07-26T22:57:15 - The payload size is 5.1 GiB, and the calculated piece size is 8.0 GiB
> 2021-07-26T22:57:16 - Proposing deal to miner <MinerId> with 0.0000001 FIL per epoch...
> 2021-07-26T22:57:16 - Watching deals unfold...
> 2021-07-26T22:57:16 - Deal with miner <MinerId> changed state to StorageDealReserveClientFunds
> 2021-07-26T22:57:19 - DealID 0 with miner <MinerId> failed and won't be active on-chain: adding market funds failed: GasEstimateMessageGas error: estimating gas used: message execution failed: exit SysErrInsufficientFunds(6), reason: failed to transfer funds (RetCode=6): transfer failed when deducting funds (0.418604 FIL): not enough funds (RetCode=6)
> 2021-07-26T22:57:19 - Cold-Storage execution failed.
> 2021-07-26T22:57:19 - Job <CID of Second Previous Failed Job> execution failed: executing cold-storage config: all started deals failed
> 2021-07-26T23:27:45 - Pushing new configuration...
> 2021-07-26T23:27:45 - Configuration saved successfully
> 2021-07-26T23:27:45 - Executing job <This Suceeded Job's Id>...
> 2021-07-26T23:27:45 - Automatically staging Cid from the IPFS network...
> 2021-07-26T23:27:45 - Executing Cold-Storage configuration...
> 2021-07-26T23:27:45 - Current replication factor is lower than desired, making 1 new deals...
> 2021-07-26T23:27:45 - Entering deal preprocessing queue...
> 2021-07-26T23:27:45 - Calculating piece size...
> 2021-07-26T23:28:53 - The payload size is 5.1 GiB, and the calculated piece size is 8.0 GiB
> 2021-07-26T23:28:54 - Proposing deal to miner <MinerId> with 0.0000001 FIL per epoch...
> 2021-07-26T23:28:54 - Watching deals unfold...
> 2021-07-26T23:28:54 - Deal with miner <MinerId> changed state to StorageDealReserveClientFunds
> 2021-07-26T23:32:30 - Deal with miner <MinerId> changed state to StorageDealClientFunding
> 2021-07-26T23:32:31 - Deal with miner <MinerId> changed state to StorageDealFundsReserved
> 2021-07-26T23:32:31 - Deal with miner <MinerId> changed state to StorageDealTransferring
> 2021-07-27T01:02:08 - Deal with miner <MinerId> changed state to StorageDealCheckForAcceptance
> 2021-07-27T02:29:31 - Deal with miner <MinerId> changed state to StorageDealProposalAccepted
> 2021-07-27T04:22:45 - Executing job <This Suceeded Job's Id>...
> 2021-07-27T04:22:45 - Automatically staging Cid from the IPFS network...
> 2021-07-27T04:22:45 - Executing Cold-Storage configuration...
> 2021-07-27T04:22:45 - Resuming 1 dettached executing deals...
> 2021-07-27T04:22:45 - Watching deals unfold...
> 2021-07-27T04:22:45 - Deal <Number> with miner <MinerId> changed state to StorageDealAwaitingPreCommit
> 2021-07-27T15:35:55 - Executing job <This Suceeded Job's Id>...
> 2021-07-27T15:35:55 - Automatically staging Cid from the IPFS network...
> 2021-07-27T15:35:55 - Executing Cold-Storage configuration...
> 2021-07-27T15:35:55 - Resuming 1 dettached executing deals...
> 2021-07-27T15:35:55 - Watching deals unfold...
> 2021-07-27T15:35:55 - Deal <Number> with miner <MinerId> changed state to StorageDealAwaitingPreCommit
> 2021-07-27T16:33:24 - Deal <Number> with miner <MinerId> is active on-chain
> 2021-07-27T16:33:25 - A total of 1 resumed deals finished successfully
> 2021-07-27T16:33:25 - The current replication factor is equal or higher than desired, avoiding making new deals.
> 2021-07-27T16:33:25 - Cold-Storage configuration ran successfully.
> 2021-07-27T16:33:25 - Executing Hot-Storage configuration...
> 2021-07-27T16:33:25 - Data was unpinned.
> 2021-07-27T16:33:25 - Hot-Storage configuration ran successfully.
> 2021-07-27T16:33:25 - Job <This Suceeded Job's Id> execution finished with status Success.

I still get the following error when I type pow storage-job watch <CID of The Actual Directory> :
                            JOB ID                            |     STATUS     | MINER | PRICE | DEAL STATUS
--------------------------------------------------------------+----------------+-------+-------+--------------
  bafybeianuktguheyxwxowjw6htnwn5xeedtrmlspnml3mc7sjud4aoweuy | awaiting state |       |       |
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xc573da]

goroutine 1 [running]:
github.com/textileio/powergate/v2/cmd/pow/common.WatchJobIds(0xc0004a4f10, 0x1, 0x1)
        /home/warpanomalythree/dev/powergate/cmd/pow/common/common.go:110 +0x41a
github.com/textileio/powergate/v2/cmd/pow/cmd/storagejobs/watch.glob..func2(0x208ee20, 0xc0004a4260, 0x1, 0x1)
        /home/warpanomalythree/dev/powergate/cmd/pow/cmd/storagejobs/watch/watch.go:23 +0x8e
github.com/spf13/cobra.(*Command).execute(0x208ee20, 0xc0004a4240, 0x1, 0x1, 0x208ee20, 0xc0004a4240)
        /home/warpanomalythree/golibs/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0x2086ea0, 0xc000084778, 0xc00036bf78, 0x406365)
        /home/warpanomalythree/golibs/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /home/warpanomalythree/golibs/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
main.main()
        /home/warpanomalythree/dev/powergate/cmd/pow/main.go:11 +0x31

Why am I still getting this error? Shouldn't it change from awaiting state to success or something? Maybe I just need to wait a little longer...
Again so sorry to bother and I really appreciate your help but can you tell me the exact command to backup my wallet? This is actually the first time I've ever used docker so I'm not too familiar with how to get inside the actual docker image. Thanks again for the help, I'm sorry to bother you.

No problem, I'm happy you got your data on-chain... that's an important milestone!
Reg the CLI error, not sure yet bue needs some attention. Looks like is from the CLI side, so shouldn't be something too bad.
Reg your wallet backup, you should do:

docker exec -it testnet_lotus_1 /bin/sh
lotus wallet list
lotus wallet export <wallet-addrs-you-want>
Then you'll see some hexadecimal output which is the private key. That same value you can use with lotus wallet import in other Lotus node.

Okay awesome it looks like my initial issue of not being able to have a deal go through was solved. The thing that solved it was going to a filecoin miner info aggregation site (I used "https://filrep.io/") and adding miners to the TrustedMiner section of the config file used in the apply command to start a Filecoin deal.
Also, as a bonus, the below comment solved the issue of how to back up the private key of you address used with powergate.

No problem, I'm happy you got your data on-chain... that's an important milestone!
Reg the CLI error, not sure yet bue needs some attention. Looks like is from the CLI side, so shouldn't be something too bad.
Reg your wallet backup, you should do:
1. `docker exec -it testnet_lotus_1 /bin/sh`

2. `lotus wallet list`

3. `lotus wallet export <wallet-addrs-you-want>`
   Then you'll see some hexadecimal output which is the private key. That same value you can use with `lotus wallet import` in other Lotus node.


Not work macos ventura asus rog gl503vd

please help sir

Same error on G532LV MacOS Ventura RC2 Version.

Same error  on g731gt in Ventura, it was working fine in Monterey


Onerror & clearItemsOnError props

Hey @kenny-hibino
This is for issue #10 , and adds two props:
onError: callback function to be executed when google.maps.places.PlacesServiceStatus is not OK. It defaults to the current behavior of writing an error message to the console.
clearItemsOnError: boolean - if true, this will clear the autocompleteItems array on error so the autocomplete predictions are not visible. Defaults false, which makes it behave as it does now.
Just let me know if there's anything you'd like me to change with this.

Also let's chat here if there is something not related to specific issue or PR 👍

@kenny-hibino
I've made the changes you requested and updated the branch. Please let me know if there's anything else you'd like me to change.
Thanks!

@cburbank LGTM 👍 I'll merge this in.


Cannot scan QR code

Hello,
I'm trying to scan a QR code to add a new token. Immediately upon touching the "scan QR code" option the image scanner quits. Sometimes it returns to the app, sometimes it quits to the home screen. Please let me know what further details you need for investigation.
Regards,
Mikolaj Menke

Hi,
same issue for me. App crashes on "Scan QR Code". I'm using FreeOTP+ on Android 8.1 if that helps.
Best regards,
Joscha

Same for me. Partial device logs is below.
Best regards
lotth
FATAL EXCEPTION: pool-2-thread-2
Process: org.liberty.android.freeotpplus, PID: 10774
java.lang.IllegalArgumentException: Incorrect image size
at org.fedorahosted.freeotp.util.TokenQRCodeDecoder.parseQRCode(TokenQRCodeDecoder.kt:34)
at org.fedorahosted.freeotp.ui.ScanTokenActivity.analyzeImage(ScanTokenActivity.kt:160)
at org.fedorahosted.freeotp.ui.ScanTokenActivity.access$analyzeImage(ScanTokenActivity.kt:39)
at org.fedorahosted.freeotp.ui.ScanTokenActivity$startCamera$2.analyze(ScanTokenActivity.kt:97)
at androidx.camera.core.ImageAnalysisAbstractAnalyzer.analyzeImage(ImageAnalysisAbstractAnalyzer.java:56)
at androidx.camera.core.ImageAnalysisNonBlockingAnalyzer$1.run(ImageAnalysisNonBlockingAnalyzer.java:138)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
at java.lang.Thread.run(Thread.java:764)

Also having this issue on Android 9.0

+1 on recent LineageOS 16.0 (official).  The original FreeOTP plus QR scanner works fine.

+1 on Android 8.1.0 - just tried to switch from working FreeOTP

I saw the crash on Google Play Store. I am not able to reproduce the issue on Emulator 7.0 to 10.0 and on my own phones. Let me make some changes there so you can test out if the crash still happens.

I pushed a commit without buffer size check. If you have crash issues, could you try this version?
If you cannot compile, you can also install this debug version. It will not overwrite the release version on your device.
https://drive.google.com/file/d/1EB-1SFfONwmttK7AWBj1cm5SzTTy4yy-/view?usp=sharing
Please report back if it work so I can release a fix soon. Thanks!

Just tested it. Debug version still crashes on Android 8.1.

Debug version still crashes on Android 9. Thanks for working on this.

I tried another attempt to fix the buffer issue. I am doing blind fix so I rely on those who can repo to test out. Feel free to try the latest master branch or this APK:
https://drive.google.com/file/d/1gGiCcyh78lnL5QoAONgMp8zhWCNY_Lil/view?usp=sharing

Using the latest APK linked above, the app does not crash now and the camera stays open but a QR code is never scanned. The progress wheel in the corner of the box keeps spinning. (Android 9)
Blind debugging is not fun or easy. Thanks for working on it.

This was a step in the right direction. The app no longer crashes, but it also doesn't detect the QR code. It shows the camera picture, a white frame to position the QR code and a spinning "loading" circle in the bottom right corner of the frame. I used a QR code from here https://freeotp.github.io/qrcode.html for testing, but nothing happened.

Yea. I think the issue is that I assume the buffer size should be just the camera image width * height for the luminous part of YUV format. But some phones is returning the image in a different format that has way more data than width * height for luminous. That will render the scanned QR image unreadable.

Would you install a debug info version and then paste back the information shown on your screen?  Thanks!
https://drive.google.com/file/d/1qXq02J8f6cKQUOnRDw285jIkHAWlmaYF/view?usp=sharing

Image format 35
Image width: 864
Image height: 480
Image planes: 3
Image pixel stride: 1
Image row stride: 896
Image buffer remaining: 430048

Thanks a lot!
Emmmmmm, why image width is 864 while stride is 896?????
864 * 480 is not 430048

Ok, I changed to use stride size instead of width for image width. Let's see if it works this time:
https://drive.google.com/file/d/1RcsmTKsBYoADkae2KIg6f7qQS2-JlY6Y/view?usp=sharing

The latest debug version you just posted fixes it for me (android 9). I was able to scan and successfully add a token to the debug version of the app. Thanks for fixing it!

The latest debug version also works for me on Android 8.1. Thank you!

That's great to know. Thanks a lot for testing and quick response so I can fix it fast.  I will wrap things up and release a new version soon.

I committed the fix in https://github.com/helloworld1/FreeOTPPlus/commit/848e0942bbf69dfbdca85a8e9b7b0f8ebc5ac045. Thank you all for reporting and helping fixing the issue.


[SCB-415] change cluster value from "test-cluster" to "default"

Signed-off-by: zhengyangyong<EMAIL_ADDRESS>Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run mvn clean install to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage increased (+0.02%) to 87.327% when pulling 2cd6b5f7cc49007d271d68fc53c7ca7b4637893f on zhengyangyong:SCB-415 into 8814a2ef936b3be64d319ba3c144adb27cd9e42c on apache:master.


Create test

Please use your own repository to test GitHub features. Things like pull request here send notifications to people that distract from actual work.


'cxxabi.h' file not found

after I updata my xcode15, the pdfium can't build. The error is 'cxxabi.h' file not found.

Hi,
I have updated everything.
You can now use the latest version.
Thanks.

Hi,
Project was updated to latest version.
If your problem was solved, can you close the issue pls?
Thanks.


Use format: uri

Fix #3235

@turt2live, @KitsuneRal: Can you review this?

@KitsuneRal: Which place needs format: uri too? The link points to data/api/client-server/definitions/public_rooms_response.yaml, line 67, which is the guest_can_join boolean property.

@KitsuneRal: Which place needs format: uri too? The link points to data/api/client-server/definitions/public_rooms_response.yaml, line 67, which is the guest_can_join boolean property.

Huh, that's not what it was supposed to point to. The piece in question is in data/api/client-server/content-repo.yaml around line 67 - the content_uri field returned by /upload.


Module 'vue' has no exported Member 'Ref'

Hey Guys,
i ran into following error after using pinia-plugin-persistedstate
ERROR in ../node_modules/pinia-plugin-persistedstate/dist/index.d.ts(2,10):
2:10 Module '"vue"' has no exported member 'Ref'. Did you mean to use 'import Ref from "vue"' instead?
    1 | import { StateTree as StateTree$1, PiniaPluginContext } from 'pinia';
  > 2 | import { Ref } from 'vue';
      |          ^
    3 |
    4 | interface CookieOptions<T> {
    5 |     decode?(value: string): T;
Version: typescript 4.4.4
Time: 4223ms

I'm currently using Vue 2 with the @vue/composition-api plugin. And installed your plugin like this:
pinia.ts
import Vue from 'vue';
import { createPinia, PiniaVuePlugin } from 'pinia';
import { createPersistedState } from 'pinia-plugin-persistedstate';
import { PINIA_STORAGE } from '@/utils/const';

Vue.use(PiniaVuePlugin);

const pinia = createPinia();

pinia.use(
  createPersistedState({
    storage: PINIA_STORAGE,
  })
);

export default pinia;

main.ts
...
// Make sure that composition api and pinia are imported first
import '@/plugins/composition-api';
import pinia from '@/plugins/pinia';
...

new Vue({
  ...
  pinia,
  ...
  render: (h) => h(App),
}).$mount('#app');

implemented in my stores i did it like this:
auth.ts
persist: {
    key: PINIA_PERSIST_KEY,
  },

My Project is working fine, the store is persited and everything seems fine, but i want to remove the error message from my command line.
Is this an error created by me or is it related to something else?

Ah i see what it is, I havent played at all with a Vue2 project with TS. And it comes down to all the types being in one place.
V2 should fix this as the core plugin will not have any dependency at all and not bother you with this.
the Ref import is used to define the Nuxt helper typings. In V2, nuxt specific imports will be in a separate file.
tl;dr: dont bother, problem comes from the plugin, and shouldnt cause you any issue besides the warning. it'll also be fixed in V2

Thanks for the quick answer!
Is there already some timeline when to release v2 or is it possible to push a hotfix for that?
I would really appreciate that so i can integrate it without failing in jenkins

well since v2 API is hopefully 100% iso with current v1 API, I could release it. just needs the more complete docs.
ETA would be smth like this weekend maybe ?

Would be awesome. Thank You!


feat : add part of the buyProduct function code

Fix: https://github.com/casdoor/casdoor-java-sdk/issues/90

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hi Casdoor Team,
I hope this message finds you well.
I have raised an issue with the Casdoor Java SDK version 1.22.1. I noticed that it has been fixed and the new version 1.32.0 was released on October 10th. However, in my Spring Boot application, the pom.xml file is still pointing to the old version (1.22.1).
Please guide how to update the pom.xml file to reflect the latest version (1.32.0) of the Casdoor SDK.
Thank you for taking the time to look into this matter.

Dup with: https://github.com/casdoor/casdoor-java-sdk/issues/90#issuecomment-2408933015


Fix/airpot demo

https://github.com/osrf/romi-dashboard/issues/115
Removed the auto zoom in feature that I was doing earlier since setting the minZoom  properly automatically brings the map to a fitting size.
setInitialBounds is required to be placed after setting min zoom values otherwise map won't automatically be resized.
To do
Find a better way to determine value for min zoom.
Find correct event type instead of setting it to (any).
Currently, the width and respective default zoom for both maps are:

office: 25.xxxx -> 5
airpot: 282.xxxx -> 2.5

Given that the magnification is x2 for each level up, I think we can deduce that minzoom for 50 ~ 4, min zoom for 100 ~3 min zoom for 200-400 ~ 2 and so on and we can come up with a formula to calculate from here.
Will there be a situation where the height is > than the width?

It seems like there isn't really a need for a major refactor. The bounds for each level have already been calculated and stored in mapFloorLayers in the initial render. Subsequent changes to bounds can be handled in handleBaseLayerChange, through the event object. The event provides the name and we can use it to access the correct bounds from there.
The minZoom value is dynamically calculated from the width at the moment in determineMinZoom, not taking height into account. It seems to be giving pretty reasonable values after testing it with a range of width values from 10 - 1000.
Right now, the algorithm is dependent on a baseWidth and baseZoom as constants. I remember there is a discussion on giving a vendors a range of size to upload their assets? (icons, maps etc ...) so we might need to change it in future if the range veers too far from what I have tested.

It seems like there isn't really a need for a major refactor. The bounds for each level have already been calculated and stored in mapFloorLayers in the initial render. Subsequent changes to bounds can be handled in handleBaseLayerChange, through the event object. The event provides the name and we can use it to access the correct bounds from there.
The minZoom value is dynamically calculated from the width at the moment in determineMinZoom, not taking height into account. It seems to be giving pretty reasonable values after testing it with a range of width values from 10 - 1000.
Right now, the algorithm is dependent on a baseWidth and baseZoom as constants. I remember there is a discussion on giving a vendors a range of size to upload their assets? (icons, maps etc ...) so we might need to change it in future if the range veers too far from what I have tested.

Thanks for this @ChawinTan  looks like a good solution for now. I tested a bit as well and it seems to be working as we want.

Will there be a situation where the map (or floor) height is > than the width?

It'll be good to know that. Let's ask @mxgrey

Will there be a situation where the map (or floor) height is > than the width?

This is definitely something that we should be able to support in the long-term, because we don't want to put a height vs width constraint on how floor layouts are defined. If it adds considerable effort, then we can open a ticket to deal with it later, but if it can be fixed now, then that would be preferable.

I just added the calculations for the situation when height > width. I have done some testing with my own maps as well and they seem to be working fine too, at least when the monitor is displayed horizontally.
Nonetheless, I will still open up an issue for us to keep a lookout for a situation where the minzoom is not calculated properly should such a situation happen.

Can you test this branch with the clinic world too?
run ros2 launch demos clinic.launch.xml instead of the office demo

Clinic world for all levels seem to be working fine. It should look better if my monitor is oriented vertically.

Came up with my first implementation on the window resize feature. Tested it in my environment and it seems to be working as I expect it to be and would be nice is other people can test it out in their environment as well.
Tried using this as it seems to be the most straightforward way but it does not work for manual resizing of the window. You would need to dynamically resize the map for it to work.
Removed the calculations to determine the minzoom and set the default to 0.

Tested it in my environment and it seems to be working as I expect it to be and would be nice is other people can test it out in their environment as well.

Hi @ChawinTan it's working correctly for me too!

Maybe I'm missing something here, if we are using min zoom of 0, why do we still need to change the zoom on resize? Handling resize events should only be needed when we are calculating the min zoom ourselves, because the min zoom will depends on the viewport size, so the min zoom we calculate initially may not match after the user resize the viewport. If we are using a fixed min zoom of 0 then it shouldn't matter if the user do any resizing.

I see where you are coming from now. I initially thought it was intended for it to be a feature such that when the user resize the browser, map will fit accordingly.

Anyway, here is the explanation to what the formulas were doing:
Every step up/down in our map increases/decreases the map size by a factor of 0.5, so this means the map will increase by (square root of 2) or decrease by (1/square root 2) every time the user toggles the button or move the scroll on his mouse.
To deal with size changes, I needed to compare if the height or width of the viewport is smaller than or greater than the image displayed.
Determining when to zoom out is quite straight forward. If the documentWidth for documentHeight is smaller than the image width or height, then set the zoom -0.5 smaller.
When trying to determine when to zoom out, I set it to be at the point where the documentHeight or documentWidth is bigger then the existing image width or height by a factor of (square root of 2) so that the image fills the viewport nicely when expanded.
We need to check with the documentWith and documentHeight to make sure that the conditions don't conflict with each other  when checking documentWidth against imageWidth and documentHeight against imageHeight.


cruft: update template

https://github.com/iterative/py-template/pull/115

Codecov Report
Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (1161685) 63.70% compared to head (7c92f31) 63.81%.
Report is 1 commits behind head on main.

Files
Patch %
Lines

src/dvc_data/hashfile/db/local.py
0.00%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #460      +/-   ##
==========================================
+ Coverage   63.70%   63.81%   +0.10%
==========================================
  Files          61       61
  Lines        4125     4118       -7
  Branches      710      706       -4
==========================================
  Hits         2628     2628
+ Misses       1349     1344       -5
+ Partials      148      146       -2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Added a boolean return status for when setting up DHCP

Fairly minimal change - should be inconsequential for existing users working on this library but for future users they'll be able to check if DHCP was set up, and then respond to that thereafter (eg: try DHCP again, or try manual IP assignment).

Thanks!


Network Architecture

Hi,
I was wondering why did you use a VAE to get the embedding instead of just the output of a middle layer from a CNN?

Because VAE would be trained with a reconstruction loss which not only ensures input information are preserved but also creates a rich low dimensional latent space on which the discriminator would be trained better.


Support "Sign In with Apple" auth

This is Apple's new identity provider for users to sign into "apps and websites". We'll need to support it just like we support Google and GitHub auth, on all platforms.
The lever by which Apple is making this required for us is their control of iOS: after a deadline of 2020-04-30, we must support it in order to publish updated versions of Zulip in the App Store. But it applies to all platforms, because in order for a user to use Sign In with Apple for Zulip on their iPhone, and also use Zulip on the web and desktop, we'll need to offer Sign In with Apple on the web and desktop.
Some helpful Apple docs:

https://developer.apple.com/sign-in-with-apple/
https://developer.apple.com/sign-in-with-apple/get-started/
https://developer.apple.com/design/human-interface-guidelines/sign-in-with-apple/
Specifically for the Web:

https://developer.apple.com/documentation/signinwithapplejs
https://developer.apple.com/documentation/signinwithapplejs/configuring_your_webpage_for_sign_in_with_apple

https://developer.apple.com/documentation/signinwithapplejs/incorporating_sign_in_with_apple_into_other_platforms
On setting up with Apple - making App ID, getting key, setting web domains:

https://help.apple.com/developer-account/#/devde676e696
https://help.apple.com/developer-account/#/dev77c875b7e
https://help.apple.com/developer-account/#/dev1c0e25352

Some helpful outside study:

Walkthrough, web, some tricky bits -- the author isn't an expert in this area, but that works out well for the exposition: https://blog.curtisherbert.com/so-theyve-signed-in-with-apple-now-what/
Walkthrough, step by step: https://developer.okta.com/blog/2019/06/04/what-the-heck-is-sign-in-with-apple
Tech-press story; mentions that it's actually OpenID Connect underneath: https://www.wired.com/story/sign-in-with-apple-mixed-reactions/

There's an open PR for python-social-auth to support this, at python-social-auth/social-core#409. Reading the thread, it sounds like it's not quite fully baked yet, and it's not clear anyone is pushing hard on it right now. It may be a good idea for us to try the PR ourselves before it's merged, and participate by helping out where we can.

@zulipbot claim

That upstream PR python-social-auth/social-core#409 is now merged! Part of a big batch of changes the maintainer merged over the past couple of days. So hopefully that will go into a release on PyPI soon; if that all works, it'll simplify things for us quite a bit.

@gnprice Do you think this is complete now?

@gnprice Do you think this is complete now?

Don't worry, we'll close the issue when it's done :slightly_smiling_face: Take a look at the linked PR #14235 if you'd like to follow along.

@gnprice I'm so sorry!  I must have followed a link from social-core over to this without realizing it, I thought I was looking at and asking about an open python-social-auth issue 🤦♂

@dcmoore-gd Ah, no worries :slightly_smiling_face: -- those back-reference links GitHub puts into issue threads don't mention the repo or org, and I've certainly often wished they would myself.

Can we close this?

Yes, closed, thanks for the reminder (and all the work on this feature) @chdinesh1089!


DM-289 add publishedAt field

Added a field for publishedAt.  Will contain the first publication date for an opportunity.

Hey @sutherlanda , could you resolve the conflicts when you get a chance? I think the last PR merge caused them.


Cut v2.33-rc.0

:rocket:

https://github.com/prometheus/prometheus/pull/10152 was last minute but added the limit field to the rules API.

#10152 was last minute but added the limit field to the rules API.

That's already in the changelog. Did you miss it, or did I misdescribe the change?

Ah, I forget we called the API "Web", but it is slightly misdescribed.

Yeah, having the API under "web" is a bit weird, but we have done so before, and it's the name of the directory this is in…
Thanks for your suggestion. I'll update the PR momentarily.

All comments addressed. Please have another look.

/prombench v2.32.1

/prombench cancel
I couldn't spot any substantial differences.

Yeah, benchmarks looked good to me when I checked yesterday.


After Pull: "Insert Into" and "error" lines displays after submitting a reservation DSA User and new dsa user acc, Error codes are consistent...

After Pull: "Insert Into" and "error" lines displays after submitting a reservation DSA User and new dsa user acc, Error codes are consistent...
The reservations are still saved....   so no problem on that... just the error lines that shows after submitting...

the reservations are still saved tho...

reason of reopened... misclicked - close with comment.

fixed


Cannot run Gradio Interface

I tried to run Gradio Interface as instructed on the ReadMe
git lfs clone https://huggingface.co/JunhaoZhuang/PowerPaint_v2/ ./checkpoints/ppt-v2
python app.py --share --version ppt-v2 --checkpoint_dir checkpoints/ppt-v2
usage: app.py [-h] --pretrained_model_path PRETRAINED_MODEL_PATH
[--base_model_path BASE_MODEL_PATH]
[--weight_dtype WEIGHT_DTYPE] [--share] [--local_files_only]
[--port PORT]
app.py: error: the following arguments are required: --pretrained_model_path
may I know what is the updated pretrained_model_path? I used ppt-v2, but it said config.json does not exist

git checkout main


Using volta-cli/action on GitHub Enterprise Server causes frequent rate-limits with no fallback

We are using this action on GitHub Enterprise Server and we're seeing frequent failures trying to get the latest volta-cli version, because we're not authenticated against GitHub.com.
For now we have pinned the volta-version to 1.0.8, but we would like to remove that pinning again, so that we will get the latest volta-cli version in the future.
I saw that the actions/setup-node action has a fallback to get the version directly from nodejs.org in case of rate-limits: https://github.com/actions/setup-node/blob/9f3a02bbd17adc05b078145bb8f676c84be77e89/src/installer.ts#L109-L129
Maybe something like that would make sense to implement in this action also.
I actually have two ideas that could solve this:

check if the request got rate-limited and fall back to https://volta.sh/latest-version
check if the request got rate-limited and do a HEAD request to https://github.com/volta-cli/volta/releases/latest which will redirect to the latest version

If there's interest and a suggestion on which route to take, I'd be willing to create a PR for it.
Ref: #77, #101, #97

Eeck, sorry about this. The changes in https://github.com/volta-cli/action/pull/101 were about moving away from using https://volta.sh/latest-version due to issues with the same sort of rate limiting failures that you are seeing now after #101.

check if the request got rate-limited and fall back to https://volta.sh/latest-version

I'd gladly accept a PR that adds this, if you have the time.


Prepare v2.1.0 release

Prepares for release
Why is this the best possible solution? Were any other approaches considered?
I went back and forth on whether this should be 3.0 or 2.1 because there is a change in required Python versions and a small breaking API change. I went with 2.1 because we're dropping support for Python versions that are very close to end of life and we have a requires-python directive. Other people have gone back and forth on this and come to various conclusions (e.g. here). There is technically a small API change but it's unlikely to have been in use and is easy to respond to.
I included dependency updates here because I think they're low risk and nice to keep on top of. I also ran ruff and made changes accordingly.
There are deprecation warnings about datetime.utcnow() that are annoying when running tests. I tried to quickly address these but got some errors in the validator updater when I did the switch there. I think we should come back to it but I figured I'd leave the easy updates in here.
What are the regression risks?
Feels low-risk. Probably the highest risk is around the Validate update since that's jumping a lot of JR versions. We also haven't run with the new dependency versions for very long so that could be risky.
Does this change require updates to documentation? If so, please file an issue here and include the link below.
https://github.com/getodk/docs/issues/1813
https://github.com/getodk/docs/issues/1795
Before submitting this PR, please make sure you have:

[x] included test cases for core behavior and edge cases in tests
[x] run python -m unittest and verified all tests pass
[x] run ruff format pyxform tests and ruff check pyxform tests to lint code
[x] verified that any code or assets from external sources are properly credited in comments

I am stuck trying to figure out why linting is failing! It passes for me locally in Python 3.12 and 3.10. I've verified that I'm using ruff 0.6.4 and that the task does a pip install.
@lindsay-stevens I'd be grateful for any ideas you may have! Please push to this branch if you want.

The lint problem was that ruff didn't like the space added between the two import lines in test_android_package_name.py.
For future reference, although the latest openpyxl is 3.1.5, this update is to 3.1.3 because since 3.1.4, openpyxl pinned lxml to 5.1.0. Their problem in 5.1.0 (here) is marked as fixed in 5.1.1, but the pin isn't removed until openpyxl 3.2 which is not released yet. Also pyxform is already using 5.2.2 and this update moves to 5.3.0.

For anyone wondering why ODK_Validate.jar more than doubled in size (from ~5MB to ~13MB), most of it (~5MB) seems to be because the underlying javarosa library recently added a dependency on bouncycastle.

The lint problem was that ruff didn't like the space added between the two import lines in test_android_package_name.py.

Wild, when I run ruff locally on master now, it fails and wants to put the space back. I wonder whether I might have some kind of global config interacting with it somehow. I don't use ruff anywhere other than here and for pyodk.

why ODK_Validate.jar more than doubled in size (from ~5MB to ~13MB)

Oh dear, I didn't think to check on size! That is highly unexpected, I will look into it.

maybe it's possible to make some javarosa dependencies optional, so validate can build without them?

Yes, that should be possible.

Maybe an alias/history thing - pyodk is ruff==0.3.4 and pyxform is ruff==0.6.4. I've had that happen before - a bunch of unexpected ruff changes but it was because I was running the ruff version from the other project.

I'll try reinstalling all dependencies if I run into it again and let you know if I need more troubleshooting help.
I have reduced Validate size down to 3.9 MB. I'll put in a PR to update that before we release for real.


Take shot results in: Aborted (core dumped)

I run AstraLite on my ubuntu 22.04 (x86_64) and connect to my remote INDI server.
After connection is established I try to take a shot but AstraLite immediately crashes with: Aborted (core dumped)

@petmos
Thanks for log! I will investigate. There is something wrong in frame noise level calculation

@petmos
I think I fixed this bug. Do you build AstraLite from sources manually or install from deb?

@art-den I build it from sources. But I have no experiences with rust/cargo.
How can I rebuild it?
git pull && cargo build --release
or something else?

@petmos yes,
git pull
cargo build --release

is enough to build the latest version

Looks good now.
Great job. Thank you very much.


refactor(labware-library): Scaffold out responsive nav

overview
This PR adds the skeleton of the Labware Library page, with breakpoints and such for the sidebar, top nav, and page. No functionality yet, but there's some dummy hardcoded data to see how stuff interacts.
Closes #3073. See ticket for acceptance criteria
changelog

refactor(labware-library): Scaffold out responsive nav

review requests
http://sandbox.labware.opentrons.com/lablib-page-ui
If possible, test across a variety of browsers and devices.

[ ] Scroll bars show up as expected when height is crunched (all sizes)
[ ] Small (< 768) screen behaves according to ticket
[ ] Medium (< 1280) screen behaves according to ticket
[ ] Large (>= 1280) screen behaves according to ticket

Sorry about the number of lines in the PR. I promise that most are snapshot tests and the associated snapshots, so it's not as bad as it seems.

Received 👍 from @howisthisnamenottakenyet via Slack, so merging


Update mergemap to reduce memory allocations

Overview
The idea is to allocate less memory when the map is already a map[string]interface{} / Values type.
What this PR does / why we need it
I decided to move the code to our repo, because

This is a single file with two functions, the size is not big enough
Code changes were required
To effectively check type aliases such as Values, the type should be imported and be used by the function (causing cyclo dependencies)

Special notes for your reviewer
goos: darwin
goarch: amd64
pkg: github.com/peterbourgon/mergemap
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
                        │       old.txt       │                new.txt                │
                        │       sec/op        │    sec/op     vs base                 │
Mapify/Map_interface-12   45830274.000n ±  9%   2.558n ±  1%  -100.00% (p=0.000 n=20)
Mapify/Map_string-12             51.12m ± 10%   40.15m ± 16%   -21.46% (p=0.000 n=20)

                        │   old.txt    │                  new.txt                  │
                        │     B/op     │     B/op      vs base                     │
Mapify/Map_interface-12   15.49Mi ± 0%    0.00Mi ± 0%  -100.00% (p=0.000 n=20)
Mapify/Map_string-12      15.48Mi ± 0%   10.31Mi ± 0%   -33.39% (p=0.000 n=20)

                        │   old.txt   │                 new.txt                  │
                        │  allocs/op  │  allocs/op   vs base                     │
Mapify/Map_interface-12   203.9k ± 0%     0.0k ± 0%  -100.00% (p=0.000 n=20)
Mapify/Map_string-12      203.9k ± 0%   201.7k ± 0%    -1.11% (p=0.000 n=20)

The only concern is that without copying the map the function can mutate some values. For now, I do not see any problems in a living cluster.


Problems to pull remote modules from git

Hi,
After an update to the 0.16.X version I've faced a weird issue related with the pull of the remote modules stored in our gitlab account.
This is my terraform/terragrunt configuration in the live repository in the terraform.tfvars file:
terragrunt = {
  include {
    path = "${find_in_parent_folders()}"
  }

  terraform {
    source = "git::ssh://git@gitlab.com/homeWiFi-devops/iac_homewifi.git?ref=0.10.0"

    extra_arguments "custom_vars" {
      commands = [
        "plan",
        "apply",
        "import",
        "push",
        "refresh"
      ]

      arguments = [
        "-var-file=${get_tfvars_dir()}/../common.tfvars",
        "-var-file=terraform.tfvars"
      ]
    }
  }
}

which is, I think, a quite standard configuration if you're working with the Gruntworks approach, now let's see the result of a get command:
javier.juarez@JJMBP $ ~ terragrunt get
[terragrunt] [/Users/javier.juarez/workspace/fon/devops/infra/iac_live/fon/eu-west-1/dev] 2018/07/11 18:09:24 Running command: terraform --version
[terragrunt] 2018/07/11 18:09:25 Reading Terragrunt config file at /Users/javier.juarez/workspace/fon/devops/infra/iac_live/fon/eu-west-1/dev/terraform.tfvars
[terragrunt] 2018/07/11 18:09:25 WARNING: no double-slash (//) found in source URL /homeWiFi-devops/iac_homewifi.git. Will insert one, but note that relative paths in downloaded Terraform code may not work.
[terragrunt] 2018/07/11 18:09:25 Cleaning up existing *.tf files in /Users/javier.juarez/workspace/fon/devops/infra/iac_live/fon/eu-west-1/dev/.terragrunt-cache/n0QBbScnymYjbyjTmgJlMNuQnlI/sqLEpOsQ21ucEHEbM3srwHrEJ1Q
[terragrunt] 2018/07/11 18:09:25 Downloading Terraform configurations from git::ssh://git@gitlab.com/homeWiFi-devops?ref=0.10.0 into /Users/javier.juarez/workspace/fon/devops/infra/iac_live/fon/eu-west-1/dev/.terragrunt-cache/n0QBbScnymYjbyjTmgJlMNuQnlI/sqLEpOsQ21ucEHEbM3srwHrEJ1Q using terraform init
[terragrunt] [/Users/javier.juarez/workspace/fon/devops/infra/iac_live/fon/eu-west-1/dev] 2018/07/11 18:09:25 Initializing remote state for the s3 backend
[terragrunt] [/Users/javier.juarez/workspace/fon/devops/infra/iac_live/fon/eu-west-1/dev] 2018/07/11 18:09:25 Running command: terraform init -backend-config=dynamodb_table=fon-homewifi-terraform-locks -backend-config=bucket=fon-homewifi-terraform-state -backend-config=key=fon/eu-west-1/dev/terraform.tfstate -backend-config=region=eu-west-1 -backend-config=encrypt=true -lock-timeout=20m -get=false -get-plugins=false -backend=false -from-module=git::ssh://git@gitlab.com/homeWiFi-devops?ref=0.10.0 /Users/javier.juarez/workspace/fon/devops/infra/iac_live/fon/eu-west-1/dev/.terragrunt-cache/n0QBbScnymYjbyjTmgJlMNuQnlI/sqLEpOsQ21ucEHEbM3srwHrEJ1Q
Copying configuration from "git::ssh://git@gitlab.com/homeWiFi-devops?ref=0.10.0"...
Error copying source module: error downloading 'ssh://git@gitlab.com/homeWiFi-devops?ref=0.10.0': /usr/local/bin/git exited with 128: Cloning into '/var/folders/sn/8x31z_g57xg2b1twtdxyqyk00000gp/T/tf746723676/module'...
GitLab: The project you were looking for could not be found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[terragrunt] 2018/07/11 18:09:27 Hit multiple errors:
exit status 1
```

in short the 0.16.X versions are cutting the git URL instead of using the original one:
git@gitlab.com/homeWiFi-devops/iac_homewifi.git?ref=0.10.0

they're trying to use this other:
git@gitlab.com/homeWiFi-devops?ref=0.10.0

I could reproduce the issue with all the 0.16 version series, but I could not reproduce it with the 0.15.3 version.

Any help about this will be welcome, because I'm a bit surprised to catch this issue, I know an a f* early adopter it's my fault :)

Try adding a double /.  Change this line:
source = "git::ssh://git@gitlab.com/homeWiFi-devops/iac_homewifi.git?ref=0.10.0"
to
source = "git::ssh://git@gitlab.com/homeWiFi-devops//iac_homewifi.git?ref=0.10.0"

Nope, I got the same results using the 0.16.2 version

Please try https://github.com/gruntwork-io/terragrunt/releases/tag/v0.16.3 (new binaries should show up in a few min) and let me know if it fixes this issue for you.

Yes, it's working again, thank you very much

Great, thanks for following up

What's the solution if your module doesn't have submodules?
https://github.com/gruntwork-io/terragrunt/issues/1675


Poorly Performing Passes

Both Lower To Ground and Expand Whens have some performance issues.
My code generates two input Vecs of length N and 1 expected output Vec of length N, the runtime I get is:

N
Lower to Ground (ms)
Expand Whens (ms)

10
9
55

100
527
4496

1000
49478
425760

Looks like the algorithms are O(n^2)

See #113


[MRG] Improved LFP calculation

This PR builds on and supersedes #150 (commits by @ntolley and @jasmainak retained in squash)

[x] account for all transmembrane currents (not just that through the mid point sec(0.5); see this gist for details)
[x] validate units and shift conductance units to S/m (default: 0.3)
[x] produces expected pattern of near-field current sinks close to active synapses (see second figure in this gist)
[x] produces far-field dipolar field patterns (see gist)
[x] fix PSA and LSA implementations
[x] illustrate laminar LFP ~and CSD~ calculation in (weak) gamma example
[x] test that net transmembrane current is (very close to) zero (implemented by setting all transfer resistances to unity)
[x] test that far-field LFPs is dipolar
[x] test that transfer resistances are correct
[x] consolidate NEURON and LFP electrode coordinates (NB y-coordinate)
[x] create a suitable example (what do we want to demonstrate?)
[x] update whats_new.rst & api.rst

~create demo using single instance of Cell~ (move to #323)
~add tests for validity of PSA and LSA values close to cells (how?)~ not needed as long as the above three tests pass
~optimise for speed~ future PR should improve implementation, possibly getting rid of extra_gather_scatter

Closes #68

The implementation in c75cf23 is "correct" in the sense that it accounts for all transmembrane currents (not just that at sec(0.5). However, it doesn't actually implement 'psa', but rather calculates the LFP based on the current at each segment. For sections with 13 segments this is a x13 increase in computational load.
PSA
All the transmembrane currents should be summed, and implicitly assigned to sec(0.5). The 'transfer resistance' for an electrode is based on the distance between its tip and sec(0.5). One electrode, one distance to section, one transmembrane current (point).
LSA
All the transmembrane currents should be summed, and implicitly assigned to a line running from sec(0.) to sec(1.).  The 'transfer resistance' for an electrode is based on the radial and transaxial distances between its tip and the section endpoints. But again: one electrode, one transfer resistance, one transmembrane current (line).
My next task is to implement these properly, and to check them against the current implementation. I've realised that some of our code is probably inspired by LFPsimpy. I can see there are some good ideas there I can hijack :)

c9b0cae fixes LSA: it produces values very close to PSA at a "safe" distance from the closest cell.

The temporary npy-array is used to test PSA against c75cf23: using MPIBackend works!
LSA and PSA now produce almost identical (tol=1e-1 uV) values for the electrode further from the network (close by the differences are much larger, though the shapes are similar)

See comment in lfp.py for LSA reference and asciiart of the geometry.

Ugh, Travis isn't happy :( I'll need to debug. Strange that I don't get these errors when testing locally...

Definitely interested where that segmentation fault is being triggered, but in general I definitely love the new commits!
Once this PR is merged, I think I will update the isoma (and potentially vsoma) recording setup. Since they setup somatic recordings for every cell, I'm wondering if there's much of an overhead difference if it just defaults to recording everything from the nrn side, and then dropping everything except the somatic recordings.

The test failures and segfaults seem to be provoked by multiple trials... I ran pytest test_lfp.py repeatedly as I was making changes, but didn't detect anything had broken. Running just that test alone actually still works, but fails if I run pytest on the entire tests-folder.
What's the difference? The problem surely is my imem_handler, but I haven't been able to track down where it messes things up.

I can't figure this out... Both joblib and MPI backends lead to segfaults when n_trials > 1. I guess I don't understand how _PC and _CVODE, both defined as globals in network_builder, interact with the pointers in the 'handler' and LFP electrodes. I've spent far too much time fiddling around without actually knowing what I'm doing.
Do we have an expert we can call for help?
The alternative is to revert back to the version (55ed0da) where each _LFPElectrode initialised a pointer vector to all segments, and gathered these values in its callback. The huge overhead in redundant gather-operations is what makes the code slow. It's frustrating that I can't get this (much faster and semantically more reasonable) implementation to work :(
WDYT? Shall I revert? The LSA fixes in this PR should go in, regardless.

For segmentation faults, print statements are your friend. I would start with Joblib backend and n_jobs=1 to figure out where the segmentation fault happens. My suspicion is that some Neuron object that needs to be cleared from memory has not been. Maybe you need to update this:
https://github.com/jonescompneurolab/hnn-core/blob/32d33cc96bd788f6c96b97339a98788b2cf3810e/hnn_core/network_builder.py#L530
we could ask Blake for help but I suspect he's too busy and/or would go through the same steps of debugging via print statements. I can also take a look after Wednesday.

I decided to revert. It's not possible to debug using print statements, the segfaults occur randomly.
No-one else is using the extra_scatter_gather method (BMTK, LFPy, Netpyne). Rather, they all gave each cell keep track of its own membrane currents. I think that might be the way for us as well, but I propose we get this PR in and iterate later.
Note that in  906e622 I've removed the recording of the time vector in each electrode. The current implementation leaves the LFP data one sample short. My "fix' calls the potential calculation callback once after the simulation has ended, but the value returned is identical to  the second-to-last (so this is a hack).
BMTK and Netpyne use (very similar implementations of) PtrVector().ptr_update_callback. NRN documentation is very vague about this method, which seems to be related to CVode().cache_efficient(). I tried using this instead of extra_scatter_gather in 79125b5, but no dice :(

that's weird that the segfault happens randomly since there should be no random component to the simulation if you use n_jobs=1. But let's iterate in a future PR. I propose we update an example and try to bring this PR to MRG state.

here it says it works only with single thread, so I'm assuming you can't use MPI-based parallelism with this

here it says it works only with single thread, so I'm assuming you can't use MPI-based parallelism with this

Well that's what I thought. But it worked with MPIBackend just fine! For one trial. But as soon as you tried to run several trials, segfault.
It has got to do with some CVode or ParallelContext related residuals. When NetworkBuilder is imported in parallel_backends:_clone_and_simulate, it's only imported once (Python). So whether JoblibBackend or MPIBackend, the globals _CVODE and _PC must remain in memory and screw up the pointers on trials > 1.
It's also why you get weird results (and eventually a segfault) if using %run several times in an ipython session. Something's not getting cleared properly, and I'll be damned if I can figure out what it is!

@cjayb can you rebase so CircleCI is triggered and we can see the new documentation?

Could use some reviews here. ddf4a23 introduces the LFPArray class and a new API for LFP electrodes. I currently have both that and LFPElectrode implemented, but I propose to deprecate the latter. WDYT?
One advantage of the 'array'-approach is that we don't redundantly re-gather the membrane current vectors for each electrode. The times have also been reintroduced to make plotting etc. easier.
I you agree, I'll next remove the old LFPElectrode-API and clean up the code (it's currently bloated due to the two parallel implementations!). Note that there's a test making their output remains the same, though.

I think for sure LFPArray should be the way to go.  Or just a single array? I think it's better to keep it simple and start with single array

This is beginning to look pretty good. The speed increased dramatically with LFPArray. I'd like to get input on the API with multiple trials.
Rather than this:
net.lfp_array['grid1'].plot(trial_idx, show_electrodes, ...)

I'd like to be able to do this:
net.lfp_array['grid1'][trial_idx].plot(show_electrodes, ...)
# OR EVEN
net.lfp_array['grid1'][trial_idx][show_electrodes].plot(...)

Any ideas? net.lfp_array is a dict, but perhaps it should be a dict-like object? The tricky thing is to implement the indexing/slicing logic at the top-level, rather than all the way down in .voltages. Note that currently net.lfp_array['grid1'].voltages is n_trials x n_contacts x n_times.

Any ideas? net.lfp_array is a dict, but perhaps it should be a dict-like object?

humm ... I need to look at the code more carefully but usually this is done using the __getitem__ object. Just a general comment though, I think it's better to stick to the slightly "less beautiful" option if it's less complex. I tend to think that there are upper bounds on total complexity a software project can handle. If you add complexity somewhere, then it needs to be reduced somewhere else for things to be sustainable.

I agree with making LFPArray a list of trials. This may be simpler if you go with only allowing one array per @jasmainak's suggestion.
Also net.vsoma and net.isoma both match that structure with the first index being the trial_idx.

I agree with making LFPArray a list of trials. This may be simpler if you go with only allowing one array per @jasmainak's suggestion.

Are we gravitating towards this sort of API (:
net.lfp = LFPArray(...)  # in Network.__init__
...
# only one array, add as many electrodes as you like to it
net.add_electrode(arr1_pos, sigma, method)  # where arr1_pos can be list
net.add_electrode(arr2_pos, sigma, method)  # arr2 would just be appended

After simulation the net would contain:
net.lfp[trial_idx].voltages  # n_channels x n_times, where n_channels == len(positions)
net.lfp.positions
net.lfp.times
net.lfp[trial_idx].plot(ch_nos, tmin=..., ...)

I like this. I think it's important for net.lfp[trial_idx] to work, as it does for dpl[trial_idx]. I will insist on net.lfp.times, though, just as we have net.cell_response.times (but `dpl[trial_idx].times, which is redundant)

Also Dipole object has different slicing logics which might get confusing for a beginner. Not an easy problem to solve I guess. MNE also has different slicing logic for raw, epoch, and evoked objects

I think this is really important: I find it hard to keep track of myself! I really dislike the fact that dpl is a plain list! Code introspection (jedi et al.) has a hard time popping up guides, so I often end up doing d = dpl[0] then d.<TAB> to get the methods. I think Dipole and LFPArray should both have __getitem__-implementations of the trial index.

Code introspection (jedi et al.) has a hard time popping up guides, so I often end up doing d = dpl[0] then d. to get the methods. I think Dipole and LFPArray should both have getitem-implementations of the trial index.

Yeah, I did this in the past too but it looks like it works in the latest IPython!

I'm not opposed to this in principal but we just need to be conscious that it's done consistently across different objects. We might also benefit from adding an average method (across trials) if this is done ...

API looks good. If you wanted to get all the trials as a numpy array, would there be a get_data method? Or would you expect users to do a list comprehension?

net.cell_response is a bit of an odd object when thinking about enforcing this API. At the moment net.cell_response.vsoma, net.cell_response.isoma etc. are separate lists for each trial. I'm not sure if it makes more sense to enforce the rule for attributes of cell_response or the object itself, because there are so many distinct element inside the object. Whereas Dipole and LFPArray have only dipoles and lfps.
Awhile back I implemented a __get_item__ method that returns a subset of the cell_response object filtered by gids through net.cell_response[gid_list]. However, that may not fit the zeitgeist of the proposed system.

For the coordinates, could we swap the y and z coordinates internally for this PR so that we can keep things moving?

For the coordinates, could we swap the y and z coordinates internally for this PR so that we can keep things moving?

Yeah we could definitely do that.

Yeah, I did this in the past too but it looks like it works in the latest IPython!

That's neat :) I might, however, say that dpls.plot_psd(agg_method='Welch', ...), where agg_method defines some form of aggregation over trials, would be a more realistic use case than plotting the PSD of individual (short) trials.
I don't have a strong opinion yet, but I'll try to implement trial-level indexing to net.lfp, and see where that takes us...

See f7f4133 where I've tried to demonstrate what a subclass (or perhaps a Mixin?) -based solution might look like. It's very brittle (no input checking yet), so don't bother reviewing in detail yet. It would be good to hear your thoughts on whether this is a good direction:

I've decided to keep the dict in net.lfp: a very realistic use case is having multiple linear shanks, where we'd likely want to keep coordinates separate for different shanks.
I like the syntax net.lfp['shank1'][trial_no].plot() (see example)
I've hidden the voltages in _data, and implemented a get_data()-method, as in MNE

BTW I haven't forgotten your earlier review comments! I just want to get the API sorted before spending more time on this.

@cjayb I would suggest coding the logic in __getitem__ for now without introducing another class. Let us do that in a separate PR so we can see clearly how it simplifies the code.

The way Mixin would work is that many different classes will inherit from the class. E.g.:
class PIckMixin:

    def __getitem__(self):
          # fancy code for slicing and dicing

class CellResponse(PickMixin):

class LFPArray(PickMixin):

class Dipole(PickMixin):

I would suggest coding the logic in __getitem__ for now without introducing another class. Let us do that in a separate PR so we can see clearly how it simplifies the code.

What would you expect to happen if you used __getitem__? That is, what is the output of net.lfp['shank1'][0]?

it should return an instance fo LFPArray ... and then the code in plot might need a bit of extra logic as well

Something like this? e1a3336 Not a huge fan of plot as it is, but it's a start. Should I start adding input validation and tests? Any functionality missing?

Yep I like e1a3336 !

yes, please. Let's add the remaining tests and wrap this up. The issue of the placement of cells still remains though ...

@cjayb for the test data, we have a separate branch. It might be good to use that so we don't bloat the version history. You can manually remove the commit that added the data using git rebase -i

I see! Good you mentioned this. I was just planning to remove the current ad hoc numpy array in a future commit, but will now rebase it out as you suggest. I'm still uncertain about the right test to write to check the output nearby. The far-field dipole test is pretty solid, I think. But for a 3x3 network with spikes...not sure what the right way to go is there.

I was a little flustered to see the recent test failures, until I realised that I'd rebased :) cde4ed29 changed the section points, which are used in extracellular calculations. Nice to see it all makes sense :) So the gold standard test code is now coming out—we'll have to come up with something better!

But for a 3x3 network with spikes...not sure what the right way to go is there.

Not sure if this is the direction you were thinking of going for an appropriate LFP test, but it should be possible to spike sort from the LFP, and compare it to the actual spike times in net.cell_response. I implemented a simple version this test for net.cell_response.vsoma:
https://github.com/jonescompneurolab/hnn-core/blob/0f96ef6c70f980f4ca1266451d4a17d0ab7f650a/hnn_core/tests/test_dipole.py#L143

I'm now (8c68a0b) satisfied that both PSA and LSA are doing what we think they are (found a bug). Tests should now cover everything.
Take it for a spin! Is the API reasonable? Code review also welcome.
I'm keeping this WIP since I still feel the cell position/orientation-matter should be dealt with first.

thanks @cjayb ! I am going to give this a spin.
Btw, I just realized an additional complication. If you change the cell positions, the weights will be affected since the lamtha is defined in terms of the distance between cells.

Yes: lamtha will have to be adjusted by whatever amount the distances are. The current implementation of distance-dependency assumes a rectangular grid. The staggering we're discussing here could still be implemented, but it would create a situation where the exact XY-plane coordinate was used for LFP, but only the grid location for connectivity. I don't think that's necessarily a show-stopper, it could be documented.

The latest push (ce1ce88) adds some plotting options, which are demonstrated in this gist. Quite frankly I'm not sure what to make of it :)

@cjayb I did a very superficial review right now. My feeling is that we should try to converge now with acknowledgements in the documentation what we think are known pitfalls. Let's iterate in future PRs. Thanks for the great work and the tremendous validation effort.

Two more tick boxes to hit before MRG... Should we use this gist as the example? Other suggestions?
The XYZ-matter is still TBD. I'd rather let this PR simmer until that's sorted, to be honest...

Should we use this gist as the example?

yeah I like this better than the gamma example to be honest. And later we can add the CSD diagram and perhaps we can also add pointers to the Lakatos paper figures (although that's auditory so different latencies ...)

Could you summarize the XYZ issue once more? Is it the problem that the coordinate systems are different or that we need to decide how to place the cells? I feel that for placing the cells, we need to start a separate issue/PR since we need to change the lamtha etc. We could either do that after merging this PR or do that first and rebase this PR against it.

I agree the spacing-issue is separate. And in a way so is this: our plotting function swaps the y and z axes, aligning the apical dendrites with Z, even though their Neuron alignment is Y. I could do the same in add_electrode_array, but just feel it's dodgy. I'd rather make our orientation explicit. Either we stick to the 'original' orientation defined in params_default and document it, or we say Z is apical, which is what most people would likely expect. Note that the variables in network.py refer to "zdiff", so even there the thought is XYZ, not XZY...

let's raise separate issues for them and address them before the release.
I think this PR should be ready once you swap out the example and I can do a full review of the code

Made some major changes, including the example. (@ntolley will look at your reviews next)
I discovered #351 in the process... With #352 applied, the example looks like this
The LFP is totally dominated by the currents around the soma due to spiking. This will likely change when we move the cells further apart.

rebase and check off last item? :)

Allrightythen, let's have some tough love on the code :)

@cjayb can I push a few commits here? I'll mostly just do cosmetic changes. I find it easier to review big code changes by messing with the code ...

of course! mess away :)

I decided against converting the internal data container here (this PR is already unreasonably large). Deferred it to #358 as it is also relevant for #341

Did Travis quit?

oh no:

I guess we got to switch to github actions now

Okay I did a full read through of the code ... looks reasonable to me. Let's iterate more after merging. I don't want to block other PRs for fear of rebase issues with this PR. Would you be able to look at the math/code soon @ntolley ?

Honestly I am ok with merging now and then raising an issue in the unlikely case that the math doesn't add up. It's more or less just my own desire to understand the calculations. Everything else I've been able to read carefully and am quite happy with what I've seen!
Also I'm less concerned since this is a completely new feature and doesn't risk breaking existing functionality.

@cjayb are you ready to merge? I think I won't be able to sleep at night with a 1500 line PR waiting to be merged and 300 comments ... :)

I'm ready to let this go :) There'll be plenty of opportunities to scrutinise the details in conjunction with the 0.2 release. Merge away, I say! I'll take the heat if we find something idiotic in there ;)
I'm (also) very convinced that eventually this needs to be pulled out of Network, but the current implementation makes it pretty trivial to do so, I think.

alright, I did the brave thing
tremendous effort @cjayb ! 🍺 🍻 And thanks @ntolley for working on the early draft

Indeed, I had plenty to work with from the get-go!


Allow hooks for when async reactors run out of retries

Why
In the case of an async reactor hitting the max number of retries, lets add a hook to allow someone to do
some custom work before the event lands in the DLQ.
What changed
Initial prototype of adding a retries_exhausted hook on Reactors.
How I tested [ Bullets for test cases covered ]

unit tests
rcon experimentation

closing this as we are moving to an ActiveJob: https://github.com/wealthsimple/eventsimple/pull/35
We can decide to provide a wrapper interface or reactors can subscribe to the hooks available directly:


Disable "resolving package graph" prompt

Is there a way to disable some of these loading messages? The VSCode Data Explorer extension runs the following to detect the pandas version, but it fails because of the loading message.
> ~/.local/bin/python -c "import pandas;print(pandas.__version__)"
info 14:43:01.705: Web view react rendered
warn 14:43:02.120: Failed to get version of Pandas to use the Data Viewer. tea: resolving package graph
```

VERBOSE=-1 python~3.7 -c "import pandas;print(pandas.__version__)"?

Thanks, but the problem is I can't easily control the env or command for these plugins. Was hoping for some kind of global configuration file or something like that to toggle these.

We output this message to stderr. The extension is behaving super strangely if it pulls in the data from both stdout AND stderr to get this information.
We certainly could and should add a global configuration system.
However I wouldn’t recommend setting tea to be silent globally just to work around one extension.
Personally I’d suggest making ~/.local/bin/python a shim shell script:
#!/bin/sh

tea --silent python "$@"

@ABevier can we add a test to verify that these messages go to stderr and not stdout so we ensure this never regresses, since if it did it would break a lot of tools in the manner of above.
I just verified locally that this is still true for my own peace of mind

Perhaps we should disable these messages when we're fd2 is not stdout?
What do other tools do here?

curl tea.xyz >/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13759  100 13759    0     0  71354      0 --:--:-- --:--:-- --:--:-- 73577
curl tea.xyz >/dev/null 2>&1

redirecting just stderr is hard; but this lets you play with it in zsh:
curl tea.xyz >/dev/null 2> >(wc)
       3      54     396

so indeed, curl outputs to stderr even when piped. That’s a pretty major one.
We are a different kind of tool. I guess an equivalent might be something like npx.

Another example of this being problematic here: https://github.com/jesseduffield/lazygit/issues/2716