Support EnforcedStyleForEmptyBraces for SpaceBeforeBlockBraces cop

I've started to integrate Rubocop into Puma development (https://github.com/puma/puma/pull/1325) and found out that SpaceBeforeBlockBraces cannot handle empty braces differently (like, for example, SpaceInsideBlockBraces does).
There is a lot of code like this:
# I don't want this one to be an offense
conf = Rack::Handler::Puma.config(->{}, @options)

Although for non-empty blocks there is always a space before the block.
This PR adds EnforcedStyleForEmptyBraces configuration parameter for Layout/SpaceBeforeBlockBracescop to allow handle empty braces differently.

Btw, it seems to me that this cop is not solving the problem it wants to. Doesn't the LambdaLiteral cop have support for this behavior? I don't see anything except lambda literals that would benefit from something like this...

Or not https://github.com/bbatsov/rubocop/blob/master/manual/cops_style.md#stylestabbylambdaparentheses

it seems to me that this cop is not solving the problem it wants to.

Sorry, don't understand what do you mean.
The problem I want to solve is to accept these two examples:
# no need of space before empty braces
fn = ->{}

# but always use space if lambda is not no-op
fn = -> { do_something }

Currently, I cannot achieve this with the SpaceBeforeBlockBraces cop, so I have to disable it (although I don't want to).

Yeah, I made a mistake I think.
Anyways, this has to be rebase on master - I see the changelog entry is in some weird place.

@bbatsov

this has to be rebase on master

Done. And green)


Godot 4 SSR jittering

Godot version
Godot_v4.0.2-stable_win64
System information
Windows 11,  Rtx 3090
Issue description
SETTINGS :
SSR max steps : 512
fade in : 0.150
fade out :  2.00
SDFGI is on.
its jittering, ugly pixelated. kinda blur? idk how to described it.
watch : https://www.veed.io/view/dd69c255-084e-4d51-822a-cc08ad77df5d?sharingWidget=true&panel=
watch 2 : https://www.veed.io/view/1d3d80e8-2ac6-49b7-9ffe-55cd4a663612/a84a4b44-1be8-4142-aa01-011655511e86?sharingWidget=true&panel=
Steps to reproduce
cant share the file. copyright texture
Minimal reproduction project
cant share the file. copyright texture

Can you reproduce the issue with another non-copyright texture? We can't really assess the issue without a minimal reproduction project.
And you can upload videos to github by dragging the file to the issue text box.

Closing due to lack of response. Please comment if you can still reproduce this bug on the latest Godot version.
PS: Since no minimal reproduction project was included in the original bug report, please upload one as well to ease troubleshooting.


Client class

Client SIde class for information about it self on a spezific server
Waiting for #15

/recheck


⚠ Joris - Perso has degraded performance

In 689d45e, Joris - Perso (https://joris-parmentier.fr) experienced degraded performance:

HTTP code: 200
Response time: 964 ms

Resolved: Joris - Perso performance has improved in 960dfac after 11 minutes.


Implement BatteryPool

What's needed?
User should be able to subscribe for data from some pool of Batteries.
BatteryPool should communicate with user using channels. This approach would be consisntent with LogicanMeter and EvChargerPool.
What data are needed:

average SoC
sum of SoP  - charge/discharge power bounds (from batteries and adjacent inverters)
sum of capacity
min/max of SoC bounds (maybe)
SUM of active power - only power from working batteries should be sum

How frequently we should send data:

Capacity, SoC bounds - when they change. These don't change frequently.
SoP - This can change frequently. However it is not needed for any calculation. It is needed only before making charge/discharge decision.

We could send new value only when it change. However this value are not needed frequently and we would compute it frequently. It seems to be lots of unnecessary work.
User can ask for this value every-time when he needs it. But then we have no channel communication. It won't be possible to move BatteryPool to separate process. (No : Implementation should be consistent with Logical Meter. So we should use channels.)

SoC - when it change
active power - with given resampling frequency.

Proposed solution
BatteryPool will use:

BatteryPoolStatus - to know what batteries are working
ComponentMetricResamplingActor - to subscribe for the metric data and receive them with some frequency.

Interpolation methods for the resampler:

Capacity, SoC bounds : Return last value -  This remains constant for most of the time, so returning last value seems to be correct.
SoP, SoC :

Return last value  (forward filling)
Linear interpolation. . Both SoC and SoP don't change the direction frequently. They either increases or decreases for some time. (or remain constant). If we find linear function based on first and last value in Resampler buffer, then we can predict the next value. How long time window we should store in ResamplerBuffer? After X seconds of inactivity battery/inverter will be considered as broken. (BatteryPoolStatus holds X argument). Time window could be X, too. Then we can interpolate as long as battery last message is considered as relevant.

active_power - average

Open questions

Different method to predict SoC, SoP.
How frequently we should send SoP - it can change frequently. We could send it in second basis and user can use peekable feature to get the latest value.

Use cases
No response
Alternatives and workarounds
No response
Additional context
No response

min/max of SoC bounds (maybe)

I think we should add this right away. Peak shaving I believe needs  to know the bounds @idlir-shkurti-frequenz
I would also suggest to provide the min and max SoP bounds. @tiyash-basu-frequenz The min bound will be a new upcoming feature by the Microgrid API that the PowerManager will need to support. Peak shaving shaving has already been tested with that feature by @idlir-shkurti-frequenz . @tiyash-basu-frequenz can shed more light onto that @ela-kotulska-frequenz Its quite an important feature we still need to test before the winter ends ;-) Its needed by the DC heating core app, however the SDK will have to support these min and max SoP bounds.

@ela-kotulska-frequenz when touching the BatteryPool I think we should start working / designing the PowerManager right away so that users would not directly need to be dealing with SoC and SoP bounds as well as talking care about the PowerDistributor. Of course up to you to decide. Just throwing that in as a point of discussion.

Should active_power be returned from BatteryPool or LogicalMeter.

Depending on whether the metric changes if a battery fails is one way to decide. I would tend to say to place it in the BatteryPool for consistency and/or offer both options. If a user wants to include the ID of a battery in a LogicMeter formula there is nothing we can prevent the user of doing so. Thats why I'd say probably giving the a consistent option via the BatteryPool might not hurt even if it feels a bit redundant. However, key point is that the LogicalMeter does not know when a battery goes offline and the battery needs to be removed from the calculation. I think this logic should always stay inside the BatteryPool. Just my few thoughts about it .... This discussion will also come back with EvChargerPool or the InverterPool etc.

@thomas-nicolai-frequenz
I discussed min/max bounds with Tiyash. I also need to add it to PowerDistributingActor.
I will focus on BatteryStatus first. The next important task would be to implement min bounds in PowerDistributingActor.
I will start PowerManager after that.

I discussed min/max bounds with Tiyash. I also need to add it to PowerDistributingActor. I will focus on BatteryStatus first. The next important task would be to implement min bounds in PowerDistributingActor. I will start PowerManager after that.

Indeed, that makes a lot of sense. Thank you very much! I simply missed that, sorry for that. Good catch!

max_soc: sum(battery.capacity*battery.max_soc for each working battery in battery pool)/total_capacity

What exactly is the battery.max_soc in this formula?

User should be able to subscribe for data from some pool of Batteries.

I struggled when I read the word "some" but it totally makes sense. A user might want to split up the entire pool of batteries in any given microgrid into two pools. That means a user should be able to:

Provide a list of battery ids that belong to a pool
Provide no list and the pool will by default use all available batteries


Decaf testing for Java 7 batch

I deployed a Java 7 batch here as part of https://github.com/phetsims/decaf/issues/89
https://phet-dev.colorado.edu/decaf/battery-resistor-circuit/1.0.0-dev.16/battery-resistor-circuit.html
https://phet-dev.colorado.edu/decaf/battery-voltage/1.0.0-dev.5/battery-voltage.html
https://phet-dev.colorado.edu/decaf/bound-states/1.0.0-dev.5/bound-states.html?simulation=band-structure
https://phet-dev.colorado.edu/decaf/bound-states/1.0.0-dev.5/bound-states.html?simulation=bound-states
https://phet-dev.colorado.edu/decaf/bound-states/1.0.0-dev.5/bound-states.html?simulation=covalent-bonds
https://phet-dev.colorado.edu/decaf/capacitor-lab/1.0.0-dev.4/capacitor-lab.html?simulation=capacitor-lab
https://phet-dev.colorado.edu/decaf/conductivity/1.0.0-dev.4/conductivity.html
https://phet-dev.colorado.edu/decaf/discharge-lamps/1.0.0-dev.4/discharge-lamps.html
https://phet-dev.colorado.edu/decaf/eating-and-exercise/1.0.0-dev.4/eating-and-exercise.html?simulation=eating-and-exercise
https://phet-dev.colorado.edu/decaf/efield/1.0.0-dev.4/efield.html
https://phet-dev.colorado.edu/decaf/electric-hockey/1.0.0-dev.4/electric-hockey.html
https://phet-dev.colorado.edu/decaf/faraday/1.0.0-dev.22/faraday.html?simulation=faraday
https://phet-dev.colorado.edu/decaf/faraday/1.0.0-dev.22/faraday.html?simulation=generator
https://phet-dev.colorado.edu/decaf/faraday/1.0.0-dev.22/faraday.html?simulation=magnet-and-compass
https://phet-dev.colorado.edu/decaf/faraday/1.0.0-dev.22/faraday.html?simulation=magnets-and-electromagnets
https://phet-dev.colorado.edu/decaf/fluid-pressure-and-flow/1.0.0-dev.7/fluid-pressure-and-flow.html?simulation=fluid-pressure-and-flow
https://phet-dev.colorado.edu/decaf/fluid-pressure-and-flow/1.0.0-dev.7/fluid-pressure-and-flow.html?simulation=under-pressure
https://phet-dev.colorado.edu/decaf/forces-1d/1.0.0-dev.5/forces-1d.html
https://phet-dev.colorado.edu/decaf/fourier/1.0.0-dev.4/fourier.html
https://phet-dev.colorado.edu/decaf/gene-network/1.0.0-dev.4/gene-network.html?simulation=gene-machine-lac-operon
https://phet-dev.colorado.edu/decaf/glaciers/1.0.0-dev.4/glaciers.html
https://phet-dev.colorado.edu/decaf/greenhouse/1.0.0-dev.5/greenhouse.html?simulation=greenhouse
https://phet-dev.colorado.edu/decaf/hydrogen-atom/1.0.0-dev.4/hydrogen-atom.html
https://phet-dev.colorado.edu/decaf/ideal-gas/1.0.0-dev.4/ideal-gas.html?simulation=balloons-and-buoyancy
https://phet-dev.colorado.edu/decaf/ideal-gas/1.0.0-dev.4/ideal-gas.html?simulation=gas-properties
https://phet-dev.colorado.edu/decaf/ideal-gas/1.0.0-dev.4/ideal-gas.html?simulation=reversible-reactions
https://phet-dev.colorado.edu/decaf/ladybug-motion-2d/1.0.0-dev.4/ladybug-motion-2d.html?simulation=ladybug-motion-2d
https://phet-dev.colorado.edu/decaf/lasers/1.0.0-dev.5/lasers.html?simulation=lasers
https://phet-dev.colorado.edu/decaf/maze-game/1.0.0-dev.4/maze-game.html
https://phet-dev.colorado.edu/decaf/membrane-channels/1.0.0-dev.4/membrane-channels.html?simulation=membrane-channels
https://phet-dev.colorado.edu/decaf/microwaves/1.0.0-dev.4/microwaves.html
https://phet-dev.colorado.edu/decaf/molecule-polarity/1.0.0-dev.3/molecule-polarity.html?simulation=molecule-polarity
https://phet-dev.colorado.edu/decaf/motion-2d/1.0.0-dev.4/motion-2d.html
https://phet-dev.colorado.edu/decaf/motion-series/1.0.0-dev.4/motion-series.html?simulation=forces-and-motion
https://phet-dev.colorado.edu/decaf/motion-series/1.0.0-dev.4/motion-series.html?simulation=ramp-forces-and-motion
https://phet-dev.colorado.edu/decaf/moving-man/1.0.0-dev.12/moving-man.html?simulation=moving-man
https://phet-dev.colorado.edu/decaf/mri/1.0.0-dev.5/mri.html
https://phet-dev.colorado.edu/decaf/natural-selection/1.0.0-dev.4/natural-selection.html?simulation=natural-selection
https://phet-dev.colorado.edu/decaf/natural-selection/1.0.0-dev.4/natural-selection.html?simulation=natural-selection
https://phet-dev.colorado.edu/decaf/nuclear-physics/1.0.0-dev.4/nuclear-physics.html?simulation=alpha-decay
https://phet-dev.colorado.edu/decaf/nuclear-physics/1.0.0-dev.4/nuclear-physics.html?simulation=beta-decay
https://phet-dev.colorado.edu/decaf/nuclear-physics/1.0.0-dev.4/nuclear-physics.html?simulation=nuclear-fission
https://phet-dev.colorado.edu/decaf/nuclear-physics/1.0.0-dev.4/nuclear-physics.html?simulation=radioactive-dating-game
https://phet-dev.colorado.edu/decaf/optical-quantum-control/1.0.0-dev.4/optical-quantum-control.html
https://phet-dev.colorado.edu/decaf/optical-tweezers/1.0.0-dev.6/optical-tweezers.html?simulation=molecular-motors
https://phet-dev.colorado.edu/decaf/optical-tweezers/1.0.0-dev.6/optical-tweezers.html?simulation=optical-tweezers
https://phet-dev.colorado.edu/decaf/optical-tweezers/1.0.0-dev.6/optical-tweezers.html?simulation=stretching-dna
https://phet-dev.colorado.edu/decaf/photoelectric/1.0.0-dev.5/photoelectric.html
https://phet-dev.colorado.edu/decaf/quantum-tunneling/1.0.0-dev.4/quantum-tunneling.html
https://phet-dev.colorado.edu/decaf/quantum-wave-interference/1.0.0-dev.4/quantum-wave-interference.html?simulation=davisson-germer
https://phet-dev.colorado.edu/decaf/quantum-wave-interference/1.0.0-dev.4/quantum-wave-interference.html?simulation=quantum-wave-interference
https://phet-dev.colorado.edu/decaf/radio-waves/1.0.0-dev.4/radio-waves.html
https://phet-dev.colorado.edu/decaf/reactions-and-rates/1.0.0-dev.6/reactions-and-rates.html
https://phet-dev.colorado.edu/decaf/rotation/1.0.0-dev.4/rotation.html?simulation=rotation
https://phet-dev.colorado.edu/decaf/rotation/1.0.0-dev.4/rotation.html?simulation=torque
https://phet-dev.colorado.edu/decaf/self-driven-particle-model/1.0.0-dev.4/self-driven-particle-model.html
https://phet-dev.colorado.edu/decaf/semiconductor/1.0.0-dev.4/semiconductor.html
https://phet-dev.colorado.edu/decaf/signal-circuit/1.0.0-dev.4/signal-circuit.html
https://phet-dev.colorado.edu/decaf/soluble-salts/1.0.0-dev.5/soluble-salts.html
https://phet-dev.colorado.edu/decaf/sugar-and-salt-solutions/1.0.0-dev.4/sugar-and-salt-solutions.html?simulation=sugar-and-salt-solutions
https://phet-dev.colorado.edu/decaf/sugar-and-salt-solutions/1.0.0-dev.4/sugar-and-salt-solutions.html?simulation=sugar-and-salt-solutions-macro
https://phet-dev.colorado.edu/decaf/sugar-and-salt-solutions/1.0.0-dev.4/sugar-and-salt-solutions.html?simulation=sugar-and-salt-solutions-micro
https://phet-dev.colorado.edu/decaf/the-ramp/1.0.0-dev.4/the-ramp.html?simulation=the-ramp
This will not invalidate testing in #497, but new testing should be on these Java 7 sims.
@ariel-phet requested that @KatieWoe add these to the testing spreadsheet.

Spreadsheet: https://docs.google.com/spreadsheets/d/10bO1Eimm5Y7TH4o7QPebpNUaIItxYK-VnKcSe24TXmc/edit#gid=147319993

Spreadsheet (https://docs.google.com/spreadsheets/d/10bO1Eimm5Y7TH4o7QPebpNUaIItxYK-VnKcSe24TXmc/edit#gid=147319993) is pretty much done. There is a QA recommendation section that has Publish, No, and Publish with discussion (probably ok to publish but with issues you might want to go over). IE and Win chrome still finishing up, but I feel pretty good at this juncture. IE pretty much is not usable, ChromeOS is spotty, and iPad is hit or miss on controls. Let me know if you have questions.

Awesome, thanks!  Closing.


XAdESVerifier verify CertDigest by index

Fix for #246. I found out that if I have multiple certificates in cert parameter to sign, the digest check would fail. Since it was signed with signxml, it was strange. Looking at code I found a problem. This is the solution.

Test suite fails for nonconformant-X_BE_CONN_10.xml. It has 2 X509Certificate nodes and 2 xades:Cert nodes. Previous code raised a InvalidDigest. The new code checks both Digests as OK as it compares by list index.
By removing "nonconformant-X_BE_CONN_10": InvalidDigest, from test_xades_interop_examples error_conditions, test suit will finish with OK.

Please add a PR description with the motivation for this change.

Notice that test is changed as one was an error before and now it is not as Digests match.

Thanks. I will take a look at that test case and see if this is the correct way to make it work.


GH2199: Add GlobPattern class

fixes #2199

@devlead your changes have been merged, thanks for your contribution 👍


bug: issue finding keys when using both mocha and arabica on 1 machine

Celestia Node version
v0.6.1
OS
Ubuntu LTS
Install tools
Node - v0.6.0
cd $HOME
rm -rf celestia-node
git clone https://github.com/celestiaorg/celestia-node.git
cd celestia-node/
git checkout tags/v0.6.0
make install
make cel-key

Node - v0.6.1
cd $HOME
rm -rf celestia-node
git clone https://github.com/celestiaorg/celestia-node.git
cd celestia-node/
git checkout tags/v0.6.1
make install
make cel-key

Golang
golang ver 1.19.1
Ethermint
git clone https://github.com/celestiaorg/ethermint.git
cd ethermint
make install
bash init.sh

For Arabica
NAMESPACE_ID=$(echo $RANDOM | md5sum | head -c 16; echo;)
DA_BLOCK_HEIGHT=$(curl https://rpc.limani.celestia-devops.dev/block | jq -r '.result.block.header.height')
ethermintd start --rollmint.aggregator true --rollmint.da_layer celestia --rollmint.da_config='{"base_url":"http://localhost:26659","timeout":60000000000,"gas_limit":6000000}' --rollmint.namespace_id $NAMESPACE_ID --rollmint.da_start_height $DA_BLOCK_HEIGHT

For Mocha
NAMESPACE_ID=$(echo $RANDOM | md5sum | head -c 16; echo;)
DA_BLOCK_HEIGHT=$(curl https://rpc-mocha.pops.one/block | jq -r '.result.block.header.height')
ethermintd start --rollmint.aggregator true --rollmint.da_layer celestia --rollmint.da_config='{"base_url":"http://localhost:26659","timeout":60000000000,"gas_limit":6000000}' --rollmint.namespace_id $NAMESPACE_ID --rollmint.da_start_height $DA_BLOCK_HEIGHT

Others
No response
Steps to reproduce it
Issue on Mocha w/ v0.6.0 & Ethermint
First, i’m using mocha with node v0.6.0 and celestiaorg/ethermint. I can confirm that:

I generated a random namespace ID
set the block height to the new mocha RPC
my node is running fine and I'm able to submit PFDs manually

starting celestia-node works fine, with a new, funded account:

celestia light start --core.ip https://rpc-mocha.pops.one:9090 --keyring.accname josh --gateway --gateway.addr <IP_ADDRESS> --gateway.port 26659

starting ethermint:

root@jcs-ethermint:~/ethermint# ethermintd start --rollmint.aggregator true --rollmint.da_layer celestia --rollmint.da_config='{"base_url":"http://loc
alhost:26659","timeout":60000000000,"gas_limit":6000000}' --rollmint.namespace_id $NAMESPACE_ID --rollmint.da_start_height $DA_BLOCK_HEIGHT

output (which is odd, I was able to manually submit a PFD successfully):

9:46PM INF Unlocking keyring                                                                                [38/459]
9:46PM INF starting ABCI with Tendermint
9:46PM INF service start impl=EventBus module=events msg={}
9:46PM INF service start impl=PubSub module=pubsub msg={}
badger 2022/12/15 21:46:34 INFO: All 3 tables opened in 0s
badger 2022/12/15 21:46:34 INFO: Discard stats nextEmptySlot: 0
badger 2022/12/15 21:46:34 INFO: Set nextTxnTs to 5
badger 2022/12/15 21:46:34 INFO: Deleting empty file: /root/.ethermintd/data/rollmint/000003.vlog
9:46PM INF service start impl=IndexerService module=txindex msg={}
9:46PM INF WARNING: using default DA block time DABlockTime=30000 module=BlockManager
9:46PM INF initializing blockchain state from genesis.json
9:46PM INF created new capability module=ibc name=ports/transfer
9:46PM INF port binded module=x/ibc/port port=transfer
9:46PM INF claimed capability capability=1 module=transfer name=ports/transfer
9:46PM INF asserting crisis invariants inv=1/11 module=x/crisis name=bank/nonnegative-outstanding
9:46PM INF asserting crisis invariants inv=2/11 module=x/crisis name=bank/total-supply
9:46PM INF asserting crisis invariants inv=3/11 module=x/crisis name=distribution/nonnegative-outstanding
9:46PM INF asserting crisis invariants inv=4/11 module=x/crisis name=distribution/can-withdraw                                                [21/459]
9:46PM INF asserting crisis invariants inv=5/11 module=x/crisis name=distribution/reference-count
9:46PM INF asserting crisis invariants inv=6/11 module=x/crisis name=distribution/module-account
9:46PM INF asserting crisis invariants inv=7/11 module=x/crisis name=gov/module-account
9:46PM INF asserting crisis invariants inv=8/11 module=x/crisis name=staking/module-accounts
9:46PM INF asserting crisis invariants inv=9/11 module=x/crisis name=staking/nonnegative-power
9:46PM INF asserting crisis invariants inv=10/11 module=x/crisis name=staking/positive-delegation
9:46PM INF asserting crisis invariants inv=11/11 module=x/crisis name=staking/delegator-shares
9:46PM INF asserted all invariants duration=3.493667 height=0 module=x/crisis
9:46PM INF service start impl=RPC msg={}
9:46PM INF service start impl=Node msg={}
9:46PM INF starting P2P client
9:46PM INF serving HTTP listen address={"IP":"<IP_ADDRESS>","Port":26657,"Zone":""}
9:46PM INF listening on address=/ip4/<IP_ADDRESS>/tcp/26656/p2p/12D3KooWG8o4Ku6waDXTJJqPXRCaz57o47PhjpvLdsjTzK8s33YZ module=p2p
9:46PM INF listening on address=/ip4/<IP_ADDRESS>/tcp/26656/p2p/12D3KooWG8o4Ku6waDXTJJqPXRCaz57o47PhjpvLdsjTzK8s33YZ module=p2p
9:46PM INF no seed nodes - only listening for connections module=p2p
9:46PM INF starting Celestia Data Availability Layer Client baseURL=http://localhost:26659 module=da_client
9:46PM INF working in aggregator mode block time=30000
9:46PM INF Using pending block height=1 module=BlockManager
9:46PM INF minted coins from module account amount=2059726034250856481aphoton from=mint module=x/bank
9:46PM INF submitting block to DA layer height=1 module=BlockManager
9:46PM ERR DA layer submission failed error="Codespace: 'sdk', Code: 19, Message: " attempt=1 module=BlockManager                              [0/459]
9:46PM ERR DA layer submission failed error="Codespace: 'sdk', Code: 19, Message: " attempt=2 module=BlockManager
9:46PM ERR DA layer submission failed error="Codespace: 'sdk', Code: 19, Message: " attempt=3 module=BlockManager
9:46PM ERR DA layer submission failed error="Codespace: 'sdk', Code: 19, Message: " attempt=4 module=BlockManager
9:46PM ERR DA layer submission failed error="Codespace: 'sdk', Code: 19, Message: " attempt=5 module=BlockManager
9:46PM ERR DA layer submission failed error="Codespace: 'sdk', Code: 19, Message: " attempt=6 module=BlockManager
9:46PM ERR DA layer submission failed error="Codespace: 'sdk', Code: 19, Message: " attempt=7 module=BlockManager
9:46PM INF Starting JSON-RPC server address=<IP_ADDRESS>:8545
9:46PM INF Starting JSON WebSocket server address=<IP_ADDRESS>:8546
9:46PM INF Served eth_getBalance conn=<IP_ADDRESS>:57685 duration=1.345641 err="failed to load hash from index: failed to load block hash for height
 0: key not found" module=geth reqid=9057584485
9:46PM ERR DA layer submission failed error="Codespace: 'sdk', Code: 19, Message: " attempt=8 module=BlockManager
^C9:46PM INF HTTP server shut down, waiting 5 sec
9:46PM INF service stop impl={"BlockIndexer":{},"IndexerService":{"Logger":{}},"Logger":{},"Mempool":{},"P2P":{},"Store":{},"TxIndexer":{}} msg={}
9:46PM INF stopping Celestia Data Availability Layer Client module=da_client
9:46PM ERR failed to read message error="context canceled" module=p2p
9:46PM ERR failed to read message error="context canceled" module=p2p
9:46PM ERR failed to read message error="context canceled" module=p2p
9:46PM ERR errors while stopping node: errors="context canceled; context canceled; context canceled; context canceled"
9:46PM INF Bye!

Switching to v0.6.1
I switched to v0.6.1 for celestia-node, and am experiencing the same issue with Mocha
Running Arabica
Running arabica, I run into an odd error. can’t seem to find the key (plus other issues) 👀
root@jcs-ethermint:~/celestia-node# ./cel-key list --keyring-backend test --node.type light
- address: celestia1634fwwck7uy72suz58krhvpkfr705g3darjeh4
  name: josh
  pubkey: '{"@type":"/cosmos.crypto.secp256k1.PubKey","key":"A1aoHxbCRqWfvLSgYve09t2RV0sFJZLywCngtacPn58x"}'
  type: local

root@jcs-ethermint:~/celestia-node# celestia light start --core.ip https://limani.celestia-devops.dev:9090 --keyring.accname josh --gateway --gateway.addr <IP_ADDRESS> --gateway.port 26659 --p2p.network arabica
2022-12-15T22:56:27.920Z        INFO    badger  v2@v2.2007.4/levels.go:183      All 0 tables opened in 0s

2022-12-15T22:56:27.923Z        INFO    badger  v2@v2.2007.4/value.go:1158      Replaying file id: 0 at offset: 0

2022-12-15T22:56:27.923Z        INFO    badger  v2@v2.2007.4/value.go:1178      Replay took: 46.198µs

2022/12/15 22:56:27 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Error: could not build arguments for function "github.com/celestiaorg/celestia-node/nodebuilder/rpc".RegisterEndpoints (/root/celestia-node/nodebuilder/rpc/constructors.go:15): failed to build state.Module: could not build arguments for function "github.com/celestiaorg/celestia-node/nodebuilder/state".ConstructModule.func3 (/root/celestia-node/nodebuilder/state/module.go:38): failed to build *state.CoreAccessor: could not build arguments for function "github.com/celestiaorg/celestia-node/nodebuilder/state".CoreAccessor (/root/celestia-node/nodebuilder/state/core.go:12): failed to build *types.KeyringSigner: received non-nil error from function "github.com/celestiaorg/celestia-node/nodebuilder/state".Keyring (/root/celestia-node/nodebuilder/state/keyring.go:18): josh.info: key not found [celestiaorg/cosmos-sdk@v1.4.0-sdk-v0.46.0/crypto/keyring/keyring.go:489]

i import the key to --p2p.network arabica and start with the flag, same error.
root@jcs-ethermint:~/celestia-node# ./cel-key list --keyring-backend test --node.type light --p2p.network arabica
- address: celestia1634fwwck7uy72suz58krhvpkfr705g3darjeh4
  name: josh
  pubkey: '{"@type":"/cosmos.crypto.secp256k1.PubKey","key":"A1aoHxbCRqWfvLSgYve09t2RV0sFJZLywCngtacPn58x"}'
  type: local

root@jcs-ethermint:~/celestia-node# ./cel-key list --keyring-backend test --node.type light --p2p.network mocha
- address: celestia1634fwwck7uy72suz58krhvpkfr705g3darjeh4
  name: josh
  pubkey: '{"@type":"/cosmos.crypto.secp256k1.PubKey","key":"A1aoHxbCRqWfvLSgYve09t2RV0sFJZLywCngtacPn58x"}'
  type: local

root@jcs-ethermint:~/celestia-node# ./cel-key list --keyring-backend test --node.type light
- address: celestia1634fwwck7uy72suz58krhvpkfr705g3darjeh4
  name: josh
  pubkey: '{"@type":"/cosmos.crypto.secp256k1.PubKey","key":"A1aoHxbCRqWfvLSgYve09t2RV0sFJZLywCngtacPn58x"}'
  type: local

root@jcs-ethermint:~/celestia-node# celestia light start --core.ip https://limani.celestia-devops.dev:9090 --keyring.accname josh --gateway --gateway.addr <IP_ADDRESS> --gateway.port 26659 --p2p.network arabica
2022-12-15T23:00:38.097Z        INFO    badger  v2@v2.2007.4/levels.go:183      All 0 tables opened in 0s

2022-12-15T23:00:38.101Z        INFO    badger  v2@v2.2007.4/value.go:1158      Replaying file id: 0 at offset: 0

2022-12-15T23:00:38.102Z        INFO    badger  v2@v2.2007.4/value.go:1178      Replay took: 96.895µs

2022/12/15 23:00:38 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Error: could not build arguments for function "github.com/celestiaorg/celestia-node/nodebuilder/rpc".RegisterEndpoints (/root/celestia-node/nodebuilder/rpc/constructors.go:15): failed to build state.Module: could not build arguments for function "github.com/celestiaorg/celestia-node/nodebuilder/state".ConstructModule.func3 (/root/celestia-node/nodebuilder/state/module.go:38): failed to build *state.CoreAccessor: could not build arguments for function "github.com/celestiaorg/celestia-node/nodebuilder/state".CoreAccessor (/root/celestia-node/nodebuilder/state/core.go:12): failed to build *types.KeyringSigner: received non-nil error from function "github.com/celestiaorg/celestia-node/nodebuilder/state".Keyring (/root/celestia-node/nodebuilder/state/keyring.go:18): josh.info: key not found [celestiaorg/cosmos-sdk@v1.4.0-sdk-v0.46.0/crypto/keyring/keyring.go:489]

Running Mocha
and if I switch to Mocha, it works fine (still can't connect the rollup to it, though):
celestia light start --core.ip https://rpc-mocha.pops.one:9090 --keyring.accname josh --gateway --gateway.addr <IP_ADDRESS> --gateway.port 26659 --p2p.network mocha

Removing keyring account from Arabica start cmd
if i remove the keyring account from the arabica start cmd, it seems to start fine, but it crashes like:
2022-12-15T23:05:39.553Z        ERROR   header/p2p      p2p/exchange.go:107     head request to trusted peer failed     {"trustedPeer": "12D3KooWHr2wqFAsMXnPzpFsgxmePgXb8BqpkePebwUgLyZc95bd", "err": "context deadline exceeded"}
2022-12-15T23:05:39.553Z        INFO    header/sync     sync/sync_head.go:145   new network head        {"height": 99533, "hash": "BF133D1DA95EBE7567E29679EA0B0933FAED9C25FEC79C6986936D783303119D"}
2022-12-15T23:05:39.553Z        WARN    header/sync     sync/sync_head.go:55    subjective initialization with an old header    {"height": 99533}
Error: node: failed to start: OnStart hook added by go.uber.org/fx.(*lifecycleHookAnnotation).Build.func1 failed: context deadline exceeded

github.com/celestiaorg/celestia-node/nodebuilder/header.newInitStore.func1() took 922.145µs from:
        github.com/celestiaorg/celestia-node/nodebuilder/header.newInitStore (/root/celestia-node/nodebuilder/header/constructors.go:91)
go.uber.org/fx.(*lifecycleHookAnnotation).Build.func1.1() took 101.374µs from:
        go.uber.org/fx.(*lifecycleHookAnnotation).Build.func1 (/root/go/pkg/mod/go.uber.org/fx@v1.18.2/annotated.go:447)
github.com/celestiaorg/celestia-node/fraud.(*ProofService).Start-fm() took 48.203µs from:
        github.com/celestiaorg/celestia-node/nodebuilder/fraud.newFraudService.func1 (/root/celestia-node/nodebuilder/fraud/constructors.go:31)
go.uber.org/fx.(*lifecycleHookAnnotation).Build.func1.1() took 33.243µs from:
        go.uber.org/fx.(*lifecycleHookAnnotation).Build.func1 (/root/go/pkg/mod/go.uber.org/fx@v1.18.2/annotated.go:447)
github.com/celestiaorg/celestia-node/nodebuilder/header.newP2PExchange.func1.1() took 4.618µs from:
        github.com/celestiaorg/celestia-node/nodebuilder/header.newP2PExchange.func1 (/root/celestia-node/nodebuilder/header/constructors.go:57)
github.com/celestiaorg/celestia-node/nodebuilder/p2p.PeerRouting.func1() took 1.794µs from:
        github.com/celestiaorg/celestia-node/nodebuilder/p2p.PeerRouting (/root/celestia-node/nodebuilder/p2p/routing.go:43)

Here's where I think I realized there may be an issue with key storage... kinda odd, if i start it on Arabica with --keyring.accname my_celes_key, it works, but that doesn’t show when i list keys in backend test. after checking in all keyring-backends i can’t find the key that I'm running with the node anywhere
root@jcs-ethermint:~/ethermint# cel-key list --keyring-backend os --node.type light
[]

root@jcs-ethermint:~/ethermint# cel-key list --keyring-backend file --node.type light
[]

root@jcs-ethermint:~/ethermint# cel-key list --keyring-backend test --node.type light
- address: celestia1634fwwck7uy72suz58krhvpkfr705g3darjeh4
  name: josh
  pubkey: '{"@type":"/cosmos.crypto.secp256k1.PubKey","key":"A1aoHxbCRqWfvLSgYve09t2RV0sFJZLywCngtacPn58x"}'
  type: local

root@jcs-ethermint:~/ethermint# cel-key list --keyring-backend os --node.type light --p2p.network arabica
[]

root@jcs-ethermint:~/ethermint# cel-key list --keyring-backend file --node.type light --p2p.network arabica
[]

root@jcs-ethermint:~/ethermint# cel-key list --keyring-backend test --node.type light --p2p.network arabica
- address: celestia1634fwwck7uy72suz58krhvpkfr705g3darjeh4
  name: josh
  pubkey: '{"@type":"/cosmos.crypto.secp256k1.PubKey","key":"A1aoHxbCRqWfvLSgYve09t2RV0sFJZLywCngtacPn58x"}'
  type: local

Expected result

No error on Ethermint rollup posting to Celestia light node
No issue with switching networks / key storage between networks (I think)

Actual result
Above 🫠
Relevant log output
No response
Notes
No response

re: rollmint & ethermint, it looks like it’s expecting a fee of 6000utia and then code 19 is for same request thereafter. so the main issue here is with keys 😄

fee field in requests was added recently (see: https://github.com/celestiaorg/celestia-node/commit/4734aac019e5d786be1148f8cbd0bc84ebc392a9).
go-cnc sets no fee, so default Go value (0) is set during deserialization of request in node, See:
https://github.com/celestiaorg/celestia-node/blob/4734aac019e5d786be1148f8cbd0bc84ebc392a9/api/gateway/state.go#L170-L171 and https://github.com/celestiaorg/celestia-node/blob/4734aac019e5d786be1148f8cbd0bc84ebc392a9/api/gateway/state.go#L186
Issue in go-cnc: https://github.com/celestiaorg/go-cnc/issues/39

thank you @tzdybal!
i think there still may lie an issue with key management on celestia-node here, can make separate issue if makes sense cc @renaynay @distractedm1nd

giving this a spin on this version:
Semantic version: v0.6.1-7-g1846a08
Commit: 1846a08bed6c3758a243ddde19e4f53dbe656b0a
Build Date: Tue Dec 20 14:01:42 EST 2022
System version: arm64/darwin
Golang version: go1.19.1

UX improvement
One thing that may help users, but I don't know how much of a lift it is: adding a field for location of key when user uses cel-key list. Or, maybe adding the --p2p.network <network> flag to cel-key commands so it is clear to users that Mocha is the default.
./cel-key list --keyring-backend test --node.type light
- address: celestia1zvy5a30mhclt4l5y3gr8jj83kyt5f2mq26jmr7
  name: my_celes_key
  pubkey: '{"@type":"/cosmos.crypto.secp256k1.PubKey","key":"AoVPF0VJ9Q7zhX60smDTXFYdizOCuVeDPuJSf5F4BpYE"}'
  type: local

testing out cel-key
If I use ./cel-key list --keyring-backend test --node.type light --p2p.network arabica, it will still create an empty node store at .celestia-light-arabica vs. checking the store in .celestia-light-arabica-2

Hey @jcstein, the commit hash is from main but can you re-run the same things on this branch https://github.com/celestiaorg/celestia-node/pull/1482 and see if the problem persists? The fix is still in PR form. Just want to confirm it solves your issue.

D'oh. Yeah, I will do tomorrow

nice, works as expected! thanks again @renaynay

I believe this issue was closed with https://github.com/celestiaorg/celestia-node/pull/1482? @renaynay


The smoke test for the example App fails due to testing framework limitations

example/src/App.test.js fails, throwing the following error:
TypeError: Cannot set property 'fillStyle' of null

This error is thrown because Phaser does not have access to a canvas element. This issue can be solved by testing the App component with @testing-library/react instead of react-dom. react-dom is currently being used only because it came with the boilerplate code generated by create-react-library.

See GameComponent.test.js for a working example of how to test React components with @testing-library/react.


Add CI Github Action for running tests

Hey @gurrpi, sorry for calling you on it here. I'm dealing with the serviceAccountKey.json file needed to use the Admin SDK in testing. How could we pass it on to Github Action? Maybe using secrets? Feel free to modify this PR, I have only defined an initial base layer to work on for the test CI.

Reference: https://firebase.google.com/docs/admin/setup

It would works to pass required auth key by using encrypted environment variable (aka. GitHub secret). How about to have e2e which could run locally (by using Firebase Emulator suite as we mentioned before)? It will run much faster so dev can get feedback much earlier and don't have to deal with a hassle of configuration for remote environments. Remote e2e test would ensures our code will run correctly on real world infrastructure.

Hey @gurrpi, sorry for calling you on it here. I'm dealing with the serviceAccountKey.json file needed to use the Admin SDK in testing. How could we pass it on to Github Action? Maybe using secrets? Feel free to modify this PR, I have only defined an initial base layer to work on for the test CI.

Reference: https://firebase.google.com/docs/admin/setup

It would works to pass required auth key by using encrypted environment variable (aka. GitHub secret). Btw, how about to have e2e which could run locally (by using Firebase Emulator suite as we mentioned before)? It will run much faster so dev can get feedback much earlier and don't have to deal with a hassle of configuration for remote environments. Remote e2e test would ensures our code will run correctly on real world infrastructure.

Great for Github Secrets :star:
Interesting as a proposal what concerns the usage of Firebase Local Emulator. When I had proposed to @daodesigner told me that it would be good for unit testing and not for e2e testing. However, we could have two e2e tests and decide whether to run them locally (emulator) or remotely (deployed dev environment). Is that what you meant? :)

Is it possible to do locally on PRs between branches and remote for merging to main? (Feel free to push back on this)

More urgent/important is definitely having tests that work against prod. Is it possible to do 'locally' on PRs between branches and remote for merging to main? Feel free to push back on this if it's more than a simple switch you flip.

It would seem easy enough switching between emulator and deployed infrastructure. What really changes is the initialization of Firebase services (whether emulated or not). I think this can be solved with an environment variable set when launching the tests.
Examples

Here is an example in using emulator => https://github.com/firebase/emulators-codelab/blob/master/codelab-final-state/functions/test.js#L67 (note the emulator init above L46).
An example without emulator but with the infra deployed can be found in our package actions => https://github.com/quadratic-funding/mpc-phase2-suite/blob/refactor/monorepo/packages/actions/test/index.test.ts#L94

LMK your thoughts @daodesigner @gurrpi

However, we could have two e2e tests and decide whether to run them locally (emulator) or remotely (deployed dev environment). Is that what you meant? :)

RIght! Thanks for the clarification.

Definitely agree with @daodesigner, let's push back local e2e and move the current scripts of the yarn test command to something more specific name (e.g. test-ci, so yarn test-ci command would call it).
Think following package.json file would be a good reference of separating test commands:
    "test:unit": "cross-env NODE_ENV=test mocha && cargo test --package parcel-js-swc-core --lib",
    "test:integration": "yarn workspace @parcel/integration-tests test",
    "test:integration-ci": "yarn workspace @parcel/integration-tests test-ci",
    "test": "yarn test:unit && yarn test:integration",

My points was yarn test command should pass without any background knowledge.

Scanning environment variables to detect the target environment can be an alternative to renaming the command. IMO renaming yarn command would be easier task. Feel free to pick any of these approaches (or 3rd way) if it makes sense.
having local e2e test to fulfil this idea. But it might require a degree of efforts, easy efforts are still an efforts XD,  to introduce Firebase Emulator.

Steps for testing with serviceAccountKey.json on prod:
Create secret called SERVICE_ACCOUNT with the entire JSON object
Use following actions:
jobs:
    test:
        name: Test
        runs-on: ubuntu-20.04
        steps:
            - name: Checkout Repo
              uses: actions/checkout@v3

            - name: Install Dependencies
              run: yarn install

            - name: Create JSON
              id: create-json
              uses<EMAIL_ADDRESS>              with:
                name: "serviceAccountKey.json"
                json: ${{ secrets.SERVICE_ACCOUNT }}
                dir: './packages/backend/'

            - name: Test
              run: yarn test:option

Complete solution to run both backend and actions tests in one workflow (Requires to set all .env variables as GitHub secrets):
Workflow file
Package-json new script: "test:ci": "export GOOGLE_APPLICATION_CREDENTIALS=\"./packages/backend/serviceAccountKey.json\" && jest --forceExit --coverage=false",
Changes to ./packages/actions/test/utils/index.ts - will need to remove .env.test and just call it .env for local testing
Changes to ./packages/actions/test/index.test.ts - add greater JEST timeout

Complete solution to run both backend and actions tests in one workflow (Requires to set all .env variables as GitHub secrets):
Workflow file Package-json new script: "test:ci": "export GOOGLE_APPLICATION_CREDENTIALS=\"./packages/backend/serviceAccountKey.json\" && jest --forceExit --coverage=false"
Changes to ./packages/actions/test/utils/index.ts - will need to remove .env.test and just call it .env for local testing Changes to ./packages/actions/test/index.test.ts - increase JEST timeout

My takeaways after proposals and discussions:

We continue with testing exclusively in the remote development environment. The local environment (i.e., Firebase local emulator) will be introduced along with unit testing. At that point, we would be complicating the workflow, now we just need to focus on e2e test for going in production (thanks @daodesigner for heads up).
We would have differentiation between test commands to achieve this goal yarn test command should pass without any background knowledge stated by @gurrpi. This way, we could call exclusively yarn test:ci for remote workflow, while yarn test will work in the future with unit and e2e tests locally in the emulator (no effort for configs). So, just run yarn test:e2e-local for testing locally on your machine right now (will be deprecated later and you will need the serviceAccountKey.json).
Thanks @ctrlc03 for the input, by transforming the serviceAccountKey.json as secret and small arrangements, we could actually achieve the goal of this PR.

Thank you all and collaborating with you is always exciting. I'm learning a lot of things every day <3 Keep up the good work!

Great summary!
You mean yarn test:e2e-local would run a test against prod environment (which does same as yarn test:ci) but for human, not CI?

Complete solution to run both backend and actions tests in one workflow (Requires to set all .env variables as GitHub secrets):
Workflow file Package-json new script: "test:ci": "export GOOGLE_APPLICATION_CREDENTIALS=\"./packages/backend/serviceAccountKey.json\" && jest --forceExit --coverage=false"
Changes to ./packages/actions/test/utils/index.ts - will need to remove .env.test and just call it .env for local testing Changes to ./packages/actions/test/index.test.ts - increase JEST timeout

My takeaways after proposals and discussions:

We continue with testing exclusively in the remote development environment. The local environment (i.e., Firebase local emulator) will be introduced along with unit testing. At that point, we would be complicating the workflow, now we just need to focus on e2e test for going in production (thanks @daodesigner for heads up).
We would have differentiation between test commands to achieve this goal yarn test command should pass without any background knowledge stated by @gurrpi. This way, we could call exclusively yarn test:ci for remote workflow, while yarn test will work in the future with unit and e2e tests locally in the emulator (no effort for configs). So, just run yarn test:e2e-local for testing locally on your machine right now (will be deprecated later and you will need the serviceAccountKey.json).
Thanks @ctrlc03 for the input, by transforming the serviceAccountKey.json as secret and small arrangements, we could actually achieve the goal of this PR.

Thank you all and collaborating with you is always exciting. I'm learning a lot of things every day <3 Keep up the good work!

Great summary! You mean yarn test:e2e-local would run a test against prod environment (which does same as yarn test:ci) but for human, not CI?

Yes, exactly @gurrpi! This command will be just a temporary command for running e2e "manually" (probably will be something like: test:e2e-local: "export GOOGLE_APPLICATION_CREDENTIALS=\"./packages/backend/serviceAccountKey.json\" && jest --forceExit" while test:ci: "jest --forceExit --coverage=false")

Complete solution to run both backend and actions tests in one workflow (Requires to set all .env variables as GitHub secrets):

Workflow file Package-json new script: "test:ci": "export GOOGLE_APPLICATION_CREDENTIALS=\"./packages/backend/serviceAccountKey.json\" && jest --forceExit --coverage=false"

Changes to ./packages/actions/test/utils/index.ts - will need to remove .env.test and just call it .env for local testing Changes to ./packages/actions/test/index.test.ts - increase JEST timeout

My takeaways after proposals and discussions:

We continue with testing exclusively in the remote development environment. The local environment (i.e., Firebase local emulator) will be introduced along with unit testing. At that point, we would be complicating the workflow, now we just need to focus on e2e test for going in production (thanks @daodesigner for heads up).

We would have differentiation between test commands to achieve this goal yarn test command should pass without any background knowledge stated by @gurrpi. This way, we could call exclusively yarn test:ci for remote workflow, while yarn test will work in the future with unit and e2e tests locally in the emulator (no effort for configs). So, just run yarn test:e2e-local for testing locally on your machine right now (will be deprecated later and you will need the serviceAccountKey.json).

Thanks @ctrlc03 for the input, by transforming the serviceAccountKey.json as secret and small arrangements, we could actually achieve the goal of this PR.

Thank you all and collaborating with you is always exciting. I'm learning a lot of things every day <3 Keep up the good work!

Great summary! You mean yarn test:e2e-local would run a test against prod environment (which does same as yarn test:ci) but for human, not CI?

Yes, exactly @gurrpi! This command will be just a temporary command for running e2e "manually" (probably will be something like: test:e2e-local: "export GOOGLE_APPLICATION_CREDENTIALS=\"./packages/backend/serviceAccountKey.json\" && jest --forceExit" while test:ci: "jest --forceExit --coverage=false")

All looks good! Will ping you about remote environment configuration soon.

Outdated due to unexpected issues with the forked repo. Here only for reference. New PR #229


Updated project to use sbt 1.7.2 and scala 2.12.19 [closes #2]

Could you please fix the merge conflicts.

Looks good for me. Merging


loop_tasks would ideally only need task_makefile, not also task_plan

right now you need to declare both as targets to be able to pass them to loop_tasks, e.g., https://github.com/USGS-R/nawqa_wqp/blob/0d82423a201b5eb6e196e385e83fe74a617a183e/1_wqpdata.yml#L92-L108
but it'd be convenient if you could have one target that made the task_plan+task_makefile and output just the task_makefile, then a second target that ran loop_tasks, then done.

Another 👍on this issue. With the introduction of as_promises in the create_task_makefile with #86 , we can create standalone remake files that don't need to be called by the parent if we choose as_promises = FALSE (I think we could probably do this before that refactor too, but as_promises is specifically designed to couple or decouple w/ the parent remake file depending on TRUE/FALSE). But, we can't use loop_tasks on the tasks within this remake file without generating the task_plan and using that as an input.


Bittrex - Freezing on Raspberry Pi

Describe the bug
Immediately froze after starting the bot,
Steps To Reproduce
install hummingbot using rpi source build
create pmm as strategy
select bittrex as exchange
set spread and order refresh time
start bot immediately hangs
2020-11-13 19:22:13,808 - 27171 - hummingbot.connector.exchange.bittrex.bittrex_api_order_book_data_source - INFO - Subscribed to ETH-USDT deltas
2020-11-13 19:22:14,272 - 27171 - hummingbot.connector.exchange.bittrex.bittrex_api_order_book_data_source - INFO - Websocket connection started...
2020-11-13 19:22:14,273 - 27171 - hummingbot.connector.exchange.bittrex.bittrex_api_order_book_data_source - ERROR - Unexpected error when listening on socket stream.
Traceback (most recent call last):
  File "/home/pi/hummingbot/hummingbot/connector/exchange/bittrex/bittrex_api_order_book_data_source.py", line 177, in listen_for_order_book_diffs
    async for raw_message in self._socket_stream():
  File "/home/pi/hummingbot/hummingbot/connector/exchange/bittrex/bittrex_api_order_book_data_source.py", line 200, in _socket_stream
    yield await conn.msg_queue.get()
AttributeError: 'Connection' object has no attribute 'msg_queue'
2020-11-13 19:22:14,276 - 27171 - hummingbot.connector.exchange.bittrex.bittrex_api_order_book_data_source - INFO - Subscribed to ETH-USDT deltas

Screenshots

Release version
0.33.0
Attachments
logs_conf_pure_mm_1.log
┆Issue is synchronized with this Clickup task by Unito

Getting the same issue when running with Bittrex on Hummingbot with RaspberryPi
Steps:

Create PMM strategy
Connect Bittrex API
Start bot then it will not respond.

Version:
0.36.0
2021-02-24 23:44:48,277 - 1 - hummingbot.connector.exchange.bittrex.bittrex_api_order_book_data_source - INFO - Websocket connection started...
2021-02-24 23:44:48,278 - 1 - hummingbot.connector.exchange.bittrex.bittrex_api_order_book_data_source - ERROR - Unexpected error when listening on socket stream.
Traceback (most recent call last):
  File "/hummingbot/connector/exchange/bittrex/bittrex_api_order_book_data_source.py", line 177, in listen_for_order_book_diffs
    async for raw_message in self._socket_stream():
  File "/hummingbot/connector/exchange/bittrex/bittrex_api_order_book_data_source.py", line 200, in _socket_stream
    yield await conn.msg_queue.get()
AttributeError: 'Connection' object has no attribute 'msg_queue'
2021-02-24 23:44:48,279 - 1 - hummingbot.connector.exchange.bittrex.bittrex_api_order_book_data_source - INFO - Subscribed to ADA-USD deltas
2021-02-24 23:44:49,125 - 1 - hummingbot.connector.exchange.bittrex.bittrex_api_order_book_data_source - INFO - Websocket connection started...
2021-02-24 23:44:49,125 - 1 - hummingbot.connector.exchange.bittrex.bittrex_api_order_book_data_source - ERROR - Unexpected error when listening on socket stream.
Traceback (most recent call last):
  File "/hummingbot/connector/exchange/bittrex/bittrex_api_order_book_data_source.py", line 177, in listen_for_order_book_diffs
    async for raw_message in self._socket_stream():
  File "/hummingbot/connector/exchange/bittrex/bittrex_api_order_book_data_source.py", line 200, in _socket_stream
    yield await conn.msg_queue.get()
AttributeError: 'Connection' object has no attribute 'msg_queue'
2021-02-24 23:44:49,126 - 1 - hummingbot.connector.exchange.bittrex.bittrex_api_order_book_data_source - INFO - Subscribed to ADA-USD deltas

I have the same error.
logs_conf_pure_mm_1_matic_bittrex.log

Hello, i found the problem. Hummingbot use an different git branch for the package signalr-client-aio
To fix this we must replace it.
pip install --upgrade --force-reinstall git+https://github.com/CoinAlpha/python-signalr-client

The issue seems to self resolve and is no longer being encountered on version dev-1.0.0

Note: if you are using conda Python 3.8.2 or higher versions and getting the issue below during ./install please update setup/environment-linux-aarch64.yml dependency from cryptography==2.8 to cryptography==3.1.1


Add to class message  method GetBodyPtr() for Marshaling

public IntPtr GetBodyPtr() => nng_msg_body(NngMsg);
or
public T AsStruct() => Marshal.PtrToStructure(nng_msg_body(NngMsg));

I added IntPtr AsPtr() so it should work with Marshal (and probably other things as well) in PR#56.


Windows Defender Ransomware protection blocks WSL

Windows 10 Family Insider Preview 1803.
When enabled, the ransomware protection feature of Windows Defender prevents access to Windows files for WSL binaries. I tried to add wsl.exe and bash.exe to the exceptions list but nothing changed.
How to reproduce: try execute 'touch test' in a ransomware protected Windows folder in WSL and it will fail with 'touch: cannot touch 'a': No such file or directory'.
Is there any way to allow some WSL binaries ?

Tried this locally and I was able to add Linux binaries via the "Allow an app through Controlled folder access" menu.  Take a look at the "recently blocked apps" list.


Invalid cross-device link

@cket
Getting OSError: [Errno 18] Invalid cross-device link when running ./jenkins in the master branch of toil-vg on an AWS instance which is using the single machine batch system. Not sure if this is a toil issue, but it appears as though whatever was merged to master last should not have passed the unit tests on jenkins. So it's either that or there's a difference between running ./jenkins on a 4.2xlarge aws instance and running the unit tests on the official jenkins server.
Versions:
toil (3.5.0a1.dev251)
toil-lib (1.1.0a1) (custom toil-lib version from https://github.com/cmarkello/toil-lib)
toil-vg (1.1.0a1)
Forwarded from  https://github.com/BD2KGenomics/toil-vg/issues/84
Traceback:
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Released lock
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Obtained lock on file /var/lib/toil/toil-0e550c40-21b0-4b43-ba7a-dcecb0e0f0a1/cache-0e550c40-21b0-4b43-ba7a-dcecb0e0f0a1/.cacheLock
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Released lock
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Obtained lock on file /var/lib/toil/toil-0e550c40-21b0-4b43-ba7a-dcecb0e0f0a1/cache-0e550c40-21b0-4b43-ba7a-dcecb0e0f0a1/.cacheLock
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Released lock
L/Y/jobRtcOBD    DEBUG:toil.job:Loading user function run_xg_indexing from module ModuleDescriptor(dirPath='/home/mesosbox/toil-vg/src', name='toil_vg.vg_index', fromVirtualEnv=False).
L/Y/jobRtcOBD    WARNING:toil.resource:Can't find resource for leader path '/home/mesosbox/toil-vg/src/toil_vg'
L/Y/jobRtcOBD    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/mesosbox/toil-vg/src', name='toil_vg.vg_index', fromVirtualEnv=False)
L/Y/jobRtcOBD    INFO:realtime:Starting xg indexing...
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Obtained lock on file /var/lib/toil/toil-0e550c40-21b0-4b43-ba7a-dcecb0e0f0a1/cache-0e550c40-21b0-4b43-ba7a-dcecb0e0f0a1/.cacheLock
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Cache miss on file with ID 'b/o/tmpZYsFhp.tmp'.
L/Y/jobRtcOBD    DEBUG:toil.fileStore:LOG-TO-MASTER: CACHE: Creating a harbinger file for (b/o/tmpZYsFhp.tmp).
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Obtained lock on file /var/lib/toil/toil-0e550c40-21b0-4b43-ba7a-dcecb0e0f0a1/cache-0e550c40-21b0-4b43-ba7a-dcecb0e0f0a1/.cacheLock
L/Y/jobRtcOBD    DEBUG:toil.fileStore:LOG-TO-MASTER: CACHE: Deleting the harbinger file for (b/o/tmpZYsFhp.tmp)
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Released lock
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Released lock
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Obtained lock on file /var/lib/toil/toil-0e550c40-21b0-4b43-ba7a-dcecb0e0f0a1/cache-0e550c40-21b0-4b43-ba7a-dcecb0e0f0a1/.cacheLock
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Released lock
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Obtained lock on file /var/lib/toil/toil-0e550c40-21b0-4b43-ba7a-dcecb0e0f0a1/cache-0e550c40-21b0-4b43-ba7a-dcecb0e0f0a1/.cacheLock
L/Y/jobRtcOBD    DEBUG:toil.fileStore:CACHE: Released lock
L/Y/jobRtcOBD    Traceback (most recent call last):
L/Y/jobRtcOBD      File "/home/mesosbox/toil-vg/.env/lib/python2.7/site-packages/toil/worker.py", line 331, in main
L/Y/jobRtcOBD        job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
L/Y/jobRtcOBD      File "/home/mesosbox/toil-vg/.env/lib/python2.7/site-packages/toil/job.py", line 1172, in _runner
L/Y/jobRtcOBD        returnValues = self._run(jobWrapper, fileStore)
L/Y/jobRtcOBD      File "/home/mesosbox/toil-vg/.env/lib/python2.7/site-packages/toil/job.py", line 1120, in _run
L/Y/jobRtcOBD        return self.run(fileStore)
L/Y/jobRtcOBD      File "/home/mesosbox/toil-vg/.env/lib/python2.7/site-packages/toil/job.py", line 1283, in run
L/Y/jobRtcOBD        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
L/Y/jobRtcOBD      File "/home/mesosbox/toil-vg/src/toil_vg/vg_index.py", line 213, in run_xg_indexing
L/Y/jobRtcOBD        read_from_store(job, options, inputGraphFileID, graph_filename)
L/Y/jobRtcOBD      File "/home/mesosbox/toil-vg/src/toil_vg/vg_common.py", line 230, in read_from_store
L/Y/jobRtcOBD        return job.fileStore.readGlobalFile(id_or_key, path)
L/Y/jobRtcOBD      File "/home/mesosbox/toil-vg/.env/lib/python2.7/site-packages/toil/fileStore.py", line 380, in readGlobalFile
L/Y/jobRtcOBD        '/.'.join(os.path.split(cachedFileName)))
L/Y/jobRtcOBD      File "/home/mesosbox/toil-vg/.env/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 240, in readFile
L/Y/jobRtcOBD        os.link(jobStoreFilePath, localFilePath)
L/Y/jobRtcOBD    OSError: [Errno 18] Invalid cross-device link
L/Y/jobRtcOBD    Exiting the worker because of a failed jobWrapper on host ip-172-31-31-184
L/Y/jobRtcOBD    ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-31-184
L/Y/jobRtcOBD    WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job L/Y/jobRtcOBD to 0

What is the full command that produced this error?

Nevermind, wrong ticket.

I believe that there is an optimization in the file job store that just hard links files from the job store to the local dir instead of copying, and that is what is causing this error. An easy fix is to copy if linking fails.
@cmarkello there are configuration differences between running ./jenkins and the actual Jenkins instances - if you want to replicate Jenkins more closely you can use CGCloud to spin up instances with Jenkins Slave roles.

@cket I was just running into this too.  It's because on the toil cluster created by cgcloud, you get
TOIL_WORKDIR="/var/lib/toil"
For some reason, this gets its own mount point.  as seen in /proc/mounts
/dev/xvda1 /var/lib/toil ext4 rw,relatime,data=ordered 0 0
Now even though this is the same device as, say, /tmp (which is also on /dev/xvda1), because it's a different mount point you get the crosslink error.   But the check within toil using os.stat().st_dev doesn't catch this (https://github.com/BD2KGenomics/toil/blob/master/src/toil/jobStores/fileJobStore.py#L236)
FWIW, you can quickly reproduce on the command line.  On instance set up as described here: https://github.com/BD2KGenomics/toil-vg
cd
touch x
ln x /var/lib/toil/x
ln: failed to create hard link ‘/var/lib/toil/x’ => ‘x’: Invalid cross-device link
But both devices the same:
df .
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1      10178756 6111572   3548480  64% /
mesosbox@ip-172-31-22-85:~$ df /var/lib/toil/
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1      10178756 6111572   3548480  64% /var/lib/toil

Stale.  Please reopen if still an issue.


SOLVED - Mentalist Not Working in Python3.11 Onwards

First, install mentalist using
sudo python3 setup.py install
Then first, let's solve this error ->

AttributeError: module 'locale' has no attribute 'format'. Did you mean: '_format'?

In VSCode (or your favorite text editor), Open /usr/local/lib/python3.12/dist-packages/Mentalist-1.0-py3.12.egg/mentalist/view/main.py.
Search for locale.format
Replace it with locale.format_string
Repeat steps from 1 to 3 for file adder.py

Now, let's solve this ->

AttributeError: module 'inspect' has no attribute 'getargspec'. Did you mean: 'getargs'?

Open /usr/local/lib/python3.12/dist-packages/Mentalist-1.0-py3.12.egg/mentalist/model.py
Replace the whole function as below (Line approx- 456 to 468)
Check for any indentation errors after pasting the code.

result = True
signature = inspect.signature(self.__class__.__init__)
param_names = [param.name for param in signature.parameters.values()]

for name in param_names:
    if name in ['self', 'controller']:
        continue
    if (name in other.__dict__) and (self.__dict__[name] != other.__dict__[name]):
        result = False

return result

If you are getting other locale related errors, run following commands.
export LC_ALL="en_US.UTF-8"
export LC_CTYPE="en_US.UTF-8"
sudo dpkg-reconfigure locales

Done!
Hope this helps. If you face any other issues, please comment below.

Plz can u upload your file fixed?


ENH: stats: add multivariate_laplace

Is your feature request related to a problem? Please describe.
Proposed new feature or change:
I am interested in submitting a new feature proposal as an issue. Specifically, I have noticed that scipy.stats.laplace only allows for the specification of location (mean) and scale (decay) parameters, and does not have an option for Multivariate Laplace. Conversely, while scipy.stats.multivariate_normal is available for normal distribution, there is currently no equivalent for multivariate Laplace.
mean = [0, 0]
cov = [[1, 0], [0, 100]]
x, y = scipy.stats.multivariate_normal(mean, cov).T
loc, scale = 0., 1.
s = scipy.stats.laplace(loc, scale, 1000)
Describe the solution you'd like.
I want to implement a new function multivariate_laplace
scipy.stats.multivariate_laplace
A multivariate laplace random variable.
The mean keyword specifies the mean. The cov keyword specifies the covariance matrix.
Parameters:
mean: array_like
Mean of the distribution.
cov: array_like
Symmetric positive (semi)definite covariance matrix of the distribution.
allow_singularbool, default: False
Whether to allow a singular covariance matrix. This is ignored if cov is a [Covariance]
seed{None, int, np.random.RandomState, np.random.Generator}, optional
Used for drawing random variates. If seed is None, the RandomState singleton is used. If seed is an int, a new RandomState instance is used, seeded with seed. If seed is already a RandomState or Generator instance, then that object is used. Default is None.
Describe alternatives you've considered.
No response
Additional context (e.g. screenshots, GIFs)
No response

@jiawei-zhang-a Please suggest this on the forum if you are still interested in this feature.

Which version of the multivariate Laplace distribution did you want? There are a few.
Here is some code for the version on Wikipedia, adapted from the multivariate normal:

import numpy as np
from scipy.stats._multivariate import multi_rv_generic, mvn_docdict_params, _LOG_2PI
from scipy import stats, special, integrate
from scipy._lib import doccer
rng = np.random.default_rng(3409235845834753)

class multivariate_laplace_gen(multi_rv_generic):

    def __init__(self, seed=None):
        super().__init__(seed)
        self.__doc__ = doccer.docformat(self.__doc__, mvn_docdict_params)

    def _process_parameters(self, mean, cov):
        """
        Infer dimensionality from mean or covariance matrix, ensure that
        mean and covariance are full vector resp. matrix.
        """
        if not isinstance(cov, stats.Covariance):
            L = np.linalg.cholesky(cov)
            cov = stats.Covariance.from_cholesky(L)

        dim = cov.shape[-1]
        mean = np.array([0.]) if mean is None else mean
        message = (f"`cov` represents a covariance matrix in {dim} dimensions,"
                   f"and so `mean` must be broadcastable to shape {(dim,)}")
        try:
            mean = np.broadcast_to(mean, dim)
        except ValueError as e:
            raise ValueError(message) from e
        return dim, mean, cov

    def logpdf(self, x, mean=None, cov=1):
        dim, mean, cov = self._process_parameters(mean, cov)
        x = np.asarray(x)
        return self._logpdf(x, mean, cov)

    def pdf(self, x, mean=None, cov=1):
        return np.exp(self.logpdf(x, mean, cov))

    def _logpdf(self, x, mean, cov):
        log_det_cov, k = cov.log_pdet, cov.rank
        dev = x - mean
        if dev.ndim > 1:
            log_det_cov = log_det_cov[..., np.newaxis]
            k = k[..., np.newaxis]
        maha = np.sum(np.square(cov.whiten(dev)), axis=-1)
        v = 1 - k/2
        return (np.log(2*special.kv(v, np.sqrt(2*maha)))
                - (k*_LOG_2PI + log_det_cov - v * np.log(maha/2))/2)

multivariate_laplace = multivariate_laplace_gen()

def pdf(x, mean, cov):
    # reference PDF
    k = cov.shape[-1]
    v = (2 - k) / 2
    x = x - mean
    maha = x @ (np.linalg.inv(cov) @ x)
    det = np.linalg.det(cov)
    den = (2*np.pi)**(k/2) * det**0.5
    t1 = (maha/2)**(v/2)
    t2 = special.kv(v, np.sqrt(2*maha))
    return 2 / den * t1 * t2

n = 2
x = rng.random(n)
cov = rng.random((n, n))
cov = cov + cov.T + n*np.eye(n)
mean = rng.random(n)
multivariate_laplace.pdf(x, mean, cov), pdf(x, mean, cov)

# integrate.dblquad(lambda x, y: pdf([x, y], mean, cov), -np.inf, np.inf, -np.inf, np.inf)

@jiawei-zhang-a Please suggest this on the forum if you are still interested in this feature.


Chore: Decimal should be Copy

Description
And also need to fix this lint:
error: incorrect implementation of `clone` on a `Copy` type
  --> crates/decimal/src/lib.rs:80:29
   |
80 |       fn clone(&self) -> Self {
   |  _____________________________^
81 | |         Self {
82 | |             mantissa: self.mantissa(),
83 | |             scale: self.scale(),
84 | |         }
85 | |     }
   | |_____^ help: change this to: `{ *self }`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#incorrect_clone_impl_on_copy_type
   = note: `#[deny(clippy::incorrect_clone_impl_on_copy_type)]` on by default

https://rust-lang.github.io/rust-clippy/master/index.html#/incorrect_clone_impl_on_copy_type
This is weird. The Copy trait should depend on Clone... Feels the other way around:
pub trait Copy: [Clone](https://doc.rust-lang.org/std/clone/trait.Clone.html) { }

Anyway, I will make this change.