Energy data not preserved when moving from StealthChesnut integration

The documentation says that when moving from stealthchesnut to this integration "Changes to the energy dashboard will not effect your older energy statistics." however when I have done this my previous energy data is not shown.
I posted an issue on the wiki but I've not yet had any helpful assistance. I realise this might not be an issue with this integration but something related to my HA setup, I would appreciate help in getting to the bottom of this and hopefully getting the 6 months of data displayed again.

I’ve never known the energy dashboard statistics to be reset, it’s something a lot of people have asked for and very difficult to manage, but looking at the graph you posted something appears to have gone wrong. Is there any historical data showing in the energy dashboard, pv, battery charge/discharge, grid consumption, export etc..?
Also worth asking the question do you have recent backups, preferably just prior to moving across to this integration in case we need to roll it back and try again without deleting the utility meters, and the old Riemann ‘sum’ entities, then not using the integration to auto-update the dashboard (it should still work on the old settings if the utility meters etc.. are in place)

It might also be worth checking the statistics, go to Developer Tools, Statistics and have a look for any entities listed that have the words ‘FIX ISSUE’ next to them - we are particularly interested in the original entity names you used in the dashboard (if you can remember) possibly named pv1_daily, pv2_daily, feedin daily. If they are listed with a fix issue next to them, click it and screenshot what it says on here it’’’ be things like ‘the unit of the entity changed’ or similar.

Hi Fozzie, thanks for getting back to me. In answer to your questions:
Is there any historical energy dashboard data - no, all the energy dashboard data stops displaying around 6pm on 2nd May which is when I switched over to this integration.
Backups? - I have been using the google drive backup add-on and I hadn't checked how it was configured. I do have a backup but it's only from after the 2nd (3rd) so I don't have a full backup from prior to switching.
Entities with 'FIX ISSUE' - yes, although I think a number of these are not related to this issue. I originally tried using a plugin for foxess that scraped the foxess portal over a year ago. It didn't work very well and I gave up on it after a few days and removed it. This left behind a number of orphaned entities I never bothered removing. I still have these and they're listed with 'there is no state available for this entity'. I'll ignore those.
pv1_daily - this does have the unit changed error, I presume I ask it to fix the issue by updating the units of the old data?

Ok, thanks - I hope not to have to go near the backups but always the first question to ask.
On the pv1_daily there is definitely been a error with the historical unit, earlier HA versions weren't that fussy but for the last 12 months or so it wouldn't have allowed you to configure statistics without fixing it first.
I think before we fix it, as you've added back in the utility meters i'd like as a test for you to go to your energy dashboard (Settings, Dashboards, Energy) and add back in the old entity names - but leave the new ones in place.
So for grid_consumption add grid_daily,
for Return to Grid add feedin_daily
Solar panels would be pv1 & 2_daily
and for the battery Energy going in bat_charge_daily, and Energy coming out would be bat_discharge_daily
Do you still have the original custom sensor sensor.feed_in_power setup in your configuration.yaml - if you do can you change the state line to this -
state: {{ states('sensor.feed_in') | float(default=0) }}
But if you have deleted the old template sensor for feed_in_power, change instead the utility meter for feedin_daily and replace the source with sensor.feed_in
After that, HA will need a restart and then a bit of time to gather it's statistics - i'm hoping that you should start to see some history displayed.

Hi,
When trying to add the utility meters back in I get the error 'Pick sensors which measure energy going in to and out of the battery in either of GJ, kWh, MJ, MWh, Wh.'
As suggested for Grid Consumption I'm trying to add 'sensor.grid_daily'.
In my configuration.yaml there's a utility meter configured as
  grid_daily:
    source: sensor.grid_consumption_sum
    cycle: daily

and grid_consumption_sum is defined as a sensor as follows:
  - method: left
    name: grid_consumption_sum
    platform: integration
    round: 2
    source: sensor.grid_consumption
    unit_time: h

and there's a template for grid consumption as follows
      - name: "Grid Consumption"
        device_class: "power"
        unit_of_measurement: "kW"
        state: >
            {% if (states('sensor.grid_ct') | float(default=0) ) < 0 %}
            {{ states('sensor.grid_ct') | float(default=0) * -1 }}
            {% else %}
            0
            {% endif %}

Ok, lets take these one at a time - the grid consumption template shouldn't be there anymore - that will clash with the new integration, can you either remove or comment out that code, once removed it's best to restart again.
Once it has restarted go to the integrations page for FoxESS modbus and list all the sensors - look for Grid Consumption in the list, click on it and then on the cog in the top right corner, about half way down the page will be the entity_id which should be sensor.grid_consumption but as you had the template in your config it's likely to be sensor.grid_consumption_2 (or similar) - if it is anything other than sensor.grid_consumption change it to that and click UPDATE.
That will correct the entity name and the both the _sum and utility meter should start to record again.
Moving next to Feedin do you have the utility meter (feedin_daily), the Riemann sum (feedin_sum) and template sensor (feed_in_power) in your config, can you post what you have for these?

ok, thanks - makes sense.
In the config can you change the state of the Feed In Power template to be as follows -
state: {{ states('sensor.feed_in') | float(default=0) }}

Save your config, click Developer Tools, and in the YAML tab click on 'TEMPLATES' to reload that section
This will now use the new integration to feed the sum and utility meter.
Now, just to go back to the Statistics 'Fix Issue' for pv1_daily, can you click on that and choose the update the historical values to kWh without converting
Can I check were you able to add any of the older xxx_daily meters back in to any section of the Energy Dashboard without error ?

I've clicked the fix issue and updated the historical data without converting but that doesn't appear to have made any difference.
For grid_daily, feedin_daily and pv1_daily all give the error that the sensor must use  GJ, kWh, MJ, MWh, Wh.

hmmm, both are set correctly? yet they are being rejected by the Energy Dashboard.
I assume the entity listed for grid consumption is Grid Consumption Today?, can you delete that entity and try and re-add grid_daily

The entity listed currently in the grid consumption dashboard is Grid Consumption Total (sensor.grid_consumption_energy_total). That's the one from this integration that seems to be working fine.
That has a bar graph icon next to it, where as sensor.grid_daily has a counter style icon. Should I try adding sensor.grid_consumption to the energy dashboard ? That's got the same bar graph icon as the current entry and the entities tab says it's provided by a sensor and not FoxESS - Modbus.

Try adding the sum instead so grid_consumption_sum - that's got to be state_class total_increasing

the bar graph means total increasing, whereas the counter is a utility meter - both should be acceptable to the dashboard :shrug:

Hmm, I'm still getting the error that implies the units are wrong.

Can we leave it overnight to let the statistics keep building and let the utility meters reset at midnight, then try again tomorrow to load grid_daily.
I've never come across this before, i'll do a bit of research and see what I can find - if we can't get it to update the dashboard, it may need that database backup to be restored and then start over again with modifying the config and re-load the integration - but this time don't update the dashboard, use the existing utility meters as they are configured.

Yes, leaving it overnight is no problem.
Unfortunately I don't believe I've got a database backup from before I switched the integration so if that's needed then I'm out of luck. While it sucks to lose that data it is a lesson in having good backups that I'll have learnt 😃

Ouch, ok keep fingers crossed - i’ll see what I can find in the meantime 👍

When you get the chance to try again, if it throws the same error - can you screenshot any errors you get at that time from your log

I'm still getting the same error, and checking the logs it seems this is appearing at the relevant time.
Logger: homeassistant.components.websocket_api.http.connection
Source: components/energy/data.py:155
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 10 May 2023 at 18:09:56 (6 occurrences)
Last logged: 11:06:59

[2591611056] Error handling message: Unknown error (unknown_error) nigel from <IP_ADDRESS> (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36)
[2591751592] Error handling message: Unknown error (unknown_error) nigel from <IP_ADDRESS> (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36)
[2579096144] Error handling message: Unknown error (unknown_error) nigel from <IP_ADDRESS> (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36)
[2587615272] Error handling message: Unknown error (unknown_error) nigel from <IP_ADDRESS> (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36)
[2635607128] Error handling message: Unknown error (unknown_error) nigel from <IP_ADDRESS> (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/connection.py", line 180, in async_handle
    handler(self.hass, self, schema(msg))
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 595, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 387, in validate_mapping
    cval = cvalue(key_path, value)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/validators.py", line 229, in _run
    return self._exec(self._compiled, value, path)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/validators.py", line 353, in _exec
    v = func(path, v)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 818, in validate_callable
    return schema(data)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 636, in validate_sequence
    cval = validate(index_path, value)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 818, in validate_callable
    return schema(data)
  File "/usr/src/homeassistant/homeassistant/helpers/config_validation.py", line 947, in key_value_validator
    return cast(dict[Hashable, Any], value_schemas[key_value](value))
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 595, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 387, in validate_mapping
    cval = cvalue(key_path, value)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/validators.py", line 229, in _run
    return self._exec(self._compiled, value, path)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/validators.py", line 353, in _exec
    v = func(path, v)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 636, in validate_sequence
    cval = validate(index_path, value)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/validators.py", line 229, in _run
    return self._exec(self._compiled, value, path)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/validators.py", line 353, in _exec
    v = func(path, v)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 818, in validate_callable
    return schema(data)
  File "/usr/src/homeassistant/homeassistant/components/energy/data.py", line 155, in _flow_from_ensure_single_price
    val["entity_energy_price"] is not None
KeyError: 'entity_energy_price'

Can you remember on your old grid consumption whether you had the costs set to None, a fixed price or the value of an entity ?

could you try adding it again but set a fixed unit price

Sure, it seems like the same error occurred.

Logger: homeassistant.components.websocket_api.http.connection
Source: components/energy/data.py:155
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 10 May 2023 at 18:09:56 (7 occurrences)
Last logged: 11:34:59

[2591751592] Error handling message: Unknown error (unknown_error) nigel from <IP_ADDRESS> (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36)
[2579096144] Error handling message: Unknown error (unknown_error) nigel from <IP_ADDRESS> (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36)
[2587615272] Error handling message: Unknown error (unknown_error) nigel from <IP_ADDRESS> (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36)
[2635607128] Error handling message: Unknown error (unknown_error) nigel from <IP_ADDRESS> (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36)
[2566378424] Error handling message: Unknown error (unknown_error) nigel from <IP_ADDRESS> (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/connection.py", line 180, in async_handle
    handler(self.hass, self, schema(msg))
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 595, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 387, in validate_mapping
    cval = cvalue(key_path, value)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/validators.py", line 229, in _run
    return self._exec(self._compiled, value, path)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/validators.py", line 353, in _exec
    v = func(path, v)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 818, in validate_callable
    return schema(data)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 636, in validate_sequence
    cval = validate(index_path, value)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 818, in validate_callable
    return schema(data)
  File "/usr/src/homeassistant/homeassistant/helpers/config_validation.py", line 947, in key_value_validator
    return cast(dict[Hashable, Any], value_schemas[key_value](value))
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 595, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 387, in validate_mapping
    cval = cvalue(key_path, value)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/validators.py", line 229, in _run
    return self._exec(self._compiled, value, path)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/validators.py", line 353, in _exec
    v = func(path, v)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 636, in validate_sequence
    cval = validate(index_path, value)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/validators.py", line 229, in _run
    return self._exec(self._compiled, value, path)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/validators.py", line 353, in _exec
    v = func(path, v)
  File "/usr/local/lib/python3.10/site-packages/voluptuous/schema_builder.py", line 818, in validate_callable
    return schema(data)
  File "/usr/src/homeassistant/homeassistant/components/energy/data.py", line 155, in _flow_from_ensure_single_price
    val["entity_energy_price"] is not None
KeyError: 'entity_energy_price'

can you check Developer Tools, States and do a search for 'cost' - and screenshot any entities that show there (particularly grid_consumption)

This is the only cost that appears.

It looks like your old energy grid consumption was probably using the sensor grid_consumption_sum rather than the utility meter
It doesn't help much as the error suggests some kind of corruption in the energy configuration, do you have a file browser installed? - the energy config is stored in \config.storage\energy - We could do with seeing what is inside that, it'll look something like this -

My inclination is to say lets delete that file and restart HA.
HA will re-create it when it starts, and then you can try adding the energy monitoring back in - starting with grid_daily

Just thinking, if it won't allow deletion (locked) you may need to rename it to energy.bak and then restart.

Yep, I'd added sensor.pv1_daily which didn't work, but sensor.pv_total_sum has been added and it now shows the historical solar data 🥇
@FozzieUK thank you very much for your patience and assistance!

@nigelspowage Great, glad it's sorted 👍
Would you mind just doing one last thing, if you could screenshot what your  \config\ .storage\energy file looks like now - i'm still intrigued how it broke and whether there is something the integration did that caused it, or could have protected against it.

I haven't been watching this closely, but looks like #172 maybe?
Fixed in the beta (and upgrading to the beta will auto-fix it).

@FozzieUK My energy file looks like this now:
{
  "version": 1,
  "minor_version": 1,
  "key": "energy",
  "data": {
    "energy_sources": [
      {
        "type": "grid",
        "flow_from": [
          {
            "stat_energy_from": "sensor.grid_daily",
            "stat_cost": null,
            "entity_energy_price": null,
            "number_energy_price": null
          }
        ],
        "flow_to": [
          {
            "stat_energy_to": "sensor.feedin_daily",
            "stat_compensation": null,
            "entity_energy_price": null,
            "number_energy_price": null
          }
        ],
        "cost_adjustment_day": 0.0
      },
      {
        "type": "battery",
        "stat_energy_from": "sensor.battery_discharge_total",
        "stat_energy_to": "sensor.battery_charge_total"
      },
      {
        "type": "battery",
        "stat_energy_from": "sensor.bat_discharge_daily",
        "stat_energy_to": "sensor.bat_charge_daily"
      },
      {
        "type": "solar",
        "stat_energy_from": "sensor.pv_total_sum",
        "config_entry_solar_forecast": null
      }
    ],
    "device_consumption": []
  }
}```

A few minutes ago I had added both this integrations sensors and the previous ones to the energy dashboard but then quickly noticed I was getting double entries for the new data. I have removed the sensors for this integration from the energy dashboard and the graphs look good now.

Sweet. I was going to ask whether there were any actions we needed to take from this to change the docs. Please makd any improvements you see fit!

@nigelspowage - well spotted, as we made changes to get the original sensors working again, it doesn't need the integration ones - they are duplicated going forwards (but the old ones hold history), however statistics don't take up much storage space. :)
@canton7 - i've updated the migration instructions to make the process smoother ;)


default & get me rqs

closes #16

@rultor merge

Codecov Report
Base: 100.00% // Head: 80.00% // Decreases project coverage by -20.00% :warning:

Coverage data is based on head (1ff8d7a) compared to base (c36cdd2).
Patch coverage: 71.42% of modified lines in pull request are covered.

Additional details and impacted files
@@              Coverage Diff               @@
##              master      #17       +/-   ##
==============================================
- Coverage     100.00%   80.00%   -20.00%
- Complexity         3        7        +4
==============================================
  Files              2        4        +2
  Lines              6       20       +14
==============================================
+ Hits               6       16       +10
- Misses             0        4        +4

Impacted Files
Coverage Δ

.../java/com/l3r8yj/elegramapi/RqDefaultTelegram.java
66.66% <66.66%> (ø)

...in/java/com/l3r8yj/elegramapi/RqGetMeTelegram.java
80.00% <80.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

@rultor merge

@l3r8yj OK, I'll try to merge now. You can check the progress of the merge here


[SPARK-47267][SQL] Add collation support for hash expressions

What changes were proposed in this pull request?
Introduce collation awareness for hash expressions: MD5, SHA2, SHA1, CRC32, MURMUR3, XXHASH64.
Why are the changes needed?
Add collation support for hash expressions in Spark.
Does this PR introduce any user-facing change?
Yes, users should now be able to use collated strings within arguments for hash functions: md5, sha2, sha1, crc32, hash, xxhash64.
How was this patch tested?
E2e sql tests.
Was this patch authored or co-authored using generative AI tooling?
No.

thanks, merging to master!


updated layout for examples.

Blocked the li's
Added colors to LI headers and backgrounds
With a little flexbox to boot

Coverage remained the same at 97.749% when pulling b3270ec87ff83a75d0425fdcdfc528819e90b0d4 on jjmulenex:feature/examples-updates into 71a5d51ddc2c7c3e55a684e9de93c4079b1f6aca on boundlessgeo:master.


Sarama using logging in UTC

This PR enables Sarama to use UTC time on logging.

Good point. Using just TZ is more general otherwise people could set TZ=CEST having canary logging in CEST but then Sarama in UTC. I will close this PR, setting the TZ is just part of the deployment.


[Feature Request] Add Attention nets (GTrXL model in particular)

🚀 Feature Request
This feature request is a duplicate from stable-baselines3 (see https://github.com/DLR-RM/stable-baselines3/issues/177).
The idea is to add the GTrXL model in the contrib repo from the paper Stabilizing Transformers for Reinforcement Learning, as done in RLlib: https://github.com/ray-project/ray/blob/master/rllib/models/torch/attention_net.py.
@araffin has already mentioned that he created it and will make it public (comment).
I wonder if this is still relevant?

I meant the SB3 contrib repo.

Sorry for the misunderstanding.

For GTrXL, are you willing to contribute that algorithm?

I'm not sure yet, I will try to implement it for my experiments first.

Also related: https://github.com/maohangyu/TIT_open_source

@RemiG3 hey, have you started to implement it? Mayba I can give a free hand in it :)

Yes, I have implemented it, but not tested properly. I'm currently having some troubles with my custom environment that I'm trying to solve.
@araffin is it possible to create a new branch for this feature (to share the code)?
If it is possible, I'll clean up the code and push it to this new branch soon.

Yes, I have implemented it, but not tested properly. I'm currently having some troubles with my custom environment that I'm trying to solve.
@araffin is it possible to create a new branch for this feature (to share the code)? If it is possible, I'll clean up the code and push it to this new branch soon.

yes, that's what a fork and pull request are meant for

I have came accross on this, this is quite modular and easy to tune, Transformers-RL, the only backside is that, it has been implemented only to gaussian policy.

Hey, I finally make the PR #176 to share the code.
It should work, but I'm not sure about the performances.
It would be nice if someone could make comparisons with other methods (or RLlib attention net for example).
I won't have time these next days.

RemiG3, Thank you for adding attention net to contrib. I try to do a quick run with the policy and here is the code:
from sb3_contrib.ppo_attention.ppo_attention import AttentionPPO
from sb3_contrib.ppo_attention.policies import MlpAttnPolicy
VE = DummyVecEnv([lambda: gym.make("CartPole-v1")])
model = AttentionPPO(
"MlpAttnPolicy",
VE,
n_steps=12,
learning_rate=0.0003,
verbose=1,
batch_size=1,
ent_coef=0.03,
vf_coef=0.5,
seed=1,
n_epochs=10,
max_grad_norm=1,
gae_lambda=0.95,
gamma=0.99,
tensorboard_log=log_path,
device='cpu',
policy_kwargs = dict(
    net_arch=dict(pi=[64,32],vf=[64,32]),
)

)
this code give some error from GTrXLNet that return all 'na' for features tensor and cause error in logit operation. Is there anything wrong when i use this policy? Also I noticed when use a dummy vector environment and set the batch size larger than one also cause error.
Thanks again.

Thank you, @eric000888, for reporting this (feel free to provide the code you tested as you did in your first edits).
I have updated the branch to fix a bug on the dimension of minibatchs.
But, I still have an exception when batch_size = 1 or n_steps = 1 and I found the same exception for RecurrentPPO.
So, it should now work for batch_size > 1 and n_steps > 1 (as RecurrentPPO).

RemiG3,
Sorry for late response, here is my first post code:
from sb3_contrib.ppo_attention.ppo_attention import AttentionPPO
from sb3_contrib.ppo_attention.policies import MlpAttnPolicy
VE = DummyVecEnv([lambda: gym.make("CartPole-v1")])
model = AttentionPPO(
"MlpAttnPolicy",
VE,
n_steps=240,
learning_rate=0.0003,
verbose=1,
batch_size=12,
ent_coef=0.03,
vf_coef=0.5,
seed=1,
n_epochs=10,
max_grad_norm=1,
gae_lambda=0.95,
gamma=0.99,
tensorboard_log=log_path,
device='cpu',
policy_kwargs = dict(
    net_arch=dict(pi=[64,32],vf=[64,32]),
)

)
First I create a vector environments and then setup the model like LSTM recurrent PPO, then run the model.learn().
I track the code and fund the internal calculation return number is ok at the biggening but after a few loop it start return 'NA' and then stopped.
I saw some other implementation use stacked frame and use sliding window as input format so I'm a little bit confused about what's should be the correct input format. But from your code I think the input should just one records at the time, don't nned to stack the records.
I follow the code and saw you concatenate the tensor of input and memory, but the input format from SB3 is one records and the after the first round of full loop it's become batch number of records and that throw the error as the memory is still just one
tensor instead of batch.

another thing is GtrXL demand more computation power , and PPO is like aiming a moving target, I fund training a GtrXL PPO is a daunting task especially you are using multiple layers.


Git is a silent requirement, but not listed anywhere

Discovered this while installing on a fresh Windows machine with Git not yet installed.
Installation fails silently except in the developer console.
I see in the todo's that there is an item that says, "Possibly merge dependencies into Terminal-Plus?" - I'm assuming this would fix the issue?
Installing package terminal-plus failed Error: Installing<EMAIL_ADDRESS>failed.
    at C:\Users\Socks\AppData\Local\atom\app-1.2.4\resources\app.asar\node_modules\settings-view\lib\package-manager.js:457:21
    at exit (C:\Users\Socks\AppData\Local\atom\app-1.2.4\resources\app.asar\node_modules\settings-view\lib\package-manager.js:73:16)
    at triggerExitCallback (C:\Users\Socks\AppData\Local\atom\app-1.2.4\resources\app.asar\src\buffered-process.js:213:47)
    at ChildProcess.<anonymous> (C:\Users\Socks\AppData\Local\atom\app-1.2.4\resources\app.asar\src\buffered-process.js:235:18)
    at emitTwo (events.js:87:13)
    at ChildProcess.emit (events.js:172:7)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12) Failed to install terminal-plus because Git was not found.

The terminal-plus package has module dependencies that cannot be installed without Git.

You need to install Git and add it to your path environment variable in order to install this package.

Cheers, and thanks for un-dullifying the Windows terminal experience!

Yes, the Terminal-Plus installation relies on the system having git installed for a git clone of the dependency repositories.
That item in the todo list is me considering bringing the dependencies into this repository to fix this issue. However, there are some unwanted side effects with that decision which prevent me from doing that. Either way thank you for pointing out that this silently fails without git. I'll have to think of some other way to fix this. You are welcome, thank you for using Terminal-Plus.


Technical Support Required - FxPortal Implementation Fail with error ‘Leaf index is too big’ (Goerli/Mumbai)

Hello! I'm the Lead Developer working on integrating GoldHunt Game (thegame.gold) with the Polygon network. I intend to build a custom implementation of the FxPortal Root & Child Contracts (https://github.com/fx-portal/contracts) to enable our community to move tokens between Ethereum <-> Polygon.
I need technical support on implementing this bridge on the Test networks (Goerli, Mumbai).
The problem: when I call function receiveMessage(bytes memory inputData) public virtual on my implementation of the Root contract, I am getting a Fail with error 'Leaf index is too big' error on the transaction.
Failed Transactions (same error):

https://goerli.etherscan.io/tx/0x8b19cb0f540d5f6fba1ff19c5c64bb99ba15ee0cc33f1f24859fd26db18b9872

https://goerli.etherscan.io/tx/0x605f6c4ed73547d956c5635aae23e5b1c54f560e2f7e6e57bfed14827c0dcd6c

https://goerli.etherscan.io/tx/0x8906d30b22d7c49366a8641ed5613654c5ab280f897a16bdc97d3c38ea019260

I generate the burn proof as such:

const execute = async () => {

  const posClient = await getPOSClient();

  const proof = await posClient.exitUtil.buildPayloadForExit(

    "0x76bd13760e3dbea702450f8741b405bf528be0078570d94c405a9183a8911ef9", // Withdraw transaction hash

    "0x8c5261668696ce22758910d05bab8f186d6eb247ceac2af2e82c7dc17669b036"  // MESSAGE_SENT_EVENT_SIG: Do not change this

  )

  console.log(" ----- BURN PROOF -----")

  console.log(proof)

}

Where https://mumbai.polygonscan.com/tx/0x76bd13760e3dbea702450f8741b405bf528be0078570d94c405a9183a8911ef9 is the Withdraw transaction I submitted on the Mumbai network.
My implementation of the FxMintableERC20RootTunnel is deployed at https://goerli.etherscan.io/address/0xC0Af0b1a10F2417D62EE044BC99dB1b40b4DC82e - you'll notice a number of failed transactions all throwing the same error.
My implementation of the FxMintableERC20ChildTunnel is deployed at https://mumbai.polygonscan.com/address/0x38530B465C2937b6f3fB7559a796156D2a0A9f02 - I submitted the Withdraw transaction on Mumbai to this contract (see above) over 24 hours ago.
I am attempting to send GGold (ERC20) back and forth across my tunnel, and am unable to resolve the last step to receive the funds back from the Tunnel on Goerli due to the error above.
GGold (Goerli) - https://goerli.etherscan.io/address/0xbdCDF80D26DDA1ae2F9C0003995d160F4893F20A
pGGold (Polygon) - https://mumbai.polygonscan.com/address/0xC440ddBA14e585fdF395bD74B85797e41fd8317c
I can see that the error is ultimately being thrown by the Merkle library defined as such:
pragma solidity ^0.8.0;

library Merkle {
    function checkMembership(
        bytes32 leaf,
        uint256 index,
        bytes32 rootHash,
        bytes memory proof
    ) internal pure returns (bool) {
        require(proof.length % 32 == 0, "Invalid proof length");
        uint256 proofHeight = proof.length / 32;
        // Proof of size n means, height of the tree is n+1.
        // In a tree of height n+1, max #leafs possible is 2 ^ n
        require(index < 2**proofHeight, "Leaf index is too big");

        bytes32 proofElement;
        bytes32 computedHash = leaf;
        for (uint256 i = 32; i <= proof.length; i += 32) {
            assembly {
                proofElement := mload(add(proof, i))
            }

            if (index % 2 == 0) {
                computedHash = keccak256(abi.encodePacked(computedHash, proofElement));
            } else {
                computedHash = keccak256(abi.encodePacked(proofElement, computedHash));
            }

            index = index / 2;
        }
        return computedHash == rootHash;
    }
}

but this entire section of code was defined as part of the FxBaseRootTunnel.sol file defined in the fx-portals repo which I am using an exact copy of so I'm not sure how to debug this issue.

Please advise on how to resolve this issue - I've already read through https://docs.polygon.technology/docs/develop/l1-l2-communication/fx-portal/ and this issue is not covered and I am unable to find any other sources online about how to resolve it.
Any help you can provide would be greatly appreciated, as we're excited about being able to onboard thousands of new users to the Polygon network once we're able to build out this infrastructure!

Hi @QEDK is this something that you can assist with? I'm unable to find any information about how to resolve a Fail with error 'Leaf Index is too big' online and I believe I have followed all of the steps for deposits and withdraws correctly, described here: https://docs.polygon.technology/docs/develop/l1-l2-communication/fx-portal/

Hi @jdkanani is this something that you can assist with? I'm unable to find any information about how to resolve a Fail with error 'Leaf Index is too big' online and I believe I have followed all of the steps for deposits and withdraws correctly, described here: https://docs.polygon.technology/docs/develop/l1-l2-communication/fx-portal/

I'm posting here because I've already taken the following actions to get support for this issue and have not received a response:
I've taken the following steps to get support and have not received a response yet:
Multiple tickets submitted to: https://polygon.technology/contact-us/
Forum Post: https://forum.polygon.technology/t/technical-support-required-fxportal-implementation-fail-with-error-leaf-index-is-too-big/1394
This Support Ticket: https://support.polygon.technology/support/tickets/31130
Github Issue on FX-Portals Repo: https://github.com/fx-portal/contracts/issues/28
Multiple Messages in Polygon Discord Support Channel: https://discord.com/channels/635865020172861441/717384861647896616/935176155298930689

Hi @jdkanani - I wanted to provide some more information about this issue. On Mumbai, you can see I submit a withdraw/burn transaction here: https://mumbai.polygonscan.com/tx/0xae1c8793cc900f22676b348802d020790057ce9fb82cd9d7404ebd3c85b5fea0
which resolves successfully. I then move on to the next step, which is generating the burnproof.
I define a utils.js file that looks like such:
const bn = require('bn.js')
const HDWalletProvider = require('@truffle/hdwallet-provider')
const config = require('./config')
const { POSClient, setProofApi, use } = require('@maticnetwork/maticjs')
const SCALING_FACTOR = new bn(10).pow(new bn(18))
const { Web3ClientPlugin } = require("@maticnetwork/maticjs-web3");

use(Web3ClientPlugin);

if (config.proofApi) {
  setProofApi(config.proofApi);
}

const privateKey = config.user1.privateKey
const userAddress = config.user1.address

const getPOSClient = (network = 'testnet', version = 'mumbai') => {
  const posClient = new POSClient()
  return posClient.init({
    log: true,
    network: network,
    version: version,
    child: {
      provider: new HDWalletProvider(privateKey, config.child.rpc),
      defaultConfig: {
        from: userAddress
      }
    },
    parent: {
      provider: new HDWalletProvider(privateKey, config.parent.rpc),
      defaultConfig: {
        from: userAddress
      }
    }
  });
}

module.exports = {
  SCALING_FACTOR,
  getPOSClient: getPOSClient,
  child: config.child,
  plasma: config.plasma,
  pos: config.pos,
  from: config.user1.address,
  privateKey: config.user1.privateKey,
  proofApi: config.proofApi
}

and a generate_burn_proof.js file that looks like such:
const { getPOSClient, from } = require('../../utils');

const execute = async () => {
  const posClient = await getPOSClient();
  const proof = await posClient.exitUtil.buildPayloadForExit(
    "0xae1c8793cc900f22676b348802d020790057ce9fb82cd9d7404ebd3c85b5fea0", // Withdraw transaction hash
    "0x8c5261668696ce22758910d05bab8f186d6eb247ceac2af2e82c7dc17669b036"  // MESSAGE_SENT_EVENT_SIG: Do not change this
  )
  console.log(" ----- BURN PROOF -----")
  console.log(proof)
}
execute().then(() => {
}).catch(err => {
  console.error("err", err);
}).finally(_ => {
  process.exit(0);
})

You'll note that I am submitting the 0xae1c8793cc900f22676b348802d020790057ce9fb82cd9d7404ebd3c85b5fea0 transaction hash here to generate the proof. By running generate_burn_proof.js, I'm able to successfully generate the burnproof for the transaction.
I submit this burnproof as input to my Root contract's receiveMessage message function but it fails due to Fail with error 'Leaf index is too big' - Failed Transaction: https://goerli.etherscan.io/tx/0xc5724512db14a1de7631441fc2e26e3da2439862515a2e15db6ef32761ab7fe6

Hi @c194 , can you try with this payload here: https://apis.matic.network/api/v1/mumbai/exit-payload/0xae1c8793cc900f22676b348802d020790057ce9fb82cd9d7404ebd3c85b5fea0?eventSignature=0x8c5261668696ce22758910d05bab8f186d6eb247ceac2af2e82c7dc17669b036

Hi @Raneet10 Thanks for the quick response, and that payload did work!
Successful Transaction: https://goerli.etherscan.io/tx/0x628ce213f9523db782d49e6f882cf2bc2a1bd7d5bce2f746edbebe20d94437a8
I am now wondering if the @maticnetwork/maticjs library needs to be updated to include whatever changes that you made.
I define the following configuration for POSClient:
const { POSClient, setProofApi, use } = require('@maticnetwork/maticjs')
// Irrelevant code excluded from this snippet

setProofApi('https://apis.matic.network/api/v1/');

const privateKey = config.user1.privateKey
const userAddress = config.user1.address

const getPOSClient = (network = 'testnet', version = 'mumbai') => {
  const posClient = new POSClient()
  return posClient.init({
    log: true,
    network: network,
    version: version,
    child: {
      provider: new HDWalletProvider(privateKey, config.child.rpc),
      defaultConfig: {
        from: userAddress
      }
    },
    parent: {
      provider: new HDWalletProvider(privateKey, config.parent.rpc),
      defaultConfig: {
        from: userAddress
      }
    }
  });
}

and generate the burnproof as such:
const execute = async () => {
  const posClient = await getPOSClient();
  const proof = await posClient.exitUtil.buildPayloadForExit(
    "0xae1c8793cc900f22676b348802d020790057ce9fb82cd9d7404ebd3c85b5fea0", // Withdraw transaction hash
    "0x8c5261668696ce22758910d05bab8f186d6eb247ceac2af2e82c7dc17669b036"  // MESSAGE_SENT_EVENT_SIG: Do not change this
  )
  console.log(" ----- BURN PROOF -----")
  console.log(proof)
}

Where I was sure to reference the same API endpoint as you sent over. However, the library produces a slightly different burnproof than the web request you sent over.
Web Request Burnproof:                    0xf90b7b8428adf890b90...
'@maticnetwork/maticjs' Burnproof: 0xf90a798428adf890808...
I can use the GET web requests to continue with my development work but wanted to make sure I brought this discrepancy to your attention while you are working on the issue.
Second Example
Withdraw Transaction - https://mumbai.polygonscan.com/tx/0xba49d07f88c5bbd00a60428333a374f4fcc58dc60e231ab74986f39886914729
Receive Message Failure (using matijs burnproof 0xf909588428d15f10808401...: https://goerli.etherscan.io/tx/0x2746142e2687b840b3ee295814b7ae99a026712e11911013c55093209fea43d1
Receive Message Success (using GET request burnproof 0xf90a5a8428d15f10b...:
https://apis.matic.network/api/v1/mumbai/exit-payload/0xba49d07f88c5bbd00a60428333a374f4fcc58dc60e231ab74986f39886914729?eventSignature=0x8c5261668696ce22758910d05bab8f186d6eb247ceac2af2e82c7dc17669b036
https://goerli.etherscan.io/tx/0x6e0cde361d4a47d8df3c9c8c5840f1227be44baff7fd9f79c43819971e6df349
Thanks again for your timely response! I really appreciate your support on this issue.

Hi @Raneet10 Thanks for the quick response, and that payload did work!
Successful Transaction: https://goerli.etherscan.io/tx/0x628ce213f9523db782d49e6f882cf2bc2a1bd7d5bce2f746edbebe20d94437a8
I am now wondering if the @maticnetwork/maticjs library needs to be updated to include whatever changes that you made.
I define the following configuration for POSClient:
const { POSClient, setProofApi, use } = require('@maticnetwork/maticjs')
// Irrelevant code excluded from this snippet

setProofApi('https://apis.matic.network/api/v1/');

const privateKey = config.user1.privateKey
const userAddress = config.user1.address

const getPOSClient = (network = 'testnet', version = 'mumbai') => {
  const posClient = new POSClient()
  return posClient.init({
    log: true,
    network: network,
    version: version,
    child: {
      provider: new HDWalletProvider(privateKey, config.child.rpc),
      defaultConfig: {
        from: userAddress
      }
    },
    parent: {
      provider: new HDWalletProvider(privateKey, config.parent.rpc),
      defaultConfig: {
        from: userAddress
      }
    }
  });
}

and generate the burnproof as such:
const execute = async () => {
  const posClient = await getPOSClient();
  const proof = await posClient.exitUtil.buildPayloadForExit(
    "0xae1c8793cc900f22676b348802d020790057ce9fb82cd9d7404ebd3c85b5fea0", // Withdraw transaction hash
    "0x8c5261668696ce22758910d05bab8f186d6eb247ceac2af2e82c7dc17669b036"  // MESSAGE_SENT_EVENT_SIG: Do not change this
  )
  console.log(" ----- BURN PROOF -----")
  console.log(proof)
}

Where I was sure to reference the same API endpoint as you sent over. However, the library produces a slightly different burnproof than the web request you sent over.
Web Request Burnproof: 0xf90b7b8428adf890b90... @maticnetwork/maticjs Burnproof: 0xf90a798428adf890808...
I can use the GET web requests to continue with my development work but wanted to make sure I brought this discrepancy to your attention while you are working on the issue.
Second Example Withdraw Transaction - https://mumbai.polygonscan.com/tx/0xba49d07f88c5bbd00a60428333a374f4fcc58dc60e231ab74986f39886914729 Receive Message Failure (using matijs burnproof 0xf909588428d15f10808401...: https://goerli.etherscan.io/tx/0x2746142e2687b840b3ee295814b7ae99a026712e11911013c55093209fea43d1 Receive Message Success (using GET request burnproof 0xf90a5a8428d15f10b...: https://apis.matic.network/api/v1/mumbai/exit-payload/0xba49d07f88c5bbd00a60428333a374f4fcc58dc60e231ab74986f39886914729?eventSignature=0x8c5261668696ce22758910d05bab8f186d6eb247ceac2af2e82c7dc17669b036 https://goerli.etherscan.io/tx/0x6e0cde361d4a47d8df3c9c8c5840f1227be44baff7fd9f79c43819971e6df349
Thanks again for your timely response! I really appreciate your support on this issue.

The likely reason is that the matic.js version used to generate the proof is older. If the issue persists on the latest version, please open a issue at https://github.com/maticnetwork/matic.js/issues with the details you provided above.


New editor setting: Window Placement -> Settings to Use

Fixes #76920
The new option is "run/window_placement/settings_to_use".

If set as Editor Settings, the editor injects the window placement CLI arguments as normal.
Otherwise, Project Settings, the editor skips the injection of window placement CLI arguments.

Closed in favor of https://github.com/godotengine/godot-proposals/issues/7448.
A bool setting makes more sense and is enough for this feature.


PR for issue #5441 "error in cell run by plot action goes to wrong location"

The same problem was in "inractive" , I fixed it.
Inractive works fine now, but not table/plot API.
Now kernel not get stuck at all casees.
But there is the problem with displaying something in console (table/plot API).
The problem is "parrent message": we need to pass correct parent message, to get ability print anything to the jupyter console.
I afraid it is not simple task.
Need to check what message passed in "interactive" and pass same in plot/table API.

👍  cool does not get stuck.
what's the difference between errors for tables and plots though?
i am seeing the ones for tables correctly but not plots.


Consider using a context cancelation in the peer.Writer/Reader (instead of ad-hoc exitChan)

This extends to the QuitChan returned by the keepAliveLoop function

Same thing for the peer.ReadLoop and peer.Serve called by OnAccept (which in turn should also accept a context.Context parameter)


err_name_not_resolved flutter

The problem appears when you open a web page with WebView
ERRO: err_name_not_resolved
Flutter is updated
The Code is revised
Dependencies are applied
WebView Works only on the emulator, but does not run on the physical device

Hi @Vitttu,
Please provide flutter doctor -v and a minimal complete reproducible code sample that shows the issue you are facing.
Also mention the device on which you are trying and not working.

Hi @Vitttu,
Please provide flutter doctor -v and a minimal complete reproducible code sample that shows the issue you are facing.
Also mention the device on which you are trying and not working.

Hello, @darshankawar ! We are facing the same problem with WebView.
In store reviews (Google Play and Apple App Store) the error of err_name_not_resolved was reported, but in debug, the app runs smoothly.
Then I put the Flutter Doctor command as requested.
[√] Flutter (Channel stable, v1.17.5, on Microsoft Windows [versÃ£o 10.0.19041.450], locale pt-BR)
    • Flutter version 1.17.5 at C:\Dev\FlutterSDK
    • Framework revision 8af6b2f038 (6 weeks ago), 2020-06-30 12:53:55 -0700
    • Engine revision ee76268252
    • Dart version 2.8.4

[√] Android toolchain - develop for Android devices (Android SDK version 30.0.0)
    • Android SDK at C:\Dev\AndroidSDK
    • Platform android-30, build-tools 30.0.0
    • ANDROID_HOME = C:\Dev\AndroidSDK
    • ANDROID_SDK_ROOT = C:\Dev\AndroidSDK
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
    • All Android licenses accepted.

[√] Android Studio (version 4.0)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin version 46.0.2
    • Dart plugin version 193.7361
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)

[√] VS Code (version 1.47.3)
    • VS Code at D:\Usuários\mathi\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.13.2

[√] Connected device (1 available)
    • SM G935F • <IP_ADDRESS>:5555 • android-arm64 • Android 8.0.0 (API 26)

• No issues found!


Getting error on click of input, keyboard avoid issue

Description
TypeError: responder.scrollResponderScrollTo is not a function.
CodeSandbox/Snack link
https://test
Steps to reproduce
Run this app on IOS
Click any input box it will throw error saying :
typeError: responder.scrollResponderScrollTo is not a function. (In 'responder.scrollResponderScrollTo({
            x: x,
            y: y,
            animated: animated
          })', 'responder.scrollResponderScrollTo' is undefined)**

I tried by adding this but still not working
"native-base/@codler/react-native-keyboard-aware-scroll-view": "2.0.0"
NativeBase Version
^2.13.12
Platform

[ ] Android
[ ] CRA
[ ] Expo
[X] iOS
[ ] Next

Other Platform
No response
Additional Information
No response

+1

It seems like currently you can't use the latest React Native with NativeBase v2 because it depends on @codler/react-native-keyboard-aware-scroll-view" (btw, https://github.com/APSL/react-native-keyboard-aware-scroll-view seems to be working fine at the moment if used from master)
And updating NativeBase from v2 to v3 takes way too much effort due to breaking changes often times for smaller apps its easier to remove NativeBase dependency


Question about NM_015068.3

I'm puzzled and need help understanding. Consider NM_015068.3. I think that the exons given in the CDOT JSON skip one base that needs to be in the CDS as it's read double (slippage). What do you think?
Here is how it looks like in cdot-0.2.24.refseq.grch37.json.
# jq '.transcripts | to_entries[] | (select(.key == "NM_015068.3"))' cdot-0.2.24.refseq.grch37.json
{
  "key": "NM_015068.3",
  "value": {
    "biotype": [
      "mRNA"
    ],
    "gene_name": "PEG10",
    "gene_version": "23089",
    "genome_builds": {
      "GRCh37": {
        "cds_end": 94294994,
        "cds_start": 94292868,
        "contig": "NC_000007.13",
        "exons": [
          [
            94285636,
            94285892,
            0,
            1,
            256,
            null
          ],
          [
            94292645,
            94299007,
            1,
            257,
            6618,
            null
          ]
        ],
        "note": "protein translation is dependent on -1 ribosomal frameshift%3B isoform 1 is encoded by transcript variant 1",
        "strand": "+",
        "url": "https://ftp.ncbi.nlm.nih.gov/genomes/all/annotation_releases/9606/105.20220307/GCF_000001405.25_GRCh37.p13/GCF_000001405.25_GRCh37.p13_genomic.gff.gz"
      }
    },
    "hgnc": "14005",
    "id": "NM_015068.3",
    "protein": "NP_055883.2",
    "start_codon": 479,
    "stop_codon": 2605
  }
}

NCBI Gene says
[...]
     CDS             join(480..1436,1436..2605)
[...]

The original GFF3 says
 egrep -w 'ID=gene14272|Parent=rna16954|NM_015068.3' ref_GRCh37.p10_top_level.gff3
NC_000007.13    RefSeq  gene    94285637        94299007        .       +       .       ID=gene14272;Name=PEG10;Dbxref=GeneID:23089,HGNC:14005,MIM:609810;description=paternally expressed 10;gbkey=Gene;gene=PEG10;gene_synonym=EDR,HB-1,Mar2,Mart2,MEF3L,RGAG3
NC_000007.13    RefSeq  mRNA    94285637        94299007        .       +       .       ID=rna16953;Name=NM_015068.3;Parent=gene14272;Dbxref=GeneID:23089,Genbank:NM_015068.3,HGNC:14005,MIM:609810;gbkey=mRNA;gene=PEG10;product=paternally expressed 10%2C transcript variant 1;transcript_id=NM_015068.3
NC_000007.13    RefSeq  exon    94285682        94285903        .       +       .       ID=id161180;Parent=rna16954;Dbxref=GeneID:23089,Genbank:NM_001172437.1,HGNC:14005,MIM:609810;gbkey=mRNA;gene=PEG10;product=paternally expressed 10%2C transcript variant 2;transcript_id=NM_001172437.1
NC_000007.13    RefSeq  exon    94292646        94299007        .       +       .       ID=id161181;Parent=rna16954;Dbxref=GeneID:23089,Genbank:NM_001172437.1,HGNC:14005,MIM:609810;gbkey=mRNA;gene=PEG10;product=paternally expressed 10%2C transcript variant 2;transcript_id=NM_001172437.1
NC_000007.13    RefSeq  CDS     94285899        94285903        .       +       0       ID=cds13063;Name=NP_001165908.1;Parent=rna16954;Note=isoform 3 is encoded by transcript variant 2;Dbxref=GeneID:23089,Genbank:NP_001165908.1,HGNC:14005,MIM:609810;exception=ribosomal slippage;gbkey=CDS;product=retrotransposon-derived protein PEG10 isoform 3;protein_id=NP_001165908.1
NC_000007.13    RefSeq  CDS     94292646        94293825        .       +       1       ID=cds13063;Name=NP_001165908.1;Parent=rna16954;Note=isoform 3 is encoded by transcript variant 2;Dbxref=GeneID:23089,Genbank:NP_001165908.1,HGNC:14005,MIM:609810;exception=ribosomal slippage;gbkey=CDS;product=retrotransposon-derived protein PEG10 isoform 3;protein_id=NP_001165908.1
NC_000007.13    RefSeq  CDS     94293825        94294994        .       +       0       ID=cds13063;Name=NP_001165908.1;Parent=rna16954;Note=isoform 3 is encoded by transcript variant 2;Dbxref=GeneID:23089,Genbank:NP_001165908.1,HGNC:14005,MIM:609810;exception=ribosomal slippage;gbkey=CDS;product=retrotransposon-derived protein PEG10 isoform 3;protein_id=NP_001165908.1

Also
NM_001172437.2
NM_001184961.1

The following transcripts are protein coding but don't have a CDS length of a multiple of 3. Some are also marked as slippage. However they don't have the overlap in CDS exons as far as I could see.
NM_004152.3
NM_001301020.1
NM_002537.3
NM_001301302.1

Here is the UTA output for NM_015068.3.
uta=> select * from uta_20210129b.tx_def_summary_v where tx_def_summary_v.tx_ac = 'NM_015068.3';
-[ RECORD 1 ]-------+---------------------------------
exon_set_id         | 349795
tx_ac               | NM_015068.3
alt_ac              | NM_015068.3
alt_aln_method      | transcript
alt_strand          | 1
hgnc                | PEG10
cds_md5             | b87df1644c80a908da9b7d8a30468084
es_fingerprint      | c0183212a58ec4219c045248df36b55b
cds_es_fp           | 95d8b77d6220eb8d205b5eb927ec114c
cds_exon_lengths_fp | 3b92d18aa7a6176dd37d372bc2f1eb71
n_exons             | 2
se_i                | 0,256;256,6618
cds_se_i            | 479,2605
starts_i            | {0,256}
ends_i              | {256,6618}
lengths             | {256,6362}
cds_start_i         | 479
cds_end_i           | 2605
cds_start_exon      | 1
cds_end_exon        | 1

and some more
uta=> select * from uta_20210129b.tx_def_summary_v where tx_def_summary_v.tx_ac = 'NM_004152.3';
-[ RECORD 1 ]-------+----------------------------------------
exon_set_id         | 349259
tx_ac               | NM_004152.3
alt_ac              | NM_004152.3
alt_aln_method      | transcript
alt_strand          | 1
hgnc                | OAZ1
cds_md5             | f5cafd85bcfead7e657f0c554125a2d2
es_fingerprint      | 35edd88f9b7d944cbda18c665b52aea7
cds_es_fp           | a6994ac96b3ecd845ff44a94fed47c18
cds_exon_lengths_fp | 85b85525c647eae35eb4deb4c26b2776
n_exons             | 5
se_i                | 0,259;259,405;405,578;578,664;664,1181
cds_se_i            | 113,259;259,405;405,578;578,664;664,801
starts_i            | {0,259,405,578,664}
ends_i              | {259,405,578,664,1181}
lengths             | {259,146,173,86,517}
cds_start_i         | 113
cds_end_i           | 801
cds_start_exon      | 0
cds_end_exon        | 4

uta=> select * from uta_20210129b.tx_def_summary_v where tx_def_summary_v.tx_ac = 'NM_001301020.1';
-[ RECORD 1 ]-------+----------------------------------------
exon_set_id         | 347618
tx_ac               | NM_001301020.1
alt_ac              | NM_001301020.1
alt_aln_method      | transcript
alt_strand          | 1
hgnc                | OAZ1
cds_md5             | 4129a9f2c8b8a5df9c8fffd30208024e
es_fingerprint      | 5f4f21d019bb1d152ef32baf9e5ec92f
cds_es_fp           | 5eab802d6b244bb49be259b5fabc424a
cds_exon_lengths_fp | 88059a870673c88c567d3f2d674fc403
n_exons             | 5
se_i                | 0,259;259,399;399,572;572,658;658,1175
cds_se_i            | 113,259;259,399;399,572;572,658;658,795
starts_i            | {0,259,399,572,658}
ends_i              | {259,399,572,658,1175}
lengths             | {259,140,173,86,517}
cds_start_i         | 113
cds_end_i           | 795
cds_start_exon      | 0
cds_end_exon        | 4

uta=> select * from uta_20210129b.tx_def_summary_v where tx_def_summary_v.tx_ac = 'NM_002537.3';
-[ RECORD 1 ]-------+----------------------------------------
exon_set_id         | 349097
tx_ac               | NM_002537.3
alt_ac              | NM_002537.3
alt_aln_method      | transcript
alt_strand          | 1
hgnc                | OAZ2
cds_md5             | e9892dbf85b12e83cfce811c6f2d79f0
es_fingerprint      | 759d543e96a31ee1bdd0e169c690d875
cds_es_fp           | 4c7bf02afed7f1a3fffaf1281a327e23
cds_exon_lengths_fp | c41e27ecfeae67b9debdffb74dea4743
n_exons             | 5
se_i                | 0,253;253,411;411,587;587,673;673,1934
cds_se_i            | 233,253;253,411;411,587;587,673;673,804
starts_i            | {0,253,411,587,673}
ends_i              | {253,411,587,673,1934}
lengths             | {253,158,176,86,1261}
cds_start_i         | 233
cds_end_i           | 804
cds_start_exon      | 0
cds_end_exon        | 4

uta=> select * from uta_20210129b.tx_def_summary_v where tx_def_summary_v.tx_ac = 'NM_001301302.1';
-[ RECORD 1 ]-------+----------------------------------------
exon_set_id         | 347760
tx_ac               | NM_001301302.1
alt_ac              | NM_001301302.1
alt_aln_method      | transcript
alt_strand          | 1
hgnc                | OAZ2
cds_md5             | b37122a9fdad16364189a05ccdec2b12
es_fingerprint      | 741e13a02e0991e512c61ceb6d652c92
cds_es_fp           | b2620ac572e16cb98b51298f39b9fe96
cds_exon_lengths_fp | 9fe058e4b9ff38d9db4bea4f13152963
n_exons             | 5
se_i                | 0,253;253,408;408,584;584,670;670,1931
cds_se_i            | 233,253;253,408;408,584;584,670;670,801
starts_i            | {0,253,408,584,670}
ends_i              | {253,408,584,670,1931}
lengths             | {253,155,176,86,1261}
cds_start_i         | 233
cds_end_i           | 801
cds_start_exon      | 0
cds_end_exon        | 4

Yes, this is a problem and the correct thing to do would be adjust for it. The GFF3 spec has Column 8: "phase" - which I think you need to use for correct calculations.
I haven't looked into it deeply, but I don't think you can just handle it by altering the start/end coordinates of a CDS (otherwise they would do that in the GFF) but rather you have to take into account this phasing in the HGVS conversion code.
Given that our transcript from GFF3 is the same as the UTA transcript, it looks like they have the same trouble. They probably don't have a way to adjust for the phasing here.
So - I think you need to raise it as a BioCommons HGVS issue  - they will probably have to alter the UTA schema as well.
CDOT
We can add the GFF3 phasing score of 0/1/2 to our cdot transcripts, in anticipation for it being fixed in HGVS, so that historical data can be used for a data provider....
The most obvious place would be adding it to a field in "exons" but this would break existing code:
for (alt_start_i, alt_end_i, exon_id, cds_start_i, cds_end_i, gap) in exons:

So we'd have to bump the major version of the data version (note to future self: use an asterisk to collect remaining fields so you can add ones in future, or avoid arrays with positional info and use dicts with keys)
A non-breaking way to add this would be to have a new array that is parallel with exons, eg "phasing" that contains the offsets

I think we should add a "phasing" array to our JSON (if it's not all zeros) so it doesn't break backwards compatibility with exon fields.
Until biocommons add support this is just making our files bigger for no benefit, but at least we'll be correct and maybe tools from other languages etc can use it.
Then, if biocommons HGVS adds support, we can modify our data provider classes to get this into biocommons HGVS
I also made an issue about breaking changes (so we can learn from our mistakes here - about using fixed length arrays where we may want to add info in the future)

I don't get why phase is 0 in GRCh38?
$ zgrep "NM_015068.3" GCF_000001405.40_GRCh38.p14_genomic.110.gff.gz
NC_000007.14	BestRefSeq	mRNA	94656325	94669695	.	+	.	ID=rna-NM_015068.3;Parent=gene-PEG10;Dbxref=GeneID:23089,Genbank:NM_015068.3,HGNC:HGNC:14005,MIM:609810;Name=NM_015068.3;gbkey=mRNA;gene=PEG10;product=paternally expressed 10%2C transcript variant 1;transcript_id=NM_015068.3
NC_000007.14	BestRefSeq	exon	94656325	94656580	.	+	.	ID=exon-NM_015068.3-1;Parent=rna-NM_015068.3;Dbxref=GeneID:23089,Genbank:NM_015068.3,HGNC:HGNC:14005,MIM:609810;gbkey=mRNA;gene=PEG10;product=paternally expressed 10%2C transcript variant 1;transcript_id=NM_015068.3
NC_000007.14	BestRefSeq	exon	94663334	94669695	.	+	.	ID=exon-NM_015068.3-2;Parent=rna-NM_015068.3;Dbxref=GeneID:23089,Genbank:NM_015068.3,HGNC:HGNC:14005,MIM:609810;gbkey=mRNA;gene=PEG10;product=paternally expressed 10%2C transcript variant 1;transcript_id=NM_015068.3
NC_000007.14	BestRefSeq	CDS	94663557	94664513	.	+	0	ID=cds-NP_055883.2;Parent=rna-NM_015068.3;Dbxref=GeneID:23089,Genbank:NP_055883.2,HGNC:HGNC:14005,MIM:609810;Name=NP_055883.2;Note=protein translation is dependent on -1 ribosomal frameshift%3B isoform 1 is encoded by transcript variant 1;exception=ribosomal slippage;gbkey=CDS;gene=PEG10;product=retrotransposon-derived protein PEG10 isoform 1;protein_id=NP_055883.2
NC_000007.14	BestRefSeq	CDS	94664513	94665682	.	+	0	ID=cds-NP_055883.2;Parent=rna-NM_015068.3;Dbxref=GeneID:23089,Genbank:NP_055883.2,HGNC:HGNC:14005,MIM:609810;Name=NP_055883.2;Note=protein translation is dependent on -1 ribosomal frameshift%3B isoform 1 is encoded by transcript variant 1;exception=ribosomal slippage;gbkey=CDS;gene=PEG10;product=retrotransposon-derived protein PEG10 isoform 1;protein_id=NP_055883.2


Update to <IP_ADDRESS> and add CUDA 12 support.

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

This PR updates cuDNN to <IP_ADDRESS> with support for CUDA 11 and CUDA 12.
Fixes https://github.com/conda-forge/cudnn-feedstock/issues/54

@conda-forge-admin, please rerender

cc: @jakirkham @leofang @adibbley @robertmaynard for notice/review. I think CI should pass this round (still running).

(and thanks in advance to @kkraus14 as package maintainer for a review, as well) 🙏

Q: Would it possible that we try moving ppc64le back to Travis? It is slightly stabler than before, so if it can complete aarch64 in less than 15 mins, it might work for ppc64le too. Not a blocking change, though, and I am no longer the maintainer here 🙂

I also, would like to no longer be a recipe maintainer for this package. Is there any chance someone else would like to be added? cc @adibbley

We were going to propose adding some folks here. Would suggest:

@adibbley
@bdice
@jakirkham
@vyasr

@conda-forge-admin, please re-render

Thanks all! 🙏
Keith, could you please give this another look when you have a chance?

Ideally this cudnn package should probably be split into a runtime and dev package, where the dev package then makes sense to depend on cudart-dev

Ideally this cudnn package should probably be split into a runtime and dev package, where the dev package then makes sense to depend on cudart-dev

I think that is a good long-term plan. I'm hesitant to introduce that change right now, but the impact of that change can be surveyed and we can move in that direction over time. I'll file an issue.

Should add this same discussion came up offline previously and we decided to punt for now

Thanks all! 🙏
Going ahead and merging to get these packages out. Let's follow up on anything else separately 🙂


Add dead server policy control and metric. Improve messages.

(cherry picked from commit 01b91f348a919fa69ccd9727e39aae11bfca2a38)

This cherry-picks #7757 into 9.1.x.  These were the conflicts:

    both modified:   mgmt/RecordsConfig.cc
    both modified:   proxy/http/HttpConfig.cc
    both modified:   proxy/http/HttpTransact.cc

One of the changes made here was to convert the dead server message to debug.

[approve ci autest]

[approve ci autest]

[approve ci autest]

[approve ci autest]


Update testing docs to match React Query V3

Updated ReactQueryCacheProvider with QueryClientProvider as current testing documentation does not work when using React Query V3

:tada: This PR is included in version 3.5.12 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 3.5.12 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


TlsConnector adds certificate chain from PKCS12 identity in reverse order

Using native-tls 0.2.1 on Linux (openssl).
When building TlsConnector with pkcs12 identity which contains 3 certificates: [leaf, intermediate, root] the intermediate and root certificates are sent to the server peer in reverse order [leaf, root, intermediate] resulting in handshake errors (in particular the javax.net.ssl.SSLException: subject/issuer name chaining check failed).

What order are the certificates stored in the PKCS12 file?

#111 should fix this - could you try it out to confirm?

I think it's only implemented for TlsAcceptor in this PR, should be also changed for TlsConnector. When I change it manually it works, yes.

Great! I'll cut a release with the fix this morning.


Add lots of things, mostly organizational things

 Here is an overview of what got changed by this pull request:

Issues
======
+ Solved 1

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
+ Solved 1

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Issues
======
+ Solved 1

See the complete overview on Codacy


Add password

Hello,
i wonder if there is any possibility to protect my server with user & pass just in example.com:9000
& not download link.
best regards

You should be able to do this by using a reverse proxy. Nginx is commonly used.

Yeah, people have been asking this a couple of times: #18, #32 and #54

why not addability to upload torrents directly to google drive

https://github.com/asapach/peerflix-server/wiki/How-to-put-a-password-on-peerflix-server

@maxerbox, thanks, I'll put it into FAQ.
Squid is not ideal, though, because it doesn't proxy websockets. If somebody could come up with a similar guide for nginx or other reverse proxies that would be great.

@asach You're welcome ! Looks like it's working without websockets. The frames are converted to http request : https://i.imgur.com/teF9g4l.png
I'm too busy to create a nginx tutorial : I'm currently watching a movie with peerflix server :P

Yeah, socket.io falls back to long-polling, etc. so it's working, but as I said it's not ideal.
Thanks again.


Remove bower dependencies, or run bower install with allow-root

When node is running as a root user, the bower install command triggered by the install script is causing the following error :
bower ESUDO         Cannot be run with sudo
[...]
npm ERR<EMAIL_ADDRESS>install: `bower install`

Couldn't the actual bower dependencies be managed in package.json instead, or at least the bower install command launched with the --allow-root option?

Check this out, my friend.
Cheers.

Why are you running node via sudo?

This npm install is executed on a distant host I don't control, hard to guess why it is ran as root.
Anyway, I see that v0.12.3 moved bower dependencies to package.json, therefore the issue is solved.


Removing some authorizations for departed users [skip ci]

Pour one out for the departed 🍺
I have a feeling more should be removed from here.

@hyperbolic2346 looks like signoff is missing on the commits.

LGTM,
Can you also help to update in spark-rapids-jni? Thanks!
https://github.com/NVIDIA/spark-rapids-jni/blob/branch-24.06/.github/workflows/blossom-ci.yml

https://github.com/NVIDIA/spark-rapids-jni/pull/1981

build


Add support for clangd 13

I have downloaded the latest clangd, and coc-clangd is not working as expect. The errors from clangd are here:
E[00:18:13.621] Client requested legacy semanticHighlights notification, which is no longer supported. Migrate to standard semanticTokens request
Please help.

Set clangd.semanticHighlighting to false to disable semantic highlight for now.

Set clangd.semanticHighlighting to false to disable semantic highlight for now.

I have set this in my doc-settings.json, it works very well for clangd 12.
"clangd.semanticHighlighting": true,
Since I noticed that, clangd 12 semantic highlight has many drawback for objective-c, so I update it to version 13.
And coc-clangd not works for that. Some protocol has changed as the error message mentioned.

Yes, clangd 12 and 13 supports different semantic highlighting protocol, coc/coc-clangd doesn't has this now, so you need to disable semanticHighlighting for now.

Set clangd.semanticHighlighting to false to disable semantic highlight for now.

I have set this in my doc-settings.json, it works very well for clangd 12.
"clangd.semanticHighlighting": true,
Since I noticed that, clangd 12 semantic highlight has many drawback for objective-c, so I update it to version 13.
And coc-clangd not works for that. Some protocol has changed as the error message mentioned.

I guess maybe those changes will fix objective-c's semantic highlight problems, so I try to update it.
https://reviews.llvm.org/D97617

Yes, clangd 12 and 13 supports different semantic highlighting protocol, coc/coc-clangd doesn't has this now, so you need to disable semanticHighlighting for now.

Thanks for replay, I still want to use coc-clangd, so I will rollback my clangd 13, waiting for you updating...

coc.nvim has added semantic tokens support, check :h coc-semantic-highlights.


how to force phone only using tcp for sip?

currently it seems sipgo only use udp,  thanks for all your info!
the reason i want to use tcp is i need to transfer large message than mtu, udp is not fit.
{"level":"debug","caller":"transaction.Layer","error":"size of packet larger than MTU","req":"MESSAGE sip:1000@<IP_ADDRESS>:7060 SIP/2.0","time":"2023-12-15T14:43:35+08:00","message":"Fail to write request on init"}

no all protocols are supported. It can be set by recipient header parameter or by setting msg.SetTransport on message.


consider support for implementation-specific members

The JSON:API specification forbids implementation-specific members unless they are explicitly allowed:

Unless otherwise noted, objects defined by this specification or any applied extensions MUST NOT contain any additional members. Client and server implementations MUST ignore non-compliant members.

@lode proposed relaxing this statement to allow implementation-specific members of links object. Similar to implementation-specific query parameters we could allow implementation-specific members.
I feel we should discuss this not only with perspective on links object. Similar arguments could be made for implementation-specific members of relationship object, an error object, or even the document itself. I think we should discuss if we want to allow it in general.
Extensions allow adding extension-specific members:

An extension MAY define new members within the document structure defined by this specification.

All use cases for implementation-specific members are already covered by extension. Not requiring content negotiation would be the only advantage of implementation-specific members over members defined by extensions.
I feel requiring extensions and explicit content negotiation is more future proof. Especially as implementation-specific members could cause conflicts if implementations at client- and server-side are not compatible. Content negotiation for extensions prevents these.
On the other hand the same argument could be made regarding query parameters. And we decided to support implementation-specific query parameters.
Opening up this issue to gather feedback both from implementers as well as other editors of the spec.

Nice to discuss this @jelhan!
I'm not sure about opening up this to everywhere in the document, although I understand your reasoning.
My request for the links object also came from my thought (I admit, my confusion) that the links object had the same free rules as attributes, relationships and meta.
Also, to me it is logical that the links object contains user data (conceptually, not currently in the spec). Where as in the rest of the document I feel that need less, there the document describes structure / meaning.

yeah agree, I think opening it up for the links object is ok, but relaxing it across the whole document feels potentially dangerous.


[frontend]: check light theme on specific computers

Chiali reported that her machine is still showing the page with the light theme. Possible solutions:

check that dark theme is default
check what happens when browser has a explicitly light theme setting on
it might be saved in localStorage from the previous try. How could we forcibly delete localStorage?

Closed in #76


Forced htmlentities() encoding in custom css 'content: ">";'?

We injected some custom CSS code (in Settings > Custom HTML head content) to display an "north east arrow icon" ↗ behind external links on BookStack pages, using the content property, code point ↗.
Since release 21.08.x BookStack is parsing the value behind content, and automatically re-encodes HTML entities, not only „high“ Unicode data points, but simply everything which supports HTML re-encoding, in particular also § towards §.
The change in 21.08.x unfortunately breaks our icon hack.
Here our said CSS injection:
<style type="text/css">
  /* adds arrow icon to external links */
  a[href*="//"]:not([href*="our.bookstack.com"])::after {
    content: "↗";
    text-decoration: none;
    padding-left: 0.15em;
    color: #999;
  }
  a[href*="//"]:not([href*="our.bookstack.com"]):hover::after {
    text-decoration: underline;
    text-decoration-color: red;
  }
</style>
<script>
  /* sets the target attribute to '_blank' if link doesn't begin with '#' or 'https://our.bookstack.com' */
  window.addEventListener('DOMContentLoaded', function() {
    document.querySelectorAll('.page-content a[href]').forEach(function(el, x, y) {
      if (el.getAttribute('href') &&
          (!el.getAttribute('href').startsWith('#')) &&
          (!el.getAttribute('href').startsWith('https://our.bookstack.com'))) {
        el.setAttribute('target', '_blank');
      }
    });
  });
</script>

Hi @Wookbert, Thanks for reporting and sorry for this.
Just so I can be sure, could you confirm the exact version you're on which has this issue?
I made some more significant changes to Custom HTML Head content within v21.08.2 which could have caused this but if you happen to be on v21.08 or v21.08.1 it'll be something unexpected.

I'm seeing this issue now on v21.08.2. Would have to ask my partner if it appeared in earlier versions.

@Wookbert Thanks, That's all I need. Pretty sure it'd be a v21.08.2 change that's caused this but wanted to confirm just in case I was jumping to assumptions.

Just learned that we went from 21.05.4 straight to 21.08.2, and skipped the in-betweens.

Just applied ef459ca4c45bd2450ecd5d67a4d73c07f836ab25,
Will be part of the next patch release that should be released shortly.
Thanks again for raising @Wookbert.

Now that was quick. The icons are back. Thanks, mate!


WEB-5684 moved to CircleCI

What are the links to relevant tickets?
WEB-5684
What does this PR do?
Moves the build agent from shippable over to CircleCI.
What functional/unit tests does this PR have?
N/A
How should a developer review this?
Eyes
Any background context you want to provide?
You know... shippable and all that
Screenshots (if appropriate)

Quality checklist (for PR author to complete BEFORE code review):

[x] Bumped version in package.json.
[x] Created circle.yml in project.
[x] Deleted shippable.yml.
[ ] Split any deploy script to deployToStaging.sh & deployToProduction.sh.
[x] Setup GHUSER environment variable in Circle.
[x] Setup GHPASS environment variable in Circle.
[x] Setup GRAPHITE_API_KEY environment variable in Circle.
[x] Setup GRAPHITE_ENDPOINT_PREFIX environment variable in Circle.
[x] Setup slack webhook.
[x] Setup API status token in Circle.
[x] Update Readme with new build badge.
[ ] Added AWS credentials for shortbreaksBuildAgent
[x] Added shortbreaksBuildAgent Deploy Key
[x] I am ready for this to be code reviewed, merged and tested.

Reviewer 1 @taijuten

[x] I agree with the assumptions made in the quality checklist.
[ ] I’ve witnessed the work behaving as expected.
[x] I’ve checked for appropriate test coverage.
[x] I’ve checked for coding anti-patterns.
[x] I've checked this work against the requirements of the Jira.
[x] I don’t believe this work introduces unnecessary technical debt.
[x] +1 from me!

Reviewer 2 @lukefrake

[x] I agree with the assumptions made in the quality checklist.
[x] I’ve witnessed the work behaving as expected.
[x] I’ve checked for appropriate test coverage.
[x] I’ve checked for coding anti-patterns.
[x] I've checked this work against the requirements of the Jira.
[x] I don’t believe this work introduces unnecessary technical debt.
[x] +1 from me!

Also, can we delete the TODO.md


Investigate https://github.com/nicolaiarocci/cerberus

For data validation.

Without having a proper look, I think we may be able to use this to tidy up the validify_dictionary method.

Closing this as incorporated into issue #79


Add action for MSYS2 builds.

This adds CI jobs that compile and test OpenBLAS in different build environments that are provided by msys2 on Windows.
It uses cmake/ninja for configuration and building. GNU autotools are notably slower compared to them on Windows for some reason.
It already adds the configuration switch -DC_LAPACK=ON for the jobs that run with the CLANG64 build environment. So, it should pick up the changes from #3539 when they are committed. That switch is ignored currently.
I hope this can be useful.

Thank you


Bebop : support sonar over IIO

We should support the Bebop's sonar unit. A bit convoluted, because we need to actually trigger the sonar pings via SPI and then read back via the ADC.
I already have the software part of optical flow sorted more or less, but need the AGL estimate to scale the readings.
@eyeam3 Are you working on this by any chance? Would you mind syncing over email/Skype on what's planned/ongoing for Bebop?

Duplicate


Clarify/Simplify how the logout/shutdown works

From Jonathan,

right now you have to stop your server manually before you logout.  Or, if you log out without shutting down the server, then the server continues to run.  I think the request is that logout = logout + server shutdown.

Just to add some context here, to the average user I think "logout" implies shutting down the server (which it doesn't actually do). It's not intuitive to me that the actual desired behavior is to click "Hub Control Panel", which opens a new tab/window, and then "stop my server", when the "logout" button is right there.
In my fantasy world, there would be a big red "shut down server" button at the top of every session, that requires a single click to shut down without opening any new tabs.


kernel: config: Improve CI for kernel config changes

There is two part in this issue.
Keep kernel config in sync
We need to maintain the kernel configs consistent between each other (across architectures) because we already have some discrepancies about some components that are not architecture specific. I am not sure how we can maintain some sort of synchronization across archs and which tool already exist for this, but we need this if we want to make sure we'll have Kata working consistently across x86, ARM and PPC.
Validate kernel config changes
We need the CI to run some specific check when modification are done to the kernel/config directory. This means every time a config gets modified, it should try to build the kernel with this config and then recreate an image to perform a sanity check (or maybe the full CI ???) that Kata Containers still work as expected. And to go one step further, we would need specific CI jobs to trigger the same tests for ARM and PPC configs on the appropriate worker VMs, so that we try on the appropriate arch. Maybe a simple way to bypass the complexity of having several worker VMs with different architectures would be to use cross compilation (but this means we can only try the build, we couldn't test Kata after this).
/cc @jodh-intel

Thanks for raising this @sboeuf!
Keep kernel config in sync
+1.
As mentioned on https://github.com/kata-containers/osbuilder/issues/6, I've been digging around a bit and comparing the kernel configs and they do not appear to be consistent. For example, wouldn't we expect the following three values to be identical?
$ cd $GOPATH/src/github.com/kata-containers/packaging
$ for config in kernel/configs/*.x; do grep ^CONFIG "$config"|grep CONFIG_ARCH|sort -u|wc -l;done
37
49
62

What we need is for a kernel expert to pick over the three config files and work out:

Which configs are missing key features
aka: platform-inconsistency - running a container on platform X fails on platform Y).

which configs have "unnecessary" features enabled
aka: bloat, boot speed and security considerations.

/cc @bergwolf, @gnawux, @egernst, @grahamwhaley, @annabellebertooch, @brecky777.
Related:

https://github.com/kata-containers/community/wiki/Document-essential-kernel-config-options-for-ppc64le,-arm64-and-x86_64.
https://github.com/kata-containers/packaging/issues/8
https://github.com/kata-containers/packaging/issues/55
https://github.com/kata-containers/packaging/issues/56

Validate kernel config changes
+1.
The question is how realistic is it that we'll be able to re-test the kernel build for all three architectures if a file in kernel/configs/ changes? Would zuul be able to support our three architectures (x86_64, arm64, ppc64le) @mnaser, and @cboylan?
/cc @Pennyzct, @Weichen81, @nitkon, @harche.


Cannot get the example to work

Steps to produce the issue:

follow cloning and copying component.mk files
make manuconfig and change serial flasher to 115200
I am using this display http://www.lcdwiki.com/2.8inch_SPI_Module_ILI9341_SKU:MSP2807, ESP module is https://wiki.wemos.cc/products:d32:d32
connect the following pins (MOSI, SCLK, CS, DC, BCKL, RST) as per default config
make flash monitor
the error is bellow

ILI9341 initialization.
Enable backlight.
E (11314) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (11314) task_wdt:  - IDLE0 (CPU 0)
E (11314) task_wdt: Tasks currently running:
E (11314) task_wdt: CPU 0: main
E (11314) task_wdt: CPU 1: IDLE1

Have you changed anything in the code?

No, I didn't change any code, also I tried to copy the example to my own project and got the same error.
The screen background led will go on and nothing on the display.

I'll update the project to the v6.0 version of LittlevGL next week. Hopefully, the problem will be solved then.
If not, at least I will have an ESP32 with a display to make further tests.
BTW, the error message says "watchdog got triggered". Is the watchdog enabled by default?
To see what happens in LittelvGL you can enable Logging

Thanks I will wait for the updated repo.
About the error message I think it is from FreeRTOS.
I have also enabled logging for LittelvGL, it is logging that the UI is being drown but nothing on the screen and I get the same watchdog error.
I have tried different boards and screens and different example configurations it all give the same error.
By the way I update the LittelvGL library, can I just pull from master?
Regards

By the way I update the LittelvGL library, can I just pull from master?

Yes, but note that the prototype of the drivers has been changed.  See these about the Display and Touchpad.
You should also update the lv_example repo.

I disabled the watchdog interrupt to don't get those warnings, it's not recommended to do so but it's just for trying out lvgl.
After doing make menuconfig go to Component config > Common ESP Related and disable Interrupt watchdog and Initialize Task watchdog timer on startup.

I've updated the project ad found the issue. It was the touchpad driver.
Is it working there too?

I could not get the touch to work on either this repo or https://github.com/espressif/esp-iot-solution/tree/master/examples/hmi
On the iot-solutions example I got the display to work only.

In this project, is the display working?

yes display is working in  iot-solutions  https://github.com/espressif/esp-iot-solution/tree/master/examples/hmi

yes display is working in iot-solutions

However, it still uses  v5.3. In this issue, the goal is to update this project to v6.0

Not sure if you've solved this issue but I've implemented a driver for the ADS7843 touch screen driver used in this display (http://www.lcdwiki.com/2.8inch_SPI_Module_ILI9341_SKU:MSP2807), integrated this with great LittlevGL and mongoose OS. It's available at https://github.com/pjaos/mgos_esp32_littlevgl if you're interested.

Hi
Actually I did not fix the issue as I was working on other parts of the project (it is part time so it is moving slow). I will definitely check the new one also and report back here is the issue is solved.
Thanks again

I suggest continuing it on the Forum because more people are reading that.
https://forum.littlevgl.com

I can confirm the display example is working with latest esp idf pulled today.

Awesome, thank you for the feedback!


Update of qiskit solution

Some fixes for session on https://techorama.nl/
Latest qiskit version, removes need for linux subsystem on windows.
Support for, IbmQx16Melbourne, which I use because the ibmqx4 is having api issues (suspecting its going into maintenance soon).

Lot of changes that don't belong in this pull request,.


Total Abundance and Enrichment Fraction - multi

Summary Change Description
Updated the Total Abundance and Enrichment Fraction cached_functions.
Affected Issue Numbers

Partially addresses #394

[x] PeakGroups:'Total Abundance'...
[x] PeakGroups:'Enrichment Fraction'...

Code Review Notes
Since the enrichment fraction has to be calculated per labeled element the measured compound has in common with the tracer labeled elements, I changed the return value to a dict keyed on element.  I'm not sure how that's going to play out downstream.  (I'm not sure there exists a different model to move it to where the relationship is 1:1, so I invite discussion on that.)
There are some exception catches that could possibly be converted to tests, because given the model structure (e.g. unique constraints), some should never happen, but my thinking was to make the code robust to change/model-editing.
I added a bunch of TODO comments that should all be resolved by the end of issue #394.
I couldn't get anything to work until I blew away the migrations.  Let me know if there's a way for me to restore the migrations and re-roll the commits.
The new methods tracer_labeled_elements and common_labels mirror those of the accucor_data_loader.  They take different inputs (because one's for the database and one's for processing file input.  Let me know if you have suggestions on how to consolidate them.
I added a finally block under a series of catches just so I don't have repeated code.  It was awkward doing it that way, having to maintain an "error" variable.  Would it be better to have a single except line with all the exception types and conditionally handle them based on the exception type/class?
I'm sure there can be lots of improvements that can be made, and I'm eager to make them, so please rip it apart!
Checklist

[ ] All issue requirements satisfied (or no linked issues)

[x] PeakGroups:'Total Abundance'...
[x] PeakGroups:'Enrichment Fraction'...

[x] Linting passes.
[x] Migrations created & committed (or no model changes)
[x] Tests implemented (or no code changes)
[ ] All tests pass

[x] DataRepo.tests.test_models.PropertyTests.test_peak_group_peak_data_1
[x] DataRepo.tests.test_models.PropertyTests.test_peak_group_peak_data_4
[x] DataRepo.tests.test_models.PropertyTests.test_peak_group_peak_data_serum
[x] DataRepo.tests.test_models.PropertyTests.test_enrichment_fraction_missing_compounds
[x] DataRepo.tests.test_models.PropertyTests.test_enrichment_fraction_missing_labeled_element
[x] DataRepo.tests.test_models.PropertyTests.test_peak_group_total_abundance_zero
[x] DataRepo.tests.test_models.PropertyTests.test_peak_group_common_labels
[x] DataRepo.tests.test_models.PropertyTests.test_peak_group_tracer_labeled_elements
[x] DataRepo.tests.test_models.PropertyTests.test_no_common_labels

With the caveat that a bit of this code is changed in the next PR, I have taken this out of draft.

I agree on the creation of a hypothetical "peak_group_derived_data" model, which I think we had discussion on, but probably never got around to codifying.  I think I saw an issue created for that.  I say press on, and we will keep refining.  Sorry I haven't been much assistance.


why are  many goroutines got stuck  at batchPerform

Output of helm version:
helm v3.9.0
I use helm sdk(v3.9.0) in my operator to CRUD charts,  I find the operator's memory is growing .  I use pprof to detact this problem. Finally, I find many goroutines are got stuck at batchPerform()

Can any one help me? thanks

location that calls batchPerform: https://github.com/helm/helm/blob/76157c6d06f981a42bfbb9ec636bae5443cb2c8f/pkg/kube/client.go#L540
The stuck chan send (I believe): https://github.com/helm/helm/blob/76157c6d06f981a42bfbb9ec636bae5443cb2c8f/pkg/kube/client.go#L582
From a quick glance, the error handling in (perform](https://github.com/helm/helm/blob/76157c6d06f981a42bfbb9ec636bae5443cb2c8f/pkg/kube/client.go#L543-L546) seems a little suspect. If I am reading correctly, batchPerform is doing the synchronization, but perform will return immediately on first error. The channel is unbuffered, so once an error happens, send will block as nothing will attempt to wait anymore.

I read the same, @gjenkins8. This could either be fixed with a quit channel, or a buffered channel. I do believe the quit channel is the more acceptable choice.

@joejulian  what's a quit channel ? please give me example,thanks

I think @joejulian means an additional channel which allows the caller (perform here) to tell the callee (batchPerform) to "quit" (ie. finish executing).
I would propose more that simply the synchronization/error handling mechanism between these two functions needs to be fixed. However that is achieved, as long as it is done to a good standard (if I were to code a fix, I would look to use a standard implementation of the scatter/gather pattern used here; rather than a custom, and perhaps (unsurprisingly) buggy, impl.

Right. Rob Pike covers them in this talk: https://www.youtube.com/watch?v=f6kdp27TYZs


Merge duplicates modifier/modal

Based on total cents with a configurable tolerance. Replace duplicates with their arithmetic mean in pitch space.

hikicawmori — Today at 4:21 PM

I'd like a "Remove duplicate notes" feature for ScaleWorkshop. One use case would be tempering out a certain interval that occurs in a JI scale and removing the duplicate values that occur afterwards.

With a non-zero tolerance there should be an option to replace the intervals with their average or keep the less complex one.

Abnormality — Today at 8:45 PM

can you add a "unique" or "remove duplicate values" function bc when i wanna do something like this i dont wanna hafta manually delete all of the duplicates


Request to upgrade Go packages to fix a vulnerability

We appreciate the value this tool brings to our production environment. However, our vulnerability scans have identified vulnerability with the package github.com/emicklei/go-restful, an indirect dependency of this project. Could this be addressed?
Vulnerability details
+------------------+----------+------+-----------------------------------+---------+------------------+-----------+------------+------------+----------------------------------------------------+-------------------+
|       CVE        | SEVERITY | CVSS |              PACKAGE              | VERSION |      STATUS      | PUBLISHED | DISCOVERED | GRACE DAYS |                    DESCRIPTION                     | TRIGGERED FAILURE |
+------------------+----------+------+-----------------------------------+---------+------------------+-----------+------------+------------+----------------------------------------------------+-------------------+
| PRISMA-2022-0227 | high     | 7.50 | github.com/emicklei/go-restful/v3 | v3.9.0  | fixed in v3.10.0 | > 2 years | < 1 hour   | -629       | github.com/emicklei/go-restful/v3 module prior     | Yes               |
|                  |          |      |                                   |         | > 1 years ago    |           |            |            | to v3.10.0 is vulnerable to Authentication Bypass  |                   |
|                  |          |      |                                   |         |                  |           |            |            | by Primary Weakness. There is an inconsistency in  |                   |
|                  |          |      |                                   |         |                  |           |            |            | how...                                             |                   |
+------------------+----------+------+-----------------------------------+---------+------------------+-----------+------------+------------+----------------------------------------------------+-------------------+

Expected Behavior

Current Behavior

Possible Solution
go mod why shows that multiple packages depend on the go-restful package. In my local experiments, upgrading the github.com/elastic/cloud-on-k8s package bumps go-restful to >v3.10.0, which includes the fix for the vulnerability.
Context
Your Environment

Operating System and version:
Link to your project (optional):

Since you've already locally done the needed package update, would you want to submit a PR with the updates?

I'd be happy to submit a PR, but it looks like my experiment was incomplete. I had only checked the package versions without validating if it would work. Today, I tried running it locally and encountered compatibility issues
go test controllers/*.go
# github.com/tigera/operator/pkg/render
pkg/render/logstorage.go:318:15: cannot use corev1.ResourceRequirements{…} (value of type "k8s.io/api/core/v1".ResourceRequirements) as "k8s.io/api/core/v1".VolumeResourceRequirements value in struct literal
pkg/render/logstorage.go:330:32: cannot use overridePvcRequirements(pvcTemplate.Spec.Resources, userOverrides) (value of type "k8s.io/api/core/v1".ResourceRequirements) as "k8s.io/api/core/v1".VolumeResourceRequirements value in assignment
pkg/render/logstorage.go:330:56: cannot use pvcTemplate.Spec.Resources (variable of type "k8s.io/api/core/v1".VolumeResourceRequirements) as "k8s.io/api/core/v1".ResourceRequirements value in argument to overridePvcRequirements
FAIL	command-line-arguments [build failed]
FAIL

I will look into this and see if I can make the version changes without affecting compatibility.