Periodic disconnects - add scheduled reboot?

Hi CircuitSetup,
I'm really enjoying my Split Single Phase Energy meter, with only one issue - every now and then, approximately every 10 days, it disappears from the network, and requires a restart.
I've had other ESP devices in the past that needed a periodic reboot (once / week, etc) to perform reliably - I could never quite get to the bottom of what the issue was, and didn't have the time to resolve it 'properly' either. After reading a few things online, I thought maybe the meter might be swapping APs or something (multiple APs in the house), and having issues, but I've switched it to an SSID that is only on one AP (with only the ESP on the SSID), so it should not be this.
Please note that I have read "Using web interface and/or leaving it open causes ESP32 memory leak/crash" - I'm running v2.5.5, so should not be this either (although I do always use Chrome).
Right now, I've got the meter connected through a smart outlet, so if it goes down I can disconnect and reconnect it remotely, but I would rather have this automated on the ESP itself.
Can you help with solving this issue? I can certainly send logs (let me know how), or, alternatively, could you add a scheduled reboot to the WebUI? I think this would 'solve' the issue.
Has anyone else reported this?
Kind regards,
Andrew

Hi @amil109 - I'm sorry it's giving you issues! It looks like you have the kit that came programmed. When you updated to v2.5.5, did you compile it, or just load the binary via the web interface? I ask because if you compiled and the libraries weren't up to date, that could still cause memory leak issues.
It seems that some people still do have this issue, but it's usually related to the signal strength, and the ESP32 being able to reconnect to the AP when it looses the connection. After trying to reconnect, it will go into AP mode, which uses a lot of power. It can sometimes cause the brownout detector to trip, and the ESP32 to freeze.
What specific AC transformer are you using? Is it at least 1000mA?

Hi @CircuitSetup ,
Thanks for the quick response!
I used the binary to flash via the web interface.
My AC transformer is 2000mA, it's one for 230V, not listed, so I've got a calibrated value in the WebUI.
The ESP is very close to my router (~1m, they're in the same cupboard) I don't know if this might be an issue?
 What is the voltage output. Alsom is it one that plugs into the wall? Some people have had some issues with transformers that do not have any type of filtering caps or protection from voltage spikes.
The closer to ESP32 is to the AP, the better. That's probably not the issue either.

Hi @CircuitSetup ,
It is a 'wall wart' type transformer, 9V AC 2000mA output - this is the link.
If you can recommend a better / different brand for use in the UK, please let me know - I will happily purchase it!
Kind regards,
Andrew Mills

There isn't too much detail about how that transformer works on that page, but I can see that it's passed UK standards, so it's probably doing what it is supposed to do.
In EmonCMS, you can also take the available memory and make that an input, graph it, and see if it running out of memory. If it is there is an unexpected leak somewhere. If not, then it's likely a connection/power issue.
If it is a power issue the only other suggestion I would have before trying a different ESP32, is to put a 10uf to 50uf through hole style capacitor between the 3v3 and gnd pins on the ESP32. This can either be soldered in or put in the pins that hold the ESP32 on the underside of the adapter board.
Let me know what you find.

One other thing - Are you sending any data via MQTT? I just fixed a bug with that when it got disconnected.

Yes - I am sending MQTT data to my OpenHab instance - should I stop this? TBH I'm not using it right now. I'm also happy to flash a Beta with the fix if you have it?

I fixed an issue in the latest release that would cause MQTT to try to reconnect for 14 min straight if the connection was lost. It was only supposed to do this for 5 seconds, then every 10 seconds. This could cause the ESP32 to lock up, which may be what you are experiencing. Please flash the latest bin to the ESP32 via the web interface, and let me know if this helps.

I've flashed it now - I'll keep you posted!
Prior to this, it worked for ~15 days without issue, so it might be a hard one to detect.  I can certainly try that. It's completely possible that the energy monitor failures were aligned with me messing around with the MQTT server, and I just didn't realise in the past.

Hi @CircuitSetup,
My device has been operating without issue for a week since flashing - I think if it lasts one more week without issue we can consider this topic closed! Thanks very much for your help. I'll post if I have any issues in the next 7 days.

Great! I'm glad to hear it's working better for you now.

Hi @CircuitSetup  I got my device and it's great. Works really well, besides the disconnects. I thought it was my wifi signal as my AP was very far.. I have since got another AP in the mix ( Unifi AP Setup, network is solid ) and I get a great -40 dBm connection.  Yet at random.. It can happen more than once a day, but it's at minimum once a day I will need to unplug the power supply and plug it back in.. If I am running a consistent ping it comes back really fast, 3-5 seconds. Is that even a full boot? Any idea what could be going on?  I am only running Emoncms also .. No MQTT from this device.
Free RAM: 213K
Flash Size: 4096K
Version: v2.5.7

@CircuitSetup John, Could you please advise.. It's really annoying this is not a stable device..

Sorry @McGeaverBeaver, I overlooked your initial message.
Did you update the firmware from the provided bin or did you compile from source?
Also, when it randomly disconnects is there a browser window open anywhere?
It may help to collect data for available ram in emoncms to see if the esp is running out, then freezing. That would indicate a memory leak. I've seen this happen when the software isnt closing connections correctly.

@CircuitSetup No problem.. I have updated the FW from the provided bins.. You have 2 , do I need both uploaded? I have done that src.spiffs & src.ino.esp32.. It's on v 2.5.7 and said successful after both uploads..
No browsers opened when this happens. I only have it logging to Emoncms..
I can also check the lights before I reboot it next time this happens but I think the red light is on and no blue light..
is a  reboot normal to only take a few seconds? unplug power supply from wall and plug it back in..

Tou shouldn't need to update the SPIFFS file, no. The latest just has some text changes in the admin interface.
It sounds like the esp32 is freezing. Are you using the transformer that came with the kit or another one?
If you can, I would monitor the available memory, which you should see as an input in emoncms. This could indicate a memory leak.

Ok I have added RAM to start logging.. I will let you know the results .. Currently sitting at 225464

@CircuitSetup  Hey , So it happened again, here is a short clip of showing the ram usage .. doesn't look to be memory related.. https://fastshare.space/33d36b3b6RS.mp4

@McGeaverBeaver I'm sorry it's still giving you issues. If it's not memory, the ESP32 is either having issues with power, or staying connected to your AP. Is your network configured with any repeaters where 2 physical APs have the same name? Or 5ghz and 2.5ghz bands with the same name?

Hey @CircuitSetup
So I don't think it will be my WiFi , I have a Unifi setup with 2 of the latest AP Pros.  I also have a " IoT " SSID set for only 2.4ghz.. I have over 30 devices on my WiFi and none have issues.
As for power issues , you mean delivery? I can ensure you there are no power issues in my home when this happens..
Any way to get a daily reboot integrated, It seems that would just be the easy fix here as unplugging the transformer from the wall and plugging it back again right away, not waiting.. and if I have a ping up on another system the network comes back in just a few seconds..
so I ask again, how long does this normally take to reboot from full power loss? I really don't feel like to open my panel up and d/c the CT's so I can test this as there looks to be retained power from the CT's..

Thanks for clarifying!
I meant to say, regarding power issues, is that the ESP32, for some reason, requires more power than what is available, or there is just something wrong with it. When this happens, it could just be losing the wifi connection over and over, then freezing when trying to reconnect to the AP.
Normally, it should take around 10 seconds to reboot and re-connect to the programmed AP.
The right way to do it, is to set up some kind of watchdog - if the meter doesn't send data for x amount of time. Power cycle the AC adapter via smart plug. But that's still a bandaid.
Either way, I want to try a new (programmed) ESP32. If you can provide your order number, I'll get it mailed out.
Also, nice homelab!

@CircuitSetup  Thanks John, Fair enough it would only be a bandaid and not identify root cause..
My Order 89095 from crowd supply.  Thanks , I didn't think about it until last night my Konnected.IO also runs on an ESP32 and it doesn't have the same disconnect issues so maybe you are right and onto something..
Thanks again John!

Oh okay - you must not be in my order system. Please send me your mailing address via this form: https://circuitsetup.us/index.php/contact/

@CircuitSetup My bad.. I sent  it twice , Forgot the address the first message. Thanks Again John!

Great, I'll send it out today!

@CircuitSetup  Thanks!! I'll give you an update after I swap it out and wait a couple days!

@CircuitSetup  Hey John, So I got the new ESP32 , swapped it out and I've def noticed that it's been more stable.. But I've had it disconnect on me twice without recovery.. Should it not remain to try to connect to AP and go back into AP mode every 5 minutes? I don't even see the ESP broadcasting it's SSID right now with the current failure and lights on it are : On your board - Green Light, Red / Blue Light flashing. On the ESP - solid red light, blue light flashing.  not sure if that helps you.

@McGeaverBeaver It should try to reconnect to your AP every 5 min if it fails, yes (assuming the ESP32 isn't frozen), and will be in AP mode otherwise.
The blue LED will indicate the status:

blinks every 2 seconds - the ESP32 is in AP mode, and it should be broadcasting.
blinks every 4 seconds - connected to your AP.
doesn't blink at all, then the ESP32 is likely locked up.

I just updated EmonESP to v2.6, which has a number of bug fixes, adds a watchdog timer, and replaces the MQTT library. See details here: https://github.com/CircuitSetup/Split-Single-Phase-Energy-Meter/releases/tag/v2.6

@CircuitSetup  So I had it on a relay and was rebooting it if it stopped responding to ping.. Was working well, didn't have any gaps in data for days.
I did as you suggested and updated to - v2.6-src.ino.esp32.bin
Now emoncms won't connect and I can't downgrade back to 2.5.7 ?

I'm not sure why you wouldnt be able to access the interface with v2.6. Was the blue light blinking on the esp32? If so, it was either in AP mode or connected to your AP with a new IP. Maybe your AP thought it was a new device?

@CircuitSetup  John.. Enough with the help desk troubleshooting.. It seems to me you're not reading also my posts and I'm legit trying to help you fix YOUR product. I have it working in a hacked up way that I'm " Fine " with. You're the one who clearly stated it should not operate this way and you continue to blame my wifi.. I feel you're treating me like a 80 year old grandmother having issues with Facebook Live.. " IS YOUR WIFI ON? "
I appreciate the fact you sent out new hardware and attempted to see if that was the issue.. But it's not. I am running UNIFI AP PRO Access Points, With a Unifi Switch and  $600 mini PC running PFSENSE. My network is not the issue of your product. WIFI is not the issue.
Now,. Please read my post again from yesterday and I will re-write it here..
As you suggested - I upgraded to v2.6-src.ino.esp32.bin
the EMONESP Interface is reachable... I never stated it was not..
EMONCMS from your EMONESP interface won't establish a connection with my server. I have tried factory reset,  upgrade firmware again, input server/api key from fresh factory reset and connected would remain " NO "
I downgraded back to 2.5.7 and EMONCMS connected fine again.
John.. I ask you to STOP blaming my WIFI or Network, Until you get logs showing me otherwise ..
The disconnecting issue is a result of your software or hardware.  Period.

Sorry, I didn't mean to be condescending. Sometimes replying from your phone isn't the best idea. I misread what you were saying, and I forgot that you actually have some good network hardware and know what you're doing.
In my tests I haven't had issues connecting to EmonCMS, so I must be missing something. There were a couple things related to how it's connecting to EmonCMS that were changed in v2.6:

The lib that http and https uses was changed since the old one was being used incorrectly.
There's now a timeout if it doesn't connect within 4 seconds to accomodate for the watchdog.
Some String's were changed to char's to prevent memory issues

Are you using http or https to connect with EmonCMS?
I thought you downgraded by connecting the ESP32 to your computer and uploaded the bin.

Chiming in here, 2.6 has definitely been more stable but I've had a couple more disconnects this week.  I'm using a variety of other ESP32 firmware (OpenGarage/WLED/ESPHome) without issues, and my AC transformer is a 1.5A 9V Jameco.  A mild annoyance for an otherwise great product!

@genehand I'm sorry its still giving you issues. I believe the cause is a combination of being able to stay connected to an AP, and the error handling around that. I'll continue to improve the software.
By the way, you can load ESPHome on the meter esp32 to see if that improves stability. Details here: https://next.esphome.io/components/sensor/atm90e32.html

I'm having issues still as well. RSSI is -72 typically. I'm on 2.6. I'll go check on it after a few days, and it'll be frozen, so it's missing data. Hard for me to calculate monthly costs when I'm missing multiple days at a time. I've had this issue since purchasing it. I've gotten to the point that I don't even rely on it anymore.

@scfigg yep still an issue.. He mentioned my WiFi also.. I went out and purchased another AP ( A Unifi AP ) - It's now VERY close to the device, on it's own SSID 2.4 broadcasting only from that AP with all the options toggled around to see if that would change stability..  I also have a  Konnected Alarm Panel and it uses the same ESP wifi chip and I have NO issues with this device.
It's software issue here.
My fix, I put a script to ping it on HA and if it can't reach it , it cycles a smart switch I have the transformer of the energy meter plugged into..  The smart switch does not change readings, and the reboot is so fast that the data loss for that few seconds is not really a big deal..

Thanks. I was actually just contemplating both of those things - whether or not to put an AP right next to it (I have some nice Ubiquiti network setup in my house, so signal is generally not an issue). I was also thinking of plugging a smart switch into it and forcing reboots.
Care to share your HA scripts for ping/reboot when you get a chance? I'm sure I could figure it out, but if you have something already setup, that would be much easier.

Yea man, WiFi not the problem here..
Let me get what I'm doing for the reboots.. I don't recall from the top of my head also what I am doing lol

@scfigg  lol it works but man I was lazy!  I just have a ping sensor setup in HA , and then an automation that checks every 5 minutes to see if the ping is still alive, if it's not.. toggle power.. I had it on an actual TOGGLE before but I find that for some reason things would be on or off when I didn't want them so I just have it toggle but with an ON/OFF
configuration.yaml

platform: ping
name: emonesp_wifi
host: <IP_ADDRESS>
scan_interval: 60

automations.yaml

id: '123456'
alias: Toggle emon ESP
description: ''
trigger:

minutes: /5
platform: time_pattern
condition:
condition: state
entity_id: binary_sensor.emonesp_wifi
state: 'off'
action:
data: {}
entity_id: switch.emondumbpower
service: switch.turn_off
delay: 00:00:02
data: {}
entity_id: switch.emondumbpower
service: switch.turn_on

Thank you!  Thank you! Now just to figure out how to estimate monthly energy bill cost in HA!

I'm sorry you guys are still having issues with connectivity. I am working on a new version of EmonESP, that hopefully will remedy these issues. There's a lot of things that have been re-worked, and there's a lot of new features too.
In the meantime, since I see Home Assistant is being used, have you considered trying ESPHome?

No worries John, The workaround is easy enough and reporting has been great since.
Yes I have briefly looked at ESPHome and I understand with this we can use to identify single devices out being on ESPHome, but I honestly don't know where to start and haven't looked very hard.. I did enjoy the ease of plug and play with your device.
Looking forward to the update! thanks again John.

Not sure if this helps, but I built a very slimmed down version of the code straight on an ESP32 with almost the only external library being wifimanager. I used it in such a way that it would put the device in AP mode if it ever lost connection.
From what I could see, every few days the device would stop broadcasting data and go into AP mode, so maybe there is an issue in the Wifi library, just an idea, maybe it helps, maybe not.

Any updates yet above 2.6? Still having random lock up and freezes on this Solar Setup?


cirrus cli

Here it is, at long last.
It still needs some testing with the large amount of changes I've made in the past 24 hours, but it is mostly here and decent.
Some outstanding items worth mentioning:

The cloudformation parsing is ill-informed and not great. Resources and Outputs work, but if you need any other cloudformation stuff it won't work. I have some ideas around how to fix this, but it is probably a medium-large lift.
The tack batch resources still have not been ported over to the task definitions (waiting to do this until we have a release we can target for dependent projects to keep them unchanged until ready).
The github workflow testing isn't running on my branch, so I don't know if that workflow config is correct.
I want to try to merge cirrus-lib into this once again. Reading the docs a couple more times, I think I can force the serverless python requirements to do my bidding and only include the lib and its deps in lambda packages, not any cli deps. But it'll take some testing to verify.

I am also sure changelog/readme/whatever docs need serious help after these changes. I am not sure how best to approach them; perhaps a team effort would be best.
Lastly, calling this a 1.0 release, even if alpha, seems generous. I would feel much more comfortable with a 0.x release; aren't minor versions of pre-1.0 releases not guaranteed to be compatible, so we don't have to go all the way to 1.0, right?

This is great @jkeifer. It's such a large change I think it's best to get it merged into main and test from there, making fixes with subsequent PRs rather than trying to continue working from this branch.

The cloudformation parsing is ill-informed and not great. Resources and Outputs work, but if you need any other cloudformation stuff it won't work. I have some ideas around how to fix this, but it is probably a medium-large lift.
The tack batch resources still have not been ported over to the task definitions (waiting to do this until we have a release we can target for dependent projects to keep them unchanged until ready).

Can you create tickets for these in this repo?

The github workflow testing isn't running on my branch, so I don't know if that workflow config is correct.

There's no github workflows set up. We can set one up for testing if we can get tests working offline.  We can add a workflow for PyPI publishing on release, I created a ticket.

I want to try to merge cirrus-lib into this once again. Reading the docs a couple more times, I think I can force the serverless python requirements to do my bidding and only include the lib and its deps in lambda packages, not any cli deps. But it'll take some testing to verify.

So do you mean cirrus-lib will just be included in this new version of cirrus?

I am also sure changelog/readme/whatever docs need serious help after these changes. I am not sure how best to approach them; perhaps a team effort would be best.

Add some line items to the CHANGELOG describing the change.  No need to get real specific at this point until after we release this version and have docs.

Lastly, calling this a 1.0 release, even if alpha, seems generous. I would feel much more comfortable with a 0.x release; aren't minor versions of pre-1.0 releases not guaranteed to be compatible, so we don't have to go all the way to 1.0, right?

Semantic Versioning:

Given a version number MAJOR.MINOR.PATCH, increment the:
MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards compatible manner, and
PATCH version when you make backwards compatible bug fixes.
Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

I think it's fine calling this a MINOR version, as it's just creating a CF template the same as before and can be used to update existing installations, so it would be backward compatible.
So we'll shoot for a 0.5.0 release, there's some tickets to create, and we can discuss what version to target. I'm going to go through the tickets now and mark any relevant ones for 0.5.0


Hide or disable some elements or attributes in the XML editor

In the schema designer, make it possible to hide or disable some XML elements and attributes so that editors can't change them in the XML editor. (Requested by Carole Tiberius.)
Even better, do this as a separate feature outside the schema designer. That way it will work even when the dictionary administrator has uploaded their own DTD (ie. when they're not using the schema designer).

This would be also useful for Lexical Computing. Often we need editors to be able to see values but not to be able to modify them. Note that flagging by clicking the dot next to a headword should not be affected even if the respective flag element has editing disabled (we want editors to flag by clicking the dot, but not to edit the flags manually, possibly introducing invalid values).

Xonomy already has support for this but Lexonomy isn't using it (yet). Same story as #26. The configuration of this should be under Configure » Entry editor.


Added some more cheat sheets

Hey, I modified some more files. There is also sh file containing only sed commands to convert files to mark down. Also I left longer cheats unchanged wherever applied.
Regards

hey, thanks for the PR. I'll go ahead and merge this, but I might consider removing some of the less useful examples (in my own opinion) in the future. Like cd.yml or cat.yml. These seem better suited for manpages.


add email addresses to users in keycloak

In preparation for adding the observability stack, Grafana requires an email address for SSO users.

ah yes. @omrishiv, mind updating it in the stacks repo?

Argh! yea, I see it. Sorry, I'll open one there


Form builder is not working

just drag something to it and try to change an attribute it will show in the console:
"Uncaught TypeError: Cannot read property 'previousAttributes' of undefined"

Is there any progress on this? I spent a good deal of time integrating this, doing the edits to output JSON, etc. and then realized the popovers did not save (doh!).
I receive the following error when hitting save in the popover...
TypeError: model is undefined
var prevId = this.modelId(model.previousAttributes());
backbone.js (line 1148, col 13)
Thanks!

Hoping to get to it this week.

Thanks so much! Please post a reply once you have an update.

@mackaboogie @kitkat14 this has been fixed on the website and is now live.


Infrastructure: bump igluctl to 0.6.0

static push from igluctl pre-0.7.0 cannot recognize a response from Server 0.6.0, when schema already exists. As a result it uploads schema, but exits with 1 code because response was not recognized.
R108 has red Travis badge because of that https://github.com/snowplow/iglu-central/issues/993

Final igluctl 0.7.0 is not released yet - bumping to 0.6.0, which also should not exit with error on unrecognized server message.


When importing csv files, create a new column as a result of rmlc function

See mapping email in example 5

done in the generation of the database, next step: change the resolver


Replace single $ with $$ in exported yml files

https://github.com/rancher/rancher/issues/2351
@ibuildthecloud just to confirm: we have to replace in both docker-compose.yml and rancher-compose.yml?
Also is there a chance that we override user defined value where $ is expected to be single? Like in "command" field for example

This applies to both files.  I can't think of anyway that we will do the wrong thing by turning $ into $$.

@ibuildthecloud ok, then the PR is ready for a review. TF happened in external volume event test @cjellick


Changed to ncc and added bundling

Added bundling so this can be used as a standalone GitHub Action

thank you so much for this work @noliran - this means we can drive this to the marketplace :)


Use composer

Hi.
Could you please make this available with composer? Just like imgix craft 3 version 🥇
Thanks!

I've published it on Packagist now. Could you test it?

I did, but the imgix plugin is in folder /vendor/sjelfull/imgix and not craft/plugins
So when that is fixed, we're good to go I recon 👍

I think it was missing a dependency. Added "composer/installers": "~1.0" now.
Could you try again by removing and adding it?

Awesome! Works if you tag it with a new release. I had to set "sjelfull/imgix": "dev-master" to make it work.

Great, new tag added.


This is test

https://github.com/fhdsl/DaSEH/blob/5528e3466a92ad622bcf446245cfba77bc80cf3a/index.Rmd#L63
Here is my comment

Thanks for this! Tackling this now!

This was an in class demo!


Net: Multiplayer game

This game will be multiplayer game.

I have worthy "BASIC" understanding of ObjectStream for my java socket now.
I under stand what is sent in ObjectStream and understand the basic of Object graph.
I can send the object over the socket stream and "I should be able to trouble shoot the bug or know how to debug a bug when it occurs"


How to initialize NewClientFromKubeConf()?

I am trying to create a Client using the NewClientFromKubeConf factory but keep getting errors.
If I try to simply pass an empty KubeConfClientOptions I get the following error:
panic: kubeconfig missing

Even though my kubeconfig is located under ~/.kube/config and $KUBECONFIG is set accordingly.
So instead I tried passing the KubeConfig parameter to KubeConfClientOptions:
kubeConfig, err := os.ReadFile(kubeConfigPath)
checkErr(err)

clientOptions := &helmclient.KubeConfClientOptions{
    KubeConfig: kubeConfig,
}

kubeConfig properly contains the kubeconfig file as []byte, but I receive the following error anyway:
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x243b3d6]

With a StackTrace that points to this line:
client, err := helmclient.NewClientFromKubeConf(clientOptions)

Am I misunderstanding something?

@zvonkok That did the trick, thank you :+1:
Now I'm running into the next error that I don't know how to solve:
panic: Kubernetes cluster unreachable: no Auth Provider found for name "gcp"

Is there any chance you have a solution for that as well?

Add this to your imports:
_ "k8s.io/client-go/plugin/pkg/client/auth/gcp" // register GCP auth provider

Great, thank you very much!

I think error handling messages could be slightly improved: Similar to how a readable error is thrown when KubeConfig is not passed, the same could be donw for Options. Or simply use a reasonable default instead of the current error that is hard to interpret.


Allow nested RecoilRoot structure

Feature request
Using daily-react inside an application that uses Recoil will lose its ability to access its state values of the nearest ancestor RecoilRoot.
Why you need this
We have application set up with Recoil as its dependency. Since daily-react uses Recoil internally, it is nearly impossible share states between nested RecoilRoot.
Alternatives you've considered
We have tried moving DailyProvider further up in the application structure. We don't think this approach is scalable, since daily-react Recoil does not accept any properties for its internal RecoilRoot implementation.

Hey @zukilover,
thank you for opening this issue and providing a PR along with it!
I'll take a look at #5 and keep you posted about updates.
Best,
Christian

A fix for this issue landed on main (cd732e9) and will be available with the next release.
Closing this issue as fixed ✅


Maximum ECChain Length

If the network goes down for a while, the chain we all need to agree on could get pretty long. If that happens, we could (in theory) start sending messages that approach or exceed the maximum pubsub message size of 1MiB.
We should set some reasonable upper limit on the length of the ECChain. This will increase the time it takes to catch up to the main chain, but the limit can easily be hundreds. We should just have some limit.

Outcome from sync discussion: Let's take a pagination approach and limit to 100 epochs per instance. The relatively low number slows down catch-up after long halts, but it will be fast enough and minimises issues with unexpectedly larger than average messages and on-chain verification.

Note that this means bootstrapping F3 when first activated will take 9 instances to catch up. But this seems fine.

Was this codified?


Cannot see 'solution explorer' in explorer panel MacOS

I have installed the extension vscode-solution-explorer however I noticed that the solution explorer does not stay present in the explorer panel, for instance If I open an empty workspace I cannot see solution explorer until I open some sort of file, then and only then will the solution explorer appear,  therefor if I want to create a new project through VS code I have to open an old project for the solution explorer to appear and then right click on solution explorer to create a new project, I cannot understand why it does not stay present? also support for creating c++ console apps would be amazing

Could you check this setting?
"vssolution.showMode": "activityBar",


Integrate CS with a FL framework

FL is a very attractive use case for MPC as it is used for more and more use cases and as it lends itself very well to be made privacy-preserving due to comparatively simple operations required to be executed using MPC.

First version of integration has been implemented and published (see carbynestack/nettle repository). This uses locally deployed MP-SPDZ instances. For secure aggregation based on Carbyne Stack carbynestack/ephemeral#31 has to be resolved first.

Put on hold due other priorities.


Downloading jar over https protocol

This change was made to avoid the error when install process is executed:
Error: Error getting DynamoDb local latest tar.gz location: 301
at ClientRequest. (.../node_modules/serverless-dynamodb-local/node_modules/dynamodb-localhost/dynamodb/installer.js:14:26)
at Object.onceWrapper (events.js:315:30)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:551:21)
at HTTPParser.parserOnHeadersComplete (_http_common.js:115:23)
at Socket.socketOnData (_http_client.js:440:20)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at addChunk (_stream_readable.js:263:12)
at readableAddChunk (_stream_readable.js:250:11)
at Socket.Readable.push (_stream_readable.js:208:10)
at TCP.onread (net.js:607:20)
Error: Protocol "https:" not supported. Expected "http:"
at new ClientRequest (_http_client.js:131:11)
at request (http.js:38:10)
at Object.get (http.js:42:13)
at ClientRequest. (.../node_modules/serverless-dynamodb-local/node_modules/dynamodb-localhost/dynamodb/installer.js:16:18)
at Object.onceWrapper (events.js:315:30)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:551:21)
at HTTPParser.parserOnHeadersComplete (_http_common.js:115:23)
at Socket.socketOnData (_http_client.js:440:20)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at addChunk (_stream_readable.js:263:12)
at readableAddChunk (_stream_readable.js:250:11)
at Socket.Readable.push (_stream_readable.js:208:10)
at TCP.onread (net.js:607:20)

for request behind a proxy, update with the following:
http.get({
        host: PROXY_HOST,
        port: PROXY_PORT,
        headers:    {
            'Proxy-Authorization':  'Basic ' + new Buffer(PROXY_USER + ':' + PROXY_PASSWORD).toString('base64')
        },
        path: downloadUrl
    }, function (response) {

FYI: This package (dynamodb-localhost) doesn't appear to be maintained, with no new releases to NPM in over 4 years.
Solution: You can use aws-dynamodb-local, a maintained fork, instead. (Disclaimer: I am a contributor to this fork). It is a drop-in replacement for this package, and is updated to fix this bug. If you're using serverless-dynamodb-local, you can also replace that with it's equivalent serverless-dynamodb.
Migrating takes about 2 minutes, with a full guide in the README. Of course, it's all still open-source and MIT licensed. Ownership of this new package sits with a registered charity, that is committed to maintaining the package into the future and is open to contributions from the community.


Hyprland crashes very often, especially closing windows seems dangerous

Hyprland Version
Hyprland, built from branch main at commit d88d5898807bf6f2462cecfb977b7624dafbf52e dirty (swipe: add events).
Date: Wed Apr 3 20:20:47 2024
Tag: v0.38.0-17-gd88d5898
flags: (if any)
System Information:
System name: Linux
Node name: poseidon
Release: 6.7.9-amd64
Version: #1 SMP PREEMPT_DYNAMIC Debian 6.7.9-2 (2024-03-13)
GPU information:
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir [Radeon RX Vega 6 (Ryzen 4000/5000 Mobile Series)] [1002:1636] (rev d1) (prog-if 00 [VGA controller])
os-release: PRETTY_NAME="Debian GNU/Linux trixie/sid"
NAME="Debian GNU/Linux"
VERSION_CODENAME=trixie
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
plugins:
Bug or Regression?
Bug
Description
Crashes when closing a foot terminal (and other windows as well)
How to reproduce
Basically close any window and there is a high chance of hyprland crashing
Heck sometimes even opening windows crashes hyprland
Crash reports, logs, images, videos
hyprlandCrashReport64009.txt

My build was build from a pull without  --recurse-submodules
Rebuilding and testing if it fixes it

#5402 ?

ye

Don't think so the rebuild allows me to spam delete foot terminals without chrashing, but it did die starting up firefox

didn't we tell you it's a dupe of 5402 above


Running a script via the commanline and -S is not working

I can run a C# script in TE2 or TE3 with no problem.
When invoking the same script via the command line I get an error as follows.  C# is all new to me
The script also works from TE2 when I open the tool from the command line using the SPN so I doubt its access related.
Executing script 0...
Script compilation errors:
Error CS1024 on line 1, col 2: Preprocessor directive expected
Error CS1025 on line 1, col 4: Single-line comment or end-of-line expected
Script has LF as EOL. Error occurirng at line 2. I'll dig around and see if I need to install anything else that comes built into TE
#r "Microsoft.AnalysisServices.Core.dll"
using ToM = Microsoft.AnalysisServices.Tabular;

@MikkyD2 The current version seems to require CRLF as EOL. Replacing LF to CRLF in the script should fix the error.

Thank you

This should now be fixed in 2.20.0!


Remove provider string from state

After the update of adding the provider to the model, the ScreenshotScraperGraph using openai fails with this message
"ValueError: Model 'openai/gpt-4o-mini' is not supported".
It seems we're sending the provider prefix to the API endpoint. This code strips that out of the state before making the request

First of all, thanks for taking interest in our project and for contributing.
This problem is probably caused by a recently introduced bug within the AbstractGraph. We'll try to fix that first, and it might make your change redundant. However, if you want, you could help us test if our fix will actually solve this.
Thanks again. We'll get back to you soon.

@f-aguzzi sure, I managed to get the ScreenshotScraperGraph to work with the above code using the latest beta release


[patch] add linting, and remove lint

Please focus CR effort on commit [patch] manually applied linting fixes, CR carefully. The rest are unlikely to break anything.
I've tried to reach a balance between consistency to airbnb style, and things that seem to be part of the original author's style choices. Happy to take your lead on this @lazd, but in any case this PR will go stale super quick so keen to get it in ASAP (and while we only have 3 other PRs open, so we don't force many people to rebase).
Hopefully this will make some of the code more readable, by consistency and cleaning up things like stray unused variables.

LGTM 👍


Account Name Validation

Tools should be provided so that contracts are able to do stateless validation of account names (without lookup), identical to the mainnet:
See: STEEM_MAX_ACCOUNT_NAME_LENGTH

I've just added a "isValidAccountName" method to the SmartContract API, see https://github.com/harpagon210/steemsmartcontracts/commit/b575a9abbab70be5d12108f3bc0d1167757228f2 and https://github.com/harpagon210/steemsmartcontracts/commit/3db35b1280034d3bda4f9778d894d43f867d7e4f
The validation has been taken from SteemJS (see https://github.com/steemit/steem-js/blob/63871f7fdc7b7884c6534fca8ed3851f49ec2393/src/utils.js#L8)
The "tokens" contract will be updated at the same time as the "witnesses" branch is released

Looks great !!


ZenML Hub Client & CLI Commands

Describe changes
This PR adds several CLI commands to interact with the ZenML Hub:

zenml hub list: List all plugins available on the Hub.
zenml hub install Install a plugin from the Hub. Installed plugins can be imported via from zenml.hub.<plugin_name> import .... Docker building then requires required_hub_plugins to be set in the DockerSettings of the pipeline.
zenml hub uninstall: Uninstall an installed plugin.
zenml hub clone: Clone the repository of a plugin (without installing it). Useful, e.g., for extending an existing plugin or for getting the examples of a plugin.
zenml hub submit: Contribute a plugin to the Hub. Performs extensive client-side validation before submitting. Can be run in --interactive mode to ask and validate inputs one at a time. Requires login.
zenml hub login: Login to the ZenML Hub. This sets hub_token of the active user in the DB.
zenml hub logout: Logout by unsetting hub_auth_token.
zenml hub logs: Get the build logs of a plugin.

All those commands interact with the hub via a new zenml.hub.client.HubClient class. The __init__.py files across the repos are configured such that hub plugins can be imported via from zenml.hub.<plugin_name> import ... but zenml.hub.client cannot be overridden.
Pre-requisites
Please ensure you have done the following:

[x] I have read the CONTRIBUTING.md document.
[ ] If my change requires a change to docs, I have updated the documentation accordingly.
[ ] If I have added an integration, I have updated the integrations table and the corresponding website section.
[ ] I have added tests to cover my changes.

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Other (add details above)

@schustmi only requesting review of the docker_settings changes for now

As for tests, I would suggest waiting till the Hub API itself is stable. Not much point in testing the interaction with an unstable untested API IMO.

Regarding the tests: I still think it would be useful them as lots of them are not really related to the Hub API I think, e.g. verifying that the repository structure check works both with and without a subdirectory supplied. However I think that should not be a blocker to get this out for the release, we can do them in some follow up PR.
We should just be aware that our manual tests and reviews probably don't catch all the bugs/corner cases.


Add support for Volumes

Controller changes for this task: https://github.com/UffizziCloud/uffizzi_app/issues/126

Waiting for CI/CD being configured from the open source app


Is any funny business going on with big richness jumps?

I'd feel better if I knew what was going on with big richness jumps like the one plotted below (site_id 27057).

Is that a real change in richness, or did a new observer just start counting the birds in 1993? These big jumps have a big impact on the tails of the random walk, so I think we'll want to see if we can understand them better.
My RPostgres isn't behaving at the moment, so I can't easily access the observer data right now.

@ethanwhite has mentioned funny things around this time period too when using the env data. I thought it might be something to do with species name changes and our filtering of them, but I just checked the raw data downloaded by the retriever and it shows the same thing for this site. A new observer is probably the best explanation. But I will try to do a little detective work on this.

Thanks Shawn!

Yeah, we should definitely do some digging on some of these rapid transitions. In general I think some time spent just looking through the compiled time-series and getting a feel for what sorts of things (weird and not so weird) are going on will be really useful.
Something else other than observer changes to look out for is any potential changes in the route. In concept I believe any change in the route is supposed to be accompanied by a change in the route number to avoid these issues, but I haven't looked into that in a while and could see it causing similar sorts of changes.

For this site it is the observer changing over and counting more birds. here is the data for it from the BBS_weather file.
   year    obsn totalspp
1  1985  990081       42
2  1986  990081       38
3  1987  990081       49
4  1988  990081       43
5  1989  990081       39
6  1990  990081       37
7  1991  990081       33
8  1992 1010060       66
9  1993 1010060       63
10 1994 1010060       61
11 1995 1010060       63
12 1996 1010060       64
13 1997 1010060       64
14 1998 1010060       63

I'll look around and see how prevalent this is.

Thanks Shawn! I can start looking into adding an observer effect to my Stan code if this seems like it becomes a pattern.
Here are the 20 sites with the largest one-year jumps, in case a list is helpful.
61123
17003
91041
27057
91032
66033
72035
52002
46017
7020
72012
61045
58001
92019
81043
61064
58023
14151
91025
91012

On Aug 10, 2016, at 11:47 AM, Shawn<EMAIL_ADDRESS>wrote:
For this site it is the observer changing over and counting more birds. here is the data for it from the BBS_weather file.
year    obsn totalspp
1  1985  990081       42
2  1986  990081       38
3  1987  990081       49
4  1988  990081       43
5  1989  990081       39
6  1990  990081       37
7  1991  990081       33
8  1992 1010060       66
9  1993 1010060       63
10 1994 1010060       61
11 1995 1010060       63
12 1996 1010060       64
13 1997 1010060       64
14 1998 1010060       63
I'll look around and see how prevalent this is.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/weecology/bbs-forecasting/issues/61#issuecomment-238910019, or mute the thread https://github.com/notifications/unsubscribe-auth/AAzdCaCiRC3e8xn8I6NTC804-XXffIZHks5qefKTgaJpZM4JgeJe.

Here is the richness time series for all the sites you mentioned. Note that I haven't filtered out waterfowl. The red lines are when a new observer started. This can definitely have an affect on the birds surveyed, but not always.

Nice work.
Looks like we have a real effect, even if it isn't always large.  I think that adding this to the Stan model would be pretty easy, since it just affects the observation error and not the process error.
Would you be able to send a PR that adds observer ID as a new column? I think we could basically treat it like an environmental variable.

Yes, definitely nice work! Very interesting.

I think we could basically treat it like an environmental variable.

Is there any risk of this basically becoming a site effect in the spatial models? So, we say we're controlling for observer differences, but if each person generally does one BBS route there's a close mapping between site and observer? I don't think this matters for the current Stan model, but I'm just trying to think through the implications for our understanding if/when we apply this to other models.

All of the software I normally use for mixed models (i.e. Stan, lme4, and derivatives like gamm4) can handle random effects with both “nested” and “crossed” structure, so they should do "the right thing" in this case.  The closer the mapping is between observers and sites is to "one-to-one", the more uncertainty there will be, but nothing should outright break.
Does that answer your question?

On Aug 10, 2016, at 12:47 PM, Ethan White<EMAIL_ADDRESS>wrote:
Yes, definitely nice work! Very interesting.
I think we could basically treat it like an environmental variable.
Is there any risk of this basically becoming a site effect in the spatial models? So, we say we're controlling for observer differences, but if each person generally does one BBS route there's a close mapping between site and observer? I don't think this matters for the current Stan model, but I'm just trying to think through the implications for our understanding if/when we apply this to other models.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/weecology/bbs-forecasting/issues/61#issuecomment-238928490, or mute the thread https://github.com/notifications/unsubscribe-auth/AAzdCZn4SfG4rJU81KNqbRfuHf7cTG16ks5qegCogaJpZM4JgeJe.

Thanks Dave. Yeah, that works. We can revisit come interpretation time if it comes up.

Would you be able to send a PR that adds observer ID as a new column?

Yep

@sdtaylor Once you add the observer column, could you email me any csv files in your ./data folder? Postgres is being difficult on my machine. Thanks so much!

Papers/chapters on observer effects, via @ethanwhite:
https://sora.unm.edu/sites/default/files/journals/auk/v111n01/p0050-p0062.pdf
https://pubs.er.usgs.gov/publication/5210391

I just updated the title to reflect the fact that we're now convinced that at least some of these are observer effects and we'd like to address them.


Is it possible to add a description to nodes?

Sorry for all the questions! I tried searching, but didn't see a way to add descriptions to a node on tooltip or something. Would be very useful once you get a lot of custom nodes added in

All the Unity UIElements have a tooltip field if that's what you're looking for. When you assign the tooltip of a node view it shows up below the node like this:

https://docs.unity3d.com/ScriptReference/UIElements.VisualElement-tooltip.html

How do you get it below the node? Do you need a custom node view class for each node to give it a tooltip, or is there some attribute that can be applied to a node?
Basically I'm looking for a way to have an on hover tooltip if possible, but always below works too for now, and ideally a way to see the tooltip in the create node menu too

Yeah, with a custom view. I recommend you create a base custom view that all your node inherit, this will give you a lot more control over your nodes. Like I did in my Mixture project: https://github.com/alelievr/Mixture/blob/master/Packages/com.alelievr.mixture/Editor/Views/MixtureNodeView.cs

For the node create menu I don't think it's possible to have a tooltip in it. I'm using the ISearchWindowProvider for it and it doesn't seem to support tooltips

Is it not possible to do this in a way that doesn't require a custom node view, ie using a Description attribute that is put on your classes that derive from BaseNode?
For the search window, is it possible to add a second line of text in a smaller font or something? So you could place the description below it

Is it not possible to do this in a way that doesn't require a custom node view, ie using a Description attribute that is put on your classes that derive from BaseNode?

You can do this, but you'll have to modify NodeGraphProcessor to add it in this case.

For the search window, is it possible to add a second line of text in a smaller font or something?
No idea, I never tried. But the API seems very limited (i.e. you can only give a GUIContent for an item and that's it) so it's unlikely that it'll work.

Thanks, I just added it in using a custom node view


Finalizing fuzzer

woke init

types - use script from last woke fuzzer project
tests - template will be given

rules -> flows
rule -> flow
3x warning -> info

build artficats not available
config file not found
global config file not found

insert(0, sys.path) rather than append
from woke import Fuzzer

Done.


Implement scripts/module to support Node.js

Implementation of this feature is on hold now. Please, see PR #133 for more details.

Implementation of this feature is on hold now. Please, see PR #133 for more details.


findcontours only returning one path for multiple seeds?

I seem to be only getting one path back (it's a nice one) from findcountours when I am passing in multiple seeds. Should we not see multiple paths coming out? Very likely user error, still learning this system.

Also, I am using findcontours to recover the 2d hull around each seed in a voronoi graph. I assume this is a canonical use? I tried to extract individual hulls from the voronoi directly but not clear how to.

So, it's indeed the canonical way of getting the Voronoi cells! (if there is one ahah) -- however the findcountours suffers from some legacy code, I'll update ASAP to have a single target input, with each individual point being a seed.
At the moment, it's using collections as seed, which is bad. You basically need to create individual partitions from your points in order for them to act as individual seeds. I added an example in the example project (EdgesAndGraphs/FubdVoronoiCellContours)

However as you can see, some contours are picked on the wrong side: it looks like some seeds are missing their associated contour, but it's in fact creating duplicate contours from neighboring cells >.<

So bottom line: your approach is the right one, I'm the one who fucked up.

haha so it was good input I'm happy thanks again
I thought it was partionsomething

user error, better version coming anyway :D


Upgrade spleeter to TF v2.1

[Spleeter-#197] - Convert to TF v2.1
Description
Upgrading scripts to work with TF v2.x
Conversion utility used:
tf_upgrade_v2 --intree spleeter --inplace --reportfile report.txt
Installed Tensorflow Version: v2.1
How this patch was tested
python -m spleeter separate -i audio_example.mp3 -o output -B tensorflow
Documentation link and external references
Please provide any info that may help us better understand your code.

Hi @jinay1991
We are considering migrating to TF2 at some point. Can you share with us why you closed this PR ?

This PR had converted code with TF upgrade utility though after converting it wasn’t working hence I deleted PR.


Handle gopkg.in paths with e.g. "unstable" suffix

First, quick background - gps has a set of "deducers," which are the way that it extracts project roots from import paths. These are entirely analogous to what go get uses today. However, we need more from them than does go get: these deducers also set up logic that will create and ultimately manage interaction with a source repository on disk (which is kept in the cache area).
I implemented specialized handling in a custom deducer for gopkg.in, rather than relying on the HTTP go-get metadata, for several reasons:

Most importantly, because it was the only way to translate as much as possible of the constraint system gopkg.in creates into gps' terms
It let me normalize the import paths that differed only by constraint (e.g. gopkg.in/yaml.v2 and gopkg.in/yaml.v1) into the same underlying source. Two notes on this:
a. I'm not fully sure this was the right decision, and could totally be convinced to back it out
b. I can't remember exactly how far I got with doing this - it's a very awkward pattern that kinda breaks the model
To eliminate unnecessary HTTP requests (gopkg.in's patterns are predictable, so why not)

The error @rogpeppe reported in golang/dep#126 comes from here, where we're setting up the logic that will later create a source for the named path. (The comment is incorrect - the regex isn't wrong, it's just that I didn't handle the suffix case there.) The real issue that needs addressing is noted in the code that actually retrieves a list of possible versions from the source (repository), though - we just don't handle the unstable suffix.
IIRC, the basic issue is that I just don't have a clear sense of how we'd even map the notion of "unstable" onto semver/a version list. I also may have just kicked this can down the road. Either way, though, we need a fix now.

@kris-nova this is the one you were looking at, right? please don't hesitate to ask questions, or even just to spitball - whatever helps you get over the hump with it!

AFAICT, 1.2.3-unstable falls under Semver's notion of pre-release version. eg. 1.2.3-alpha1.

It definitely does. I think the question in my mind when I initially implemented this was, given that gopkg.in treats these import suffixes as general constraints rather than exact matches, should an -unstable suffix match just the literal suffix -unstable, or some wider set of versions.
Looking at niemeyer/gopkg#25, though, it seems like the exact matching is the way to go. So I think that makes this simpler :)

Any news on this?

Nope, nobody's showed up yet to help. This is lower priority for me to work on myself than other things. If it has to fall to me to get done, it'll probably be a couple months.

Ah sorry, I totally missed my mention on this! I can try to take a look sometime next week, and follow up then!

no problem :)

This issue was moved to golang/dep#425


Fix some remarks

Tested. After opening motion create form I get still the error "Cannot read property 'value' of undefined". Can you please check it?


vendor: Update virtcontainers vendoring

This vendoring updates pull the following patches:
df324ac network: Factorize common code between network implementations
9726597 network: Fix wrong MAC address configuration
fcceae7 README: Change shim path in virtc README
7dd546e networking: Turn off multicast snooping on integration bridge
782e907 CONTRIBUTING: Fix the md links
Notice that it fixes the network in two ways

Disabling multicast snooping on the bridge inside the netns
Fixing the MAC address provided to the VM

The runtime is now able to ping IP addresses on the host.

Coverage remained the same at 37.192% when pulling e5e9b0435584fb2571d7a50e5019565f8eee478c on sboeuf/vendor_virtcontainers into 051d22051dd09045eac04534e23b2e0ab2f4e344 on master.

LGTM


Refactor/min max dimension

Description
AutoLayout specific widget logic such as setting min/max dimension is set by making use of a global class .auto-layout that is set in the main canvas. This leaks the idea of platform specific construct "auto-layout" into widget component.
This PR resolves that by making use of necessary props to the widget component
PR fixes following issue(s)
Fixes #24492

if no issue exists, please create an issue and ask the maintainers about this first

Media

A video or a GIF is preferred. when using Loom, don’t embed because it looks like it’s a GIF. instead, just link to the video

Type of change

Refactor

Testing

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Also list any relevant details for your test configuration.
Delete anything that is not relevant

[x] Manual
[ ] Jest
[ ] Cypress

Test Plan

Add Testsmith test cases links that relate to this PR

Issues raised during DP testing

Link issues raised during DP testing for better visiblity and tracking (copy link from comments dropped on this PR)

Checklist:
Dev activity

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[ ] Speedbreak features have been covered
[ ] Test plan covers all impacted features and areas of interest
[ ] Test plan has been peer reviewed by project stakeholders and other QA members
[ ] Manually tested functionality on DP
[ ] We had an implementation alignment call with stakeholders post QA Round 2
[ ] Cypress test cases have been added and approved by SDET/manual QA
[ ] Added Test Plan Approved label after Cypress tests were reviewed
[ ] Added Test Plan Approved label after JUnit tests were reviewed

/ok-to-test

/ok-to-test

/build-deploy-preview env=release


Updated link to 32-bit Windows installer; added alternative to 64-bit

Nice, thanks @schajee!
I've gave @hermwong a heads up to update the the link on the next Desktop release.
@hollyschinsky once this is live, you may want to weigh in on formatting of the two Windows installers.

Screenshot from production:


Use OSSB and structured bandits for sparse bandits

[x] work on OSSB with the optimization problem from the sparse bandit paper
[x] code that
[x] experiment
[ ] find problems where SparseUCB (and variants) beat all non-sparse-aware algorithms, and SparseOSSB beats non-Sparse OSSB

For example:


feat(schema): for hashes

for #293

does not seems any of failed CI issues relates to my code, locally cargo test was success

@frol @dj8yfo may you help me?
error: unnecessary use of `get(param).is_some()`
   --> borsh-derive/src/internals/generics.rs:159:45
    |
159 |             if associated_type_params_usage.get(param).is_some() && !params_set.contains(param) {
    |                                             ^^^^^^^^^^^^^^^^^^^^ help: replace it with: `contains_key(param)`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_get_then_check
    = note: `-D clippy::unnecessary-get-then-check` implied by `-D clippy::all`
    = help: to override `-D clippy::all` add `#[allow(clippy::unnecessary_get_then_check)]`

seems something with CI as my PR has no relation to this code

change is also backward compatible with S = RandomeSeed


add option to disable commiting and pushing

Added an option
noCommit

which is when set to a truthy value, disables the commit and the push tasks.

Thanks @szdavid92, I need a bit more time to dive into this, but I'll get back to this soon.

Thanks for this PR, but I've just released the first beta of v3, which includes this feature. Feel free to check it out and I'm happy to hear any feedback.


Fix deprecated npm packages

I get this when I make run:
matt@mattubuntu:~/Documents/grtp.co$ make run
npm install
npm WARN deprecated<EMAIL_ADDRESS>the module is now available as 'css-select'
npm WARN deprecated<EMAIL_ADDRESS>the module is now available as 'css-what'
npm WARN deprecated<EMAIL_ADDRESS>Use uuid module instead
npm WARN deprecated<EMAIL_ADDRESS>ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
loadDep:node-pngjs → fetc ▐ ╢██████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░╟
WARN engine<EMAIL_ADDRESS>wanted: {"node":"0.8.x"} (current: {"node":"4.2.6"npm WARN deprecated<EMAIL_ADDRESS>Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated<EMAIL_ADDRESS>Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated<EMAIL_ADDRESS>Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated<EMAIL_ADDRESS>graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to<EMAIL_ADDRESS>as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
npm WARN deprecated<EMAIL_ADDRESS>Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN prefer global<EMAIL_ADDRESS>should be installed with -g

I bet this is related to https://github.com/gratipay/grtp.co/issues/176.


build(yarn): upgrade yarn to the version 3.2.2

Why
Yarn version 1.22 is the legacy. This branch of the manager receives only security updates.
How it will affect on our workflow with this repo?
Will not affect. I set up the Yarn not to use Zero-installs and PnP features. This will help us to more easily migrate to the new version. Also, we can practice working with these features in the future.
How to set up my workspace for this repo?
You don't need to do something special. Current version of the Yarn supplied with the repo. Just check your version in the repo:
$ yarn -v
> 3.2.2

no need to merge cause no meaningful reasons to do it now


Kubernetes configuration

[ ] @Archinowsk Optimize Docker build times and image sizes using multistage build
[ ] CI/CD

[ ] Which runners to use? If Con2 organization runners, need to move repo to con2 org. Can also set up organization runners for Ropekonsti org.
[ ] @japsu actions-runner-controller in Qb kubernetes is aeons old. Upgrade to current version

[ ] @Archinowsk Check environment variables

Together in another session:

[ ] How to use secrets
[ ] MongoDB in production: Use Atlas or set up backups etc in local MongoDB
[ ] Use ropekonsti.fi domain in production, test.ropekonsti.fi in staging

Multistage build, CI failing but should be pretty close: https://github.com/ropekonsti/konsti/pull/763

Merged manually into development


[REEF-300] HDInsight Job Submission no longer works

This addressed the issue by
* Updating Java objects to conform to REST API specified by YARN 2.6.
* Fixing TANG configuration merge issues in HDInsightRuntimeConfiguration
* Adding tests for serialization/deserialization for JSON objects using examples from https://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
JIRA: REEF-300

@afchung thanks for the PR!
I did a mostly cosmetics-focused pass. The main points are:

Add JavaDocs to all new classes. If the class is just an implementation of a REST object, please add that information to the javadoc as well as a link to the YARN API docs, if any. If the class is genuinely ours, please add javadocs to all public members.
Make whatever you can final. IntelliJ has a Code Inspector to help you with that.
The code already had hand-written .toString() methods for JSON objects. Instead of repeating that probably bad idea, maybe we can use jackson to generate the strings?

Of course, all these changes only apply to new or updated code. It is appreciated, but not necessary to fix my mistakes from the past :-)
Please commit additional changes into this same branch. The final committer will take care of the squashing and commit message cleanup. Please comment on this thread when you want us to take another look.

Made the changes. @markusweimer if you could please take another look. Thanks!

@afchung did you test this on HDI? If so, I can go ahead and merge it.

Does REEF have HDI clusters I can use? I haven't tested against an HDI cluster yet, only against the examples provided by the Hadoop docs.

@jwang98052  can probably hook you up with one already running. Otherwise, I can start one for you.

Mine is expired. Do you want to test it on 3.1 or 3.2?

The new submission serialization only works on 3.2, so a 3.2 cluster would be good. Thanks!

@afchung , are we assuming that pre-3.2 cluster will no longer be supported with this change?

Yes. However, note that clusters earlier than 3.2 that are available from the Azure portal do not actually support REST job submission. ASA and REEF were only able to do it because we got a patch from the HDInsight team to expose the REST submission API. Also, the configuration were misconfigured before my change (the submission handler was setting the same parameter in two different configurations and merging them, causing a conflict), so it was not working before anyway.

OK. Thanks for the confirmation @afchung

@yingdachen the REST interface to YARN is a rather novel feature. Hence, I think we can forgo any backwards compatibility here.

Trying it out now, but it seems like the JobJarMaker is does not package the Launcher class into the JAR, so the job fails to start on HDInsight. Am I missing anything here?

You probably need to run the example from the command line, without Maven.

Great, that worked. Thanks!

Does that mean that you were able to run things on HDInsight?

Yes, I was able to submit HelloHDInsight, and verified the output on the cluster.

A problem I found was that I was not able to submit to an HDInsight cluster if I was not on a Windows machine, since the submission detects the OS you're submitting from and builds the java classpath based on the submission OS instead of the execution OS. HDInsight only takes the classpath separator ";", while UNIX based clusters only take ":", so Windows machines won't be able to submit jobs to a UNIX-based cluster either. Perhaps the classpath should build built based on a configuration instead of detecting the submission OS?

Yes, that is a known problem: REEF-63.
However, in the case of HDI, we always submit to Windows, so we should be able to get away with a specifc encoding.

Perhaps the classpath should build built based on a configuration instead of detecting the submission OS?

That is what we do today via the ClassPathProvider classes.

@markusweimer Is this ready to merge?

Does it work if the submission is done from something other than Windows? If not, we should at least file a JIRA on it and link it from REEF-63. Back when we first introduced the HDI support, it did work from UNIX.

Currently it does not, but it's probably just that particular test. @markusweimer Filed REEF-322 and linked to REEF-63.

OK. I will merge this.


Publish button gets stuck when publishing multiple files in a row

the client gets stuck  like this regularly when trying to publish multiple files in a row. I don't know if this is a bug or if it is on my side, but I could not see why it would be on my side.

Uploads are completed before I attempt another.
It only happens after I publish 2 or 3 files. I  can cancel and retry
and another bug in the GUI: when trying to publish with a bid of 0, the user is notified that he must complete the bid field, but the button 'Publish' turns to 'Publishing' and is not accessible to the user, leaving him only the option of cancelling and going through the whole process again
Notify adir92 in Slack when it's fixed.

I think this bug can't happen in this specific form with the new fast publish system, since it always creates the entry in the publish cache--i.e. it doesn't wait for publish to return before it sends you to the My Files page.
It feels like this could be a lbrynet bug. @jackrobison: is there any reason that doing several publish attempts in succession could result in calls to publish ? Like, maybe there's a race condition if you publish a file while the last one is still being reflected? (Bear in mind this bug is from late March.)
Also, the Redux branch completely overhauls our state management through the whole app, so if this is somehow still happening and it's on the client side, there's a good chance Redux will make this go away on its own.


[HOLD #33146] [$500] Workspace - Counter in WS invite member tab is white and hardly visible

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: v1.4.14-6
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:

Go to URL https://staging.new.expensify.com/
Login with any account
Go to Settings -> Workspace -> Open any WS -> Members -> Invite
Choose more than 8 users -> Next

Expected Result:
The counter should be black for good visibility
Actual Result:
Counter is white and hardly .visible
The same behavior with the counter in the group chat details page.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01a90cec2c60d608fb
Upwork Job ID:<PHONE_NUMBER>264674816
Last Price Increase: 2023-12-21

Thanks @Krishna2323 , put on hold, removed Help wanted and making a Weekly

https://github.com/Expensify/App/issues/33146 is still open

This has been resolved

Looks like we're fixed! Thanks @eVoloshchak , closing


Showcase performance

Hello,
It would be useful to have a benchmark in the README of the performance of the framework for some micro examples.

Thanks! Any suggestions for what micro examples to consider?

I found this for checking {con,per}formance for different C/C++ frameworks. It's possible to take out the json and run it against serde. I could work on it but I'm new to Rust and Serde.

I have an implementation of that benchmark in https://github.com/serde-rs/json-benchmark. Currently Serde is 4 to 8 times faster than rustc-serialize and up to twice as fast as the fastest C / C++ library according to those benchmarks.
Over the next few days I will be updating that repo to Serde 1.0 and zero-copy deserialization which I expect will be a nice speedup.

Awesome, thanks for the quick replies! Would you mind putting this somewhere visible to the newbies? :)

Yep, I'll take care of that when it's up to date.

another newbie here: it would also be great to have information about the relative performance of the various formats supported.


Problem about long stop when processing a given audio file

I run the code below to process a given audio file and change stdout to add output to a list.
from whisper_live.client import TranscriptionClient
import sys

import sys
class ListStream:
    def __init__(self):
        self.data = []
    def write(self, s: str):
      if s != '\n' and not s.startswith('[INFO]'):
        self.data.append(s)
# change stdout
sys.stdout = x = ListStream()

client = TranscriptionClient(
  "<IP_ADDRESS>",
  9090,
  lang="en",
  translate=False,
  model="small",
  use_vad=False,
)

client.play_file("./output.wav")
client.close_all_clients()
sys.stdout = sys.__stdout__
print(x.data[-1])

It runs well, but I found that after the audio file was processed, it waited for about 10s to end, and then it print x.data[-1] to stdout. I can't find out the reason. I wonder if I can do something to avoid the stop.
Moreover, I've got another problem. When I change client.play_file("./output.wav") to simply client() to record my voice, I found that the server can't detect if I stopped saying. After I speak, I have to use "Ctrl+C" to stop the client and print the data. It seems client.close_all_clients() does not work. I wonder if there will be a solution.

@KujoStar the reason we wait for 10s [here]https://github.com/collabora/WhisperLive/blob/cb392cbb934447ce8021c0eba06ecbd186bdf13d/whisper_live/client.py#L396
is if any response from the server hasnt been received yet(in case the server is slow; runs on cpu).
Yes, for the recording client to stop the way we stop processing for now is by a KeyboardInterrupt, else its always listening.


Merge pull request #1 from googlei18n/master

Update test metadata (#1514)

This change is

Empty PR, did you mean to update your own fork?


Sass styles are nested too deeply

The compiled CSS results in selectors like the following:
#footer .footer-meta .footer-meta-inner .open-government-licence .logo {
    ...
}

It doesn't really seem as though any CSS naming convention is being used, which for large, production-ready services, should be the case.
I think an entire re-write of the Sass/CSS using a consistent, sensible convention is required at some point, though the implications on the markup could mean that this would be a very big task.
At the very least, I think the current Sass could be re-worked without having to modify the markup.

This stuff is pretty old at this point, and naming conventions weren’t as common then as they are now. We are currently undertaking a reworking of parts of the frontend code that we produce, and we’re absolutely looking at selectors as part of it. This work will be a break from the old, so it’s an ideal time to establish better conventions.

Great news, if you need any help I'd be more than happy to assist with anything at all.


Error: Direct kernel connection broken

Issue Type: Bug
Open any file.ipynb and attempt to run a cell.
print('hello')
expect output : hello
receive:  'Error: Direct kernel connection broken'instead with the following details.
Error: Direct kernel connection broken
at getDisposedError (c:\Users\forresr2.vscode\extensions\ms-python.python-2020.8.103604\out\client\extension.js:24:684530)
at f.P.handleCodeRequest (c:\Users\forresr2.vscode\extensions\ms-python.python-2020.8.103604\out\client\extension.js:48:467398)
at c:\Users\forresr2.vscode\extensions\ms-python.python-2020.8.103604\out\client\extension.js:48:480019
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:85:5)
Extension version: 2020.8.103604
VS Code version: Code 1.48.2 (a0479759d6e9ea56afa657e454193f72aef85bd0, 2020-08-25T10:13:11.295Z)
OS version: Windows_NT x64 10.0.17134

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz (4 x 2496)

GPU Status
2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: disabled_offprotected_video_decode: unavailable_offrasterization: enabledskia_renderer: disabled_off_okvideo_decode: enabledviz_display_compositor: enabled_onviz_hit_test_surface_layer: disabled_off_okwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
7.88GB (3.46GB free)

Process Argv

Screen Reader
no

VM
0%

@GitForrest Sorry about the issue and thanks for reporting. Could you please change the VS Code setting Python->Logging->Level to debug and rerun your scenario? Then after reproing the error copy out the contents of the Output:Python window. You can save that in a file and attach it to a comment here. That has the diagnostic info that we would need to look into this.

I also faced the same problem, when I loaded a large numpy array in the memory.

Faced the same issue - a large final sum was loaded in memory as a return from function.

Same error and trace output. I'm running the kernel in a virtualenv on a remote (ssh) Linux host.  I don't get this error when I run the Jupyter kernel directly on my laptop.  It's likely that the port Jupyter wants to use is being blocked, but I don't see what port that is, yet.
Error 2020-10-23 19:33:15: Error executing a cell:  [Error: Direct kernel connection broken
    at m.getDisposedError (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:32:193270)
    at f.P.handleCodeRequest (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:56:467782)
    at /dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:56:480340
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:94:5)]
Error 2020-10-23 19:34:23: [Error: Cannot execute code, session has been disposed.
    at v.restart (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:1:546462)
    at f.restartKernel (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:56:472314)
    at f.c.value (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:1:10156)
    at f.restartKernel (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:32:152870)
    at f.restartKernelInternal (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:32:95563)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:94:5)
    at async f.restartKernel (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:32:85205)]
Error 2020-10-23 19:34:26: Error executing a cell:  [Error: Direct kernel connection broken
    at m.getDisposedError (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:32:193270)
    at f.P.handleCodeRequest (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:56:467782)
    at /dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:56:480340
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:94:5)]
Error 2020-10-23 19:36:09: Error executing a cell:  [Error: Direct kernel connection broken
    at m.getDisposedError (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:32:193270)
    at f.P.handleCodeRequest (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:56:467782)
    at /dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:56:480340
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:94:5)]
Error 2020-10-23 19:36:14: [Error: Cannot execute code, session has been disposed.
    at v.restart (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:1:546462)
    at f.restartKernel (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:56:472314)
    at f.c.value (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:1:10156)
    at f.restartKernel (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:32:152870)
    at f.restartKernelInternal (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:32:95563)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:94:5)
    at async f.restartKernel (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:32:85205)]
Error 2020-10-23 19:36:17: Error executing a cell:  [Error: Direct kernel connection broken
    at m.getDisposedError (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:32:193270)
    at f.P.handleCodeRequest (/dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:56:467782)
    at /dogfood-home/eng_home/phord/.vscode-server/extensions/ms-python.python-2020.9.114305/out/client/extension.js:56:480340
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:94:5)]

I have the same issue. any solution?

Getting same error - mainly with running matplotlib. Have to restart vscode to get kernel back.

Same problem here, restarting doesn't help in my case.


Discord Join Messages sent to Minecraft

The join messages sent to discord also show up in Minecraft with the discord formatting/escaping.  This looks really bad and was not the original behavior IIRC.  Personally, I don't need this plugin to send join messages in game at all and most servers will also likely have another plugin taking care of the MC join message.
Can you please fix the formatting and give us the option to shut of the join message on a specific side?

Please see #11


Added Microsoft Edge Beta version 91.0.864.27.

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

[x] Does your manifest conform to the 1.0 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

/azp run

/azp run

/azp run


Clarification for "output.libraryTarget" documentation

From @Daniel15 on December 1, 2015 5:20
The documentation for output.libraryTarget says:

If output.library is not set, but output.libraryTarget is set to a value other than var, every property of the exported object is copied (Except amd, commonjs2 and umd).

The "Except amd, ...." at the end confuses me, since var, amd, commonjs2 and umd are four of the six available library targets, and they're all listed in the same sentence :smile: . Does it actually mean this:

If output.library is not set and output.libraryTarget is set to this or commonjs, every property of the exported object is copied

Or does it mean that  amd, commonjs2 and umd are treated some different way altogether, and thus there's really three separate groups here:

var: Does nothing special
this and commonjs: Exports every property individually
amd, commonjs2 and umd: ???

Copied from original issue: webpack/webpack#1690

From @sokra on December 2, 2015 17:50
amd: anonymous amd module.
umd:  not useful
commonjs2 ignores the output.library property.

From @sokra on December 2, 2015 17:50
Could you update the documentation?

From @Daniel15 on December 2, 2015 18:9

Could you update the documentation?

Sure :) The reason I posted this issue was to make sure I completely understand how the libraryTarget option works, before updating the documentation. I didn't want to add any incorrect information.

Closing because looking into the new documentation, this has been fixed:

If output.library is not set, but output.libraryTarget is set to a value other than var, every property of the exported object is copied (Except amd, commonjs2 and umd).


Meter and Statistic Unavailable

The integration is unable to pull usage data from the meter, other sensors pull in fine. I am able to see all data from the eyeonwater sources directly.
Logger: homeassistant
Source: components/recorder/statistics.py:2230
First occurred: 9:49:21 AM (3 occurrences)
Last logged: 9:58:32 AM
Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 291, in async_refresh
await self._async_refresh(log_failures=True)
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 414, in _async_refresh
self.async_update_listeners()
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 168, in async_update_listeners
update_callback()
File "/config/custom_components/eyeonwater/sensor.py", line 130, in _state_update
self.import_historical_data()
File "/config/custom_components/eyeonwater/sensor.py", line 157, in import_historical_data
async_import_statistics(self.hass, metadata, statistics)
File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 2230, in async_import_statistics
raise HomeAssistantError("Invalid statistic_id")
homeassistant.exceptions.HomeAssistantError: Invalid statistic_id
Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:580
integration: Sensor (documentation, issues)
First occurred: 9:49:21 AM (3 occurrences)
Last logged: 9:58:31 AM
Error adding entity sensor.water_meter_REDACTED_temperature for domain sensor with platform eyeonwater
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 580, in _async_add_entities
await coro
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 892, in _async_add_entity
await entity.add_to_platform_finish()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1359, in add_to_platform_finish
self.async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1009, in async_write_ha_state
self._async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1132, in _async_write_ha_state
state, attr, capabilities, shadowed_attr = self.__async_calculate_state()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1067, in __async_calculate_state
state = self._stringify_state(available)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1015, in _stringify_state
if (state := self.state) is None:
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/sensor/init.py", line 543, in state
value = self.native_value
^^^^^^^^^^^^^^^^^
File "/config/custom_components/eyeonwater/sensor.py", line 188, in native_value
return self.meter.meter_info.sensors.endpoint_temperature.seven_day_min
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'endpoint_temperature'

@zackaryssmith thank you for reporting the issue.
Is it a new issue? or you just installed the extension for the first time?

@kdeyev I had it previously installed but had removed it due to the usage resulting in a negative number. This issue is appearing from a reconfiguration of the integration.
Odd behavior, if I change the unit for the stats sensor I am able to get a number but it doesn't seem to have updated in the last week.

@zackaryssmith Could you please try to downgrade to extension version v2.5.6?
I think it may be a regression in the pyonwater module

@kdeyev I spent some time downgrading versions to see when it last worked and it seems that for my issue it was 2.5.0. Curiously, this is the update before the Statistics sensor was added. I still need to toggle the units to get any usage to show and the history import service does not work but it never has in my case.
I also noticed that in the energy dashboard configuration, I have the following errors with the current versions:

The warning messages are expected: there is short explanation in the extension wiki.
But I'm puzzled with the error messages you shared, my best guess there is a problem with the meter ID parsing. If you have some programming experience - it would be very helpful if could run the pyonwater example with your credentials.
Thank you!

@zackaryssmith I cannot spot any problem: the output looks as expected.
I still suspect that it's may be something odd with the meter UUID:
homeassistant.exceptions.HomeAssistantError: Invalid statistic_id

Could you please check if it includes spaces or any special symbols? My meter UUID looks like: '200010106', and the meter sensor id: 'sensor.water_meter_200010106', statistic id: 'water_meter_200010106_statistic'.

@kdeyev When I run the example.py, I get a meter_uuid='19 digit number'.
My water meter name in home assistant looks like "Water Meter 200010106+2019-07-15" with a entry ID of "sensor.water_meter_200010106_2019_07_15_2 "
The statistic entry name is "Water Meter 200010106+2019-07-15" with a entity ID of "sensor.water_meter_200010106_2019_07_15"

@zackaryssmith It seems that HA takes care of the special symbols. I can try to normalize the UUID by replacing '+' and '-' but '_', but I'm not really sure that it's the problem.
BTW, for energy dashboard you need to use an entity with '_statistic' suffix.

@kdeyev No worries, seems like it is specific to my meter/ install for now. Thank you for taking the time to look into it, I appreciate it.

@zackaryssmith please try this one: https://github.com/kdeyev/eyeonwater/releases/tag/v2.5.8-beta.4
It's prerelease version, so you may need to check "show beta" checkbox: https://hacs.xyz/docs/faq/select_version/

@kdeyev No luck. Statistic sensor still shows as unavailable.
From the debug log when reloading the service:
2024-06-10 16:34:44.598 DEBUG (MainThread) [custom_components.eyeonwater.coordinator] Discovered 1 meter(s)
2024-06-10 16:34:44.599 DEBUG (MainThread) [custom_components.eyeonwater] Fetching latest data
2024-06-10 16:34:44.603 DEBUG (MainThread) [custom_components.eyeonwater.statistic_helper] last_stats {}
2024-06-10 16:34:44.604 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.water_meter_123456_2019_07_15_temperature for domain sensor with platform eyeonwater
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 600, in _async_add_entities
await coro
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 914, in _async_add_entity
await entity.add_to_platform_finish()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1355, in add_to_platform_finish
self.async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1000, in async_write_ha_state
self._async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1126, in _async_write_ha_state
state, attr, capabilities, shadowed_attr = self.__async_calculate_state()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1061, in __async_calculate_state
state = self._stringify_state(available)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1006, in _stringify_state
if (state := self.state) is None:
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/sensor/init.py", line 543, in state
value = self.native_value
^^^^^^^^^^^^^^^^^
File "/config/custom_components/eyeonwater/sensor.py", line 194, in native_value
return self.meter.meter_info.sensors.endpoint_temperature.seven_day_min
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'endpoint_temperature'
2024-06-10 16:34:45.234 DEBUG (MainThread) [custom_components.eyeonwater] Finished fetching EyeOnWater data in 0.635 seconds (success: True)
2024-06-10 16:34:45.235 DEBUG (MainThread) [custom_components.eyeonwater.statistic_helper] last_imported_time None - data 2024-06-10 10:00:00-07:00
2024-06-10 16:34:45.235 INFO (MainThread) [custom_components.eyeonwater.statistic_helper] 59 data points found
2024-06-10 16:34:45.235 INFO (MainThread) [custom_components.eyeonwater.sensor] 59 data points will be imported
2024-06-10 16:34:45.235 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved (None)
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 288, in async_refresh
await self._async_refresh(log_failures=True)
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 411, in _async_refresh
self.async_update_listeners()
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 165, in async_update_listeners
update_callback()
File "/config/custom_components/eyeonwater/sensor.py", line 133, in _state_update
self.import_historical_data()
File "/config/custom_components/eyeonwater/sensor.py", line 160, in import_historical_data
async_import_statistics(self.hass, metadata, statistics)
File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 2235, in async_import_statistics
raise HomeAssistantError("Invalid statistic_id")
homeassistant.exceptions.HomeAssistantError: Invalid statistic_id
From the System Logs:

Popping in here to state that my meter is named in the same format and I am having the same issue with custom integration. Let me know if there is a faster way to help debug and any additional info you require.

@zackaryssmith sorry for the delay.
@hrana welcome to the thread.
I've looked how HA core verifies the ids and added similar check in the integration:
    if not valid_entity_id(statistic_id):
        raise Exception("Invalid statistic_id {statistic_id} for meter {meter.meter_id}")

Please try this one: https://github.com/kdeyev/eyeonwater/releases/tag/v2.5.8-beta.7

@kdeyev It is pulling data when on beta 7.
I get this in the debug log but I don't see anything failing, I even get historical stats. Thank you for all the trouble you've gone through on this.

2024-06-25 09:22:54.797 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open inside the event loop by custom integration 'eyeonwater' at custom_components/eyeonwater/coordinator.py, line 43: await meter.read_historical_data( (offender: /usr/local/lib/python3.12/site-packages/pytz/__init__.py, line 108: return open(filename, 'rb')), please report it to the author of the 'eyeonwater' custom integration
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 672, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 639, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1988, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 288, in async_refresh
    await self._async_refresh(log_failures=True)
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 312, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 268, in _async_update_data
    return await self.update_method()
  File "/config/custom_components/eyeonwater/__init__.py", line 52, in async_update_data
    await eye_on_water_data.read_meters(days_to_load=3)
  File "/config/custom_components/eyeonwater/coordinator.py", line 43, in read_meters
    await meter.read_historical_data(

@kdeyev Amazing! It's pulling data for me as well. Thank you again so so so much for providing the support.
@zackaryssmith I am seeing similar errors in my logs but, like yours, mine is working.

fixed in https://github.com/kdeyev/eyeonwater/releases/tag/v2.5.7


R package - progress messages

I have switched over from the python package to the R package and I'm running the Kaggle example. I'm running R from jupyter
When I'm fitting the network, the epoch progress messages do not appear until all of the epochs have finished running. Is there any way for me to see the results of each epoch as soon as that epoch is complete?

@colinpriest I have to admit that I have never tested it on Jupyter. Let me have a look and come back to you.

@colinpriest Just confirmed this problem. But to be frank, I think this is an issue in IR kernel.
I will talk with these people.

Thanks. I haven't been able to get the script to run in R Studio yet because it can't find the cudart.so library. I suppose I will need to figure out why IPython can find the library, but R Studio cannot, so that I can switch to running mxnet in R Studio.

I guess you must be using the GPU.
The simplest solution: startup RStudio from terminal. On Linux, just rstudio; on Mac, /Applications/RStudio.app/Contents/MacOS/RStudio

I guess rstudio must have some configuration on env variables, maybe consider add this to  a new R package trouble shootting section in here ttps://github.com/dmlc/mxnet/blob/master/doc/build.md

Thanks. Starting R Studio that way works! :-)

Theoretically, the CUDA path can be added in ~/.Renviron. When R starts, it will try to find ~/.Renviron and load it.


Update pandas to 2.2.2

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 92.679%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
519

Relevant Lines:
560

💛  - Coveralls


[lit-html] Allow styleMap values to be nullable

Fixes this error:

Here's the spec for CSSStyleDeclaration.setProperty. It appears to treat null and '' (empty string) differently. Step 3 says an empty string value triggers removeProperty. Steps 6 says that after parsing the value, just return void if it's null (no removeProperty).
I'm not sure the reasoning behind making null a noop (if that's indeed the case). Of course it's a spec not a user guide.
In any case, the idea is that noops have no place in declarative code. To be declarative, the UI must match the declaration. So, if you declare the style as "empty" (null or undefined), then that style should become empty (unset).
This is the current behavior in plain JS, but typescript forbids undefined.

Interesting. I think this is fine and a good tightening of behavior. We need to have a test though.

Fixed with https://github.com/lit/lit/pull/1665