Exposure as Transport

Hi @zxdavb
Not an issue, but I'm the author of evogateway/listener - you may recall we had a brief chat a few months back in the automatedhome forum. I was wondering if you've had any further thoughts on exposing your library as a transport. No urgency!
Many thanks.

Hi @smar000
It is fully exposed as a transport, using the standard asyncio Protocol/Transport model.
Have a look at client.py, or - more simply:
from evohome_rf import Gateway

def process_message(msg) -> None:
    # e.g. msg.payload is the the message dict, msg.src.id is the source device address
    dtm = f"{msg.dtm:%H:%M:%S.%f}"[:-3]
    print(f"{dtm} {msg}")

async main(serial_port, **kwargs):
    gwy = Gateway(serial_port, **kwargs)
    protocol, _ = gwy.create_client(process_message)

    await asyncio.create_task(gwy.start())

if __name__ == "__main__":
    asyncio.run(main("/dev/ttyUSB0"))

The idea is - as I'm sure you'll now - is to add your evohome_rf --> RESTful gateway in process_message().
Receiving messages (an inbound msg) is pretty good, but sending commands (an outbound msg) isn't fully fleshed out yet - you can use gwy.send_data("RQ ...") for now.
There are a shedload of config parameters available (see lib_kwargs in client.py and also schema.py) - reduce_processing and the allow/block lists are a good start, I guess.
Workflow is a bit broken, presently: I'll tidy up the master branch today/tomorrow & move dev to another branch.
Any questions, just ask!

Thanks for the quick response back, and that's great. I haven't actually had a chance to look at your latest code but will hopefully do so over the weekend.
Thanks again and I'm sure I will have questions... :)

No worries: I'd be very happy to see someone else using it!

You have clearly done a tremendous amount of work - really impressive!!
One small issue - I am getting the following error/stack trace:
Traceback (most recent call last):
  File "./evohome2mqtt.py", line 65, in <module>
    asyncio.run(main("/dev/evoftdi"))
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "./evohome2mqtt.py", line 59, in main
    await asyncio.create_task(gwy.start())
  File "/opt/scripts/evohome/evohome_rf/evohome_rf/__init__.py", line 157, in start
    serial_port=self.serial_port,
  File "/opt/scripts/evohome/evohome_rf/evohome_rf/transport.py", line 720, in create_pkt_stack
    ser_instance.set_low_latency_mode(True)  # only for FTDI?
AttributeError: 'Serial' object has no attribute 'set_low_latency_mode'

Commenting out the respective lines in transport.py allows me to continue, without any noticeable issues. I am using a couple of nanoCUL devices. pyserial-asyncio is version 0.5.

Thanks - I will fix this - it seems your nanoCUL is not FTDI?
I had a look at your code - this library will do a lot of the heavy lifting for you - designed exactly for your use-case. Good luck!

Yes abosolutely - I should be able shrink my code down signficantly.
WRT the latency issue, I tried with 2 nanoCULs. One with an FTDI and one with CH340. Both gave the same issue, so I'm not entirely sure that it is FTDI related.
Anyway, its running without the lines, and so not a big issue!

Hi @zxdavb
I have got a skeleton version of my system working with your library - looking great so far. Would you mind giving me a full sample scheme (I had a look at schema.py but it would be quicker if I had a sample to look at), but haven't been able to get device names to show correctly.
Also are you open to exposing some further properties/attributes, e.g. in Message object, in the __repr__ function, you have some useful bits that could be re-used, e.g, src, dst, code_name, payload from the code below (appropropriately renamed so as not to clash with the main class properties of the same name):
  if self.src.id == self.devs[0].id:
            src = display_name(self.src)
            dst = display_name(self.dst) if self.dst is not self.src else ""
        else:
            src = ""
            dst = display_name(self.src)

        code_name = CODE_NAMES.get(self.code, f"unknown_{self.code}")
        payload = self.raw_payload if self.len < 4 else f"{self.raw_payload[:5]}..."[:9]

Thanks!

I have got a skeleton version of my system working with your library - looking great so far

@smar000 apologies for barging in on the thread, but, I was literally just looking at seeing if evohome_rf could be used for a project to integrate directly with MQTT so your question is great timing.
Currently using an HGI80 on a really old and heavily customised Domoticz install just for evohome. I've been considering migrating to Home Assistant but my whole system is modular built using node-red & MQTT, so while moving to HA would get me an up-to-date evohome integration I'm trying to move away from using a myriad of different smart home software just to provide specific integrations.  I'd much rather have hardware <-> MQTT directly!
Is your evohome2mqtt project public anywhere?  Happy to help with testing in return for access, I've got a spare HGI80 that I used for development in the past so would run with that so as not to impact my live system.

Hi @martynwendon I have an existing evohome -> MQTT system in my repo (evolistener/gateway), where I rewrote a lot of the domoticz packet decoding/encoding to python. That system is working reasonably well.
Having looked at @zxdavb work there though, he has a built a much more robust - and comprehensive - packet decoding/encoding library, and so it makes sense for me to migrate over to his library.
My current 'skeleton' system is no where near ready for public use, and will be many weeks/month or so away before I upload it publicly.

@zxdavb I came across an interesting error a short while ago:
16:17:12.784 000 RQ --- 01:191706 13:102710 --:------ 0008 001 00 < Validation error: Invalid verb/code for 01:191706: RQ/0008
16:17:12.786 044 RP --- 13:102710 01:191706 --:------ 0008 002 00C8 < CorruptStateError
16:17:12.786 handle_exception(): Caught: Exception in callback SerialTransport._read_ready()
16:17:12.786 Unhandled error in exception handler
context: {'message': 'Exception in callback SerialTransport._read_ready()', 'exception': CorruptStateError(CorruptStateError(...), '13:102710 (BDR) has changed controller: 01:139901 to 01:191706'), 'handle': <Handle SerialTransport._read_ready()>}
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/base_events.py", line 1639, in call_exception_handler
    self._exception_handler(self, context)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/__init__.py", line 124, in handle_exception
    raise exc
  File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.7/dist-packages/serial_asyncio/__init__.py", line 106, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/transport.py", line 408, in data_received
    self._data_received(*create_pkt(line))
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/transport.py", line 612, in _data_received
    self._callback(pkt)  # only wanted PKTs up to the MSG transport's handler
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/protocol.py", line 139, in _pkt_receiver
    [p.data_received(msg) for p in self._protocols]
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/protocol.py", line 139, in <listcomp>
    [p.data_received(msg) for p in self._protocols]
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/protocol.py", line 368, in data_received
    self._callback(msg)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/message.py", line 550, in process_msg
    create_devices(msg)  # from pkt header & from msg payload (e.g. 000C)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/message.py", line 378, in create_devices
    this._gwy._get_device(this.src, ctl_addr=this.dst)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/__init__.py", line 234, in _get_device
    device._set_ctl(ctl)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/devices.py", line 201, in _set_ctl
    f"{self} has changed controller: {self._ctl.id} to {ctl.id}"
evohome_rf.exceptions.CorruptStateError: The system state is inconsistent: 13:102710 (BDR) has changed controller: 01:139901 to 01:191706 (try restarting the client library)

I certainly haven't changed controllers, and don't recognise 01:191706 as one of my devices... I don't believe any of my neighbours have an evohome system either.
Any ideas?
Thanks.

Hi @martynwendon I have an existing evohome -> MQTT system in my repo (evolistener/gateway), where I rewrote a lot of the domoticz packet decoding/encoding to python. That system is working reasonably well.

I think I did try this a few years ago but didn't have much success with the HGI80.  I don't recall what the problem was, but I didn't have a spare HGI80 back then so didn't spend too much time on it. I stayed with Domoticz as that was already working.
I'll take another look though since I have the spare HGI80 now so can afford to play around a bit more.

Having looked at @zxdavb work here though, he has a built a much more robust - and comprehensive - packet decoding/encoding library, and so it makes sense for me to migrate over to his library.

Agreed, the progress has been amazing and I've been tracking the thread on the Home Assistant forum for quite some time.  I had a bit of spare time so thought I'd dig into possibly writing my own MQTT direct gateway using the evohome_rf library, hence I spotted your question.

My current 'skeleton' system is no where near ready for public use, and will be many weeks/month or so away before I upload it publicly.

No problem at all, I'll check back at a later date.  In the meantime if you do want anybody to test please let me know.

Many thanks for the above - very helpful, and noted on the likely cause of the error message.

Have you tried to do anything like:

I haven't yet, but I should have the packet logs saved. Let's see how I get on, and if the problem persists, I will get back to you.

Everything you might need should be exposed

Thanks and yes, for the most part I found they are. One specific one that I didn't fine is the code_name in a Message. I had to lookup the name by importing CODE_NAMES from .message and checking against this. Not a big deal TBH, but I thought it might be helpful to have this within the Message class itself.
One other suggestion - I notice that opentherm_msg responses are put into the dictionary keys msg_name and value (plus msg_description and couple of others), e.g.
{'msg_id': '0x19', 'msg_name': 'BoilerWaterTemperature', 'msg_type': 'Read-Ack', 'value': 70.0, 'description': 'Boiler water temperature'
Would it be possible to get these with the msg_name as the parameter combined with its corresponding value, e.g. something like:
{'msg_id': '0x19', 'BoilerWaterTemperature': 70.0, 'description': 'Boiler water temperature', 'msg_type': 'Read-Ack'
The reason for I ask is two fold:

(a) it makes my code slightly neater, as I don't have to specifically test for opentherm_msg and process that differently to other messages (as I break down the json response and post the individual bits to MQTT, the opentherm_msg topic only shows the last msg_name - all the other opentherm messages are of course overwritten, and
(b) it would be more consistent with the other message items - I don't think you have used name and value for any of the other responses.

Again, not a big issue, and I can easily work around it, but thought it would be nice!
Thanks again.

You can use a packet log as a source, rather than a serial port - the will be useful fro test/dev:
cat packet.log | python client.py parse

I think code_name will be a useful addition!

Re: Opentherm - I am driven/limited by other projects, from which I am reluctant to deviate far... I see that JSON as distinct from my schema. E.g. https://github.com/mvn23/pyotgw

Re: Opentherm - I am driven/limited by other projects, from which I am reluctant to deviate far... I see that JSON as distinct from my schema. E.g. https://github.com/mvn23/pyotgw

Thanks and that is fine. After thinking about it, and seeing the different results under opentherm_msg, I think the way you have done it is logical. I have coded to take this into account.
With regards to sending commands, is there any callback that confirms the status of the sending, e.g. command sent/success/fail notifications? I see various packet messages showing retry/expired etc, but haven't (yet) looked to see how it all works in your code.

Another question - in your schema, what is the correct way to define underfloor_heating zone?  In your example above, you gave:
 "01": {
            "heating_type": "underfloor_heating",
            "sensor": "34:190267",
            "devices": [
                "34:190267"
            ]
        },

If I use the temperature sensor in the controller, I assume we would put the controller ID for the sensor?

If I have an HCC80R controlling UFH, how is this specified under devices for each individual zone, particularly given that tthe HCC80R has its own 'sub-zones' (as identified by ufh_idx in your message responses)? Would these sub-zones need to be specified? If so, where does this go?

Hope the above makes sense!
Thanks.

OMG, the answer to this will be worth making into a wiki entry...
1st (useful to know): the output schema can be used as an input schema, so just use:
python client.py monitor /dev/ttyUSBx -x "RQ 01:xxxxxx 1F09 00"

and hit Ctrl-C after the last RP/01:xxxxxx/0418/xx - the schema should be there for you. Have you run the CLI yet - to do so is very useful, so get involved in that.
2nd: I have not had a UFH system to test against, so expect some issues. To answer this question:

If I have an HCC80R controlling UFH, how is this specified

Please start by providing me the schema, so we can discuss it without confusion.
3rd (the biggie): There are three means used by evohome_rf to construct the schema (in order of development):

from eavesdropping (by capturing packets between the controller and other devices)
from the input schema provided as a dict
from discovery (i.e. the controller is asked a series of RQ s, and the corresponding RP are used)

There are advantages to all 3 options - and each has issues, too. One is that when an received packet - usually an eavesdropped packet) does not match the schema: CorruptStateError (another reason why your input schema should be 'right').
The schema is important, it is used to construct some meta-data (e.g. zone heat demand from the child devices of that zone), and the objects (so you can, e.g. set the mode of a zones).
4th...

If I use the temperature sensor in the main controller itself for one zone

I just want to say, just want to avoid confusion: the schema should reflect reality: (evohome_rf doesn't change the behaviour of evohome), so 'making' the controller the sensor for a zone is only appropriate if it is the sensor for that zone.
There is no simple means to learn the sensor for a zone, when that sensor is the controller - I do have some code for this, but it is only 'nearly-deterministic' and is currently disabled. That is, the only 'hard/fast' way to know which zone the controller is a sensor for (if any) is via an input schema (which we hope is 'true' to reality for the above reasons).

AttributeError: 'Serial' object has no attribute 'set_low_latency_mode'
What version of pyserial / pyserial-asyncio are you using?

Thanks for the above - very helpful. I did try the cli, but didn't realise that it built the full schema for me - definitely one for the wiki!
Schema output from discovery below:
Schema[01:139901 (evohome)] = {
    "controller": "01:139901",
    "system": {
        "heating_control": null,
        "orphans": []
    },
    "ufh_controllers": {
        "02:043392": {
            "ufh_circuits": {
                "00": {
                    "zone_idx": "07"
                },
                "01": {
                    "zone_idx": "0A"
                },
                "02": {
                    "zone_idx": "0B"
                }
            }
        }
    },
    "stored_hotwater": {
        "hotwater_sensor": "07:043555",
        "hotwater_valve": "13:133904",
        "heating_valve": "13:102710"
    },
    "zones": {
        "00": {
            "heating_type": "radiator_valve",
            "sensor": "34:015243",
            "devices": [
                "04:228044",
                "04:143928",
                "04:143926",
                "04:143922"
            ]
        },
        "01": {
            "heating_type": "radiator_valve",
            "sensor": "04:143924",
            "devices": [
                "04:227952"
            ]
        },
        "02": {
            "heating_type": "radiator_valve",
            "sensor": "04:228096",
            "devices": [
                "04:228042"
            ]
        },
        "03": {
            "heating_type": "radiator_valve",
            "sensor": "04:227954",
            "devices": [
                "04:227954"
            ]
        },
        "04": {
            "heating_type": "radiator_valve",
            "sensor": "04:000868",
            "devices": [
                "04:000868"
            ]
        },
        "05": {
            "heating_type": "radiator_valve",
            "sensor": "04:228048",
            "devices": [
                "04:228048"
            ]
        },
        "06": {
            "heating_type": "zone_valve",
            "sensor": "34:231315",
            "devices": [
                "13:230917"
            ]
        },
        "07": {
            "heating_type": "underfloor_heating",
            "sensor": null,
            "devices": []
        },
        "08": {
            "heating_type": "radiator_valve",
            "sensor": "04:001048",
            "devices": [
                "04:001048"
            ]
        },
        "09": {
            "heating_type": "radiator_valve",
            "sensor": null,
            "devices": [
                "04:228094"
            ]
        },
        "0A": {
            "heating_type": "underfloor_heating",
            "sensor": "34:112193",
            "devices": []
        },
        "0B": {
            "heating_type": "underfloor_heating",
            "sensor": "34:214769",
            "devices": []
        }
    }
}

Params[01:139901 (evohome)] = {
    "system": {
        "mode": {
            "system_mode": "day_off",
            "until": null
        },
        "language": "en",
        "heating_control": {}
    },
    "stored_hotwater": {
        "dhw_params": null
    },
    "zones": {
        "00": {
            "name": "Living Room",
            "mode": {
                "zone_idx": "00",
                "mode": "follow_schedule",
                "setpoint": 24.0
            },
            "zone_config": {
                "zone_idx": "00",
                "min_temp": 5.0,
                "max_temp": 35.0,
                "local_override": true,
                "openwindow_function": true,
                "multiroom_mode": false
            }
        },
        "01": {
            "name": "Main Bedroom",
            "mode": {
                "zone_idx": "01",
                "mode": "follow_schedule",
                "setpoint": 23.0
            },
            "zone_config": {
                "zone_idx": "01",
                "min_temp": 5.0,
                "max_temp": 35.0,
                "local_override": true,
                "openwindow_function": true,
                "multiroom_mode": true
            }
        },
        "02": {
            "name": "XX's Room",
            "mode": {
                "zone_idx": "02",
                "mode": "follow_schedule",
                "setpoint": 22.0
            },
            "zone_config": {
                "zone_idx": "02",
                "min_temp": 5.0,
                "max_temp": 35.0,
                "local_override": true,
                "openwindow_function": true,
                "multiroom_mode": true
            }
        },
        "03": {
            "name": "YY's Room",
            "mode": {
                "zone_idx": "03",
                "mode": "follow_schedule",
                "setpoint": 5.0
            },
            "zone_config": {
                "zone_idx": "03",
                "min_temp": 5.0,
                "max_temp": 35.0,
                "local_override": true,
                "openwindow_function": true,
                "multiroom_mode": false
            }
        },
        "04": {
            "name": "ZZ's Room",
            "mode": {
                "zone_idx": "04",
                "mode": "follow_schedule",
                "setpoint": 21.0
            },
            "zone_config": {
                "zone_idx": "04",
                "min_temp": 5.0,
                "max_temp": 35.0,
                "local_override": true,
                "openwindow_function": true,
                "multiroom_mode": false
            }
        },
        "05": {
            "name": "Office",
            "mode": {
                "zone_idx": "05",
                "mode": "follow_schedule",
                "setpoint": 21.0
            },
            "zone_config": {
                "zone_idx": "05",
                "min_temp": 5.0,
                "max_temp": 35.0,
                "local_override": true,
                "openwindow_function": true,
                "multiroom_mode": false
            }
        },
        "06": {
            "name": "Garage",
            "mode": {
                "zone_idx": "06",
                "mode": "follow_schedule",
                "setpoint": 20.0
            },
            "zone_config": {
                "zone_idx": "06",
                "min_temp": 5.0,
                "max_temp": 35.0,
                "local_override": true,
                "openwindow_function": true,
                "multiroom_mode": false
            }
        },
        "07": {
            "name": "Dining Room",
            "mode": {
                "zone_idx": "07",
                "mode": "follow_schedule",
                "setpoint": 23.5
            },
            "zone_config": {
                "zone_idx": "07",
                "min_temp": 5.0,
                "max_temp": 35.0,
                "local_override": true,
                "openwindow_function": true,
                "multiroom_mode": false
            }
        },
        "08": {
            "name": "Landing",
            "mode": {
                "zone_idx": "08",
                "mode": "follow_schedule",
                "setpoint": 23.0
            },
            "zone_config": {
                "zone_idx": "08",
                "min_temp": 5.0,
                "max_temp": 35.0,
                "local_override": true,
                "openwindow_function": true,
                "multiroom_mode": false
            }
        },
        "09": {
            "name": "Bathroom",
            "mode": {
                "zone_idx": "09",
                "mode": "follow_schedule",
                "setpoint": 22.0
            },
            "zone_config": {
                "zone_idx": "09",
                "min_temp": 5.0,
                "max_temp": 35.0,
                "local_override": true,
                "openwindow_function": true,
                "multiroom_mode": false
            }
        },
        "0A": {
            "name": "Kitchen UFH",
            "mode": {
                "zone_idx": "0A",
                "mode": "follow_schedule",
                "setpoint": 24.5
            },
            "zone_config": {
                "zone_idx": "0A",
                "min_temp": 5.0,
                "max_temp": 35.0,
                "local_override": true,
                "openwindow_function": true,
                "multiroom_mode": false
            }
        },
        "0B": {
            "name": "Hallway",
            "mode": {
                "zone_idx": "0B",
                "mode": "follow_schedule",
                "setpoint": 23.5
            },
            "zone_config": {
                "zone_idx": "0B",
                "min_temp": 5.0,
                "max_temp": 35.0,
                "local_override": true,
                "openwindow_function": true,
                "multiroom_mode": false
            }
        }
    },
    "mode": {
        "system_mode": "day_off",
        "until": null
    },
    "language": "en"
}

I have an HCC80R installed, and so can give you any data that you may need from this.
Also in answer to your 4th point, yes, the scenario is "actual", in that my Dining Room zone (with ID "07" in the schema) is UFH, whose actuators are controlled by HCC80R, and whose temperature sensing is done by the main controller, which is situated in the dining room.  I noticed that in the auto-discovered schema above, it is not showing any sensor for this zone "07"... I don't know if that is because it is confused with the controller being there, or may be I did not run the client long enough?
Given the importance of getting the schema correct, it might be helpful for end users if there was a pre-defined command for the cli that just went and got the schema and saved it to a file (or stdout) once the schema was complete. WDYT?

AttributeError: 'Serial' object has no attribute 'set_low_latency_mode'

What version of pyserial / pyserial-asyncio are you using?

pyserial is 3.4, and pyserial-asyncio is version 0.5

Just a follow on WRT the schema file. I added the above schema and tried to run the client with the schema as input. It is giving me the following error for the ufh_controller section:
client.py: Starting evohome_rf...
Traceback (most recent call last):
  File "client.py", line 370, in <module>
    cli()
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "client.py", line 199, in monitor
    asyncio.run(main(lib_kwargs, command=MONITOR, **cli_kwargs))
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "client.py", line 320, in main
    gwy = Gateway(lib_kwargs[CONFIG].pop(SERIAL_PORT, None), **lib_kwargs)
  File "/opt/scripts/evohome/evohome_rf/evohome_rf/__init__.py", line 103, in __init__
    self._schema, self.known_devices = load_schema(self, **kwargs)
  File "/opt/scripts/evohome/evohome_rf/evohome_rf/schema.py", line 216, in load_schema
    schema = SYSTEM_SCHEMA(kwargs.get("schema", {})) if kwargs.get("schema") else {}
  File "/usr/local/lib/python3.7/dist-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/usr/local/lib/python3.7/dist-packages/voluptuous/schema_builder.py", line 594, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/usr/local/lib/python3.7/dist-packages/voluptuous/schema_builder.py", line 432, in validate_mapping
    raise er.MultipleInvalid(errors)
voluptuous.error.MultipleInvalid: not a valid value for dictionary value @ data['ufh_controllers']

This is the ufh_controller section in the auto-discovered schema:
"ufh_controllers": {
        "02:043392": {
            "ufh_circuits": {
                "00": {
                    "zone_idx": "07"
                },
                "01": {
                    "zone_idx": "0A"
                },
                "02": {
                    "zone_idx": "0B"
                }
            }
        }
    },

Any suggestions as to what the issue may be?
Thanks.

   "07": {
       "heating_type": "underfloor_heating",
       "sensor": null,
       "devices": []
   },

So simply change the above to have:
        "07": {
            "heating_type": "underfloor_heating",
            "sensor": "01:139901",
            "devices": []
        },

But! Don't be surprised it it doesn't work - I think I made some breaking changes - I will check it now.
I will first test this:
    "schema": {
        "controller": "01:145038",
        "zones": {
            "00": {
                "sensor": "01:145038",
            }
        }
    }

Then, later, a full schema & let you know.
Of course, you can test it too.

I noticed that in the auto-discovered schema above, it is not showing any sensor for this zone "07"... I don't know if that is because it is confused with the controller being there, or may be I did not run the client long enough?

As I said, above: there is no deterministic way to work out what zone the controller is a sensor for (if any) - you have to analyse the meta-data, and even then it's not fully guaranteed to be 'truth'.
The best scenario is that you have N+1 zones (presumably all with sensors), and only N sensors (other than the controller).

Given the importance of getting the schema correct, it might be helpful for end users if there was a pre-defined command for the cli that just went and got the schema and saved it to a file (or stdout) once the schema was complete. WDYT?

Reasonable idea - will add it as a feature, but until then:
python client.py monitor /dev/ttyUSB0 > msg.out

Just a follow on WRT the schema file. I added the above schema

Let me test it (as I said, I haven't done a lot of UFH testing).

Later, I'll test a full schema & let you know.
Of course, you can test it too.

Thanks and yes, I will be testing too. Packet log with above schema attached:
packet.log

Use this for now:
    "schema": {
        "controller": "01:139901",
        "zones": {
            "07": {
                "sensor": "01:139901"
            }
        }

Even starting with just the above, autodiscovery seems to overwrite the zone 7 sensor value to null.

Yes sector makes sense in retrospect - it's an easy fix.
what's happening is that evohome-rf is asking the controller for the sensor
of that zone and it's saying that there is no sensor for that zone.
On Wed, 10 Feb 2021, 15:09 smar<EMAIL_ADDRESS>wrote:

Use this for now:
"schema": {
    "controller": "01:139901",
    "zones": {
        "07": {
            "sensor": "01:139901"
        }
    }

Even starting with just the above, autodiscovery seems to overwrite the
zone 7 sensor value to null.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/zxdavb/evohome_rf/issues/15#issuecomment-776772832,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABTYKCG6STW2EXFSFMWB4LDS6KOUFANCNFSM4XEYN6VA
.

it's saying that there is no sensor for that zone.

Thanks and yes, I had guessed as much.

Would it be possible for you to remove the basicConfig configuration in packet.py:
logging.basicConfig(level=DEFAULT_LEVEL, format=DEFAULT_FMT, datefmt=DEFAULT_DATEFMT)
This seems to take over the logger in other scripts, .e.g if I create a new logger in the client.py with separate handlers for writing to file and showing on console with writing to file set at DEBUG, and to console on ERROR, it always writes to console, regardless of the log level in the console handler. Commenting out the above line in packets.py resolves it for me.
Thanks.

... makes sense in retrospect - it's an easy fix.

Actually, not that straight-forward at all! I am also a little troubled by having to commit to a schema - give me a few days to think about it...

Would it be possible for you to remove the basicConfig configuration in packet.py

I hope so - will have a look - the library should definitely not affect logging at the client level. Any change I can see you code - may be helpful.

Actually, not that straight-forward at all! I am also a little troubled by having to commit to a schema - give me a few days to think about it...

No worries and will leave it with you. For my purposes, I don't think it is as much of an issue - more a nice to have (I think!).

I hope so - will have a look - the library should definitely not affect logging at the client level. Any change I can see you code - may be helpful.

Can you try the code below, with both including the from evohome_rf import Gateway and commenting it out. This will hopefully show what I mean.
import logging
from logging.handlers import RotatingFileHandler
from evohome_rf import Gateway

log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# Log file handler
file_handler = RotatingFileHandler("./test.log", maxBytes=1000000, backupCount=2)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
log.addHandler(file_handler)

# Log console handler
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.ERROR)
console_handler.setFormatter(formatter)
log.addHandler(console_handler)

log.info("Info: This should be in log file only")
log.debug("Debug: This should be in log file only")
log.warning("Warning: This should in log file only")
log.error("Error: This should be both on console and in log file")

I've got a small feature dump lined up too...

Great, look forward to it!
The one feature that would be helpful to me is a callback that lets me know once a send command has completed - whether with success or failure/expired. Again, I can work around it by tracking messages at the client end, and matching send/responses etc, but as it looks as if you are already doing it in PacketProtocolQos, it would make life easier and more elegant!

I think I haven't yet answered these questions...

With regards to sending commands, is there any callback that confirms the status of the sending, e.g. command sent/success/fail notifications? I see various packet messages showing retry/expired etc, but haven't (yet) looked to see how it all works in your code.

This is not fully complete. The RAMSES protocol is based upon casting - I have been trying to - essentially - turn UDP into TCP... In short: there is an architecture; client -> Msgs -> Pkts -> serial port or (read/only) file.  There is QoS - callbacks and re-transmits - built into the Pkt layer..
In addition: there is currently Pkt callbacks utilized for schedules & fault logs at the Client layer - the rest is TBD. That is you ask for a zone's schedule & it sends the required sequence of packets and you receive the schedule.
If you wish you can just hook into them at the Pkt layer. This says it all, I think:
        def rq_callback(msg) -> None:
            ....

        rq_callback = {"func": rq_callback, "timeout": td(seconds=10)}
        self._gwy.send_data(
            Command("RQ", self._ctl.id, "0418", f"{log_idx:06X}", callback=rq_callback)
        )

Question: what would you prefer: callbacks, or an async wait?  I think I could do either, but I was thinking of the latter.
I have added a bunch of constructors to the Command class to start this process: The Command class will shortly have a bunch of constructors, e.g.:
    @classmethod
    def zone_setpoint(cls, ctl_id, zone_idx, setpoint: float, **kwargs):
        payload = f"{zone_idx:02X}" if isinstance(zone_idx, int) else zone_idx
        payload += temp_to_hex(setpoint)
        return cls(" W", ctl_id, "2309", payload, **kwargs)

You would have code like this:
cmd = Command.zone_setpoint("01:123456", "00", 12.0)
try:
    gwy.send_cmd(cmd)
except evoFailed:
    raise

This is just the construction of a command packet - you will also be able to:
await gwy._evo.zone_by_id["00"].set_setpoint(12.0)

... but it will just be calling the above.
Life is - oddly - more complicated for getters, like this:
result = await gwy._evo.zone_by_id["00"].get_setpoint()

EDIT: Also is there any way to 'label' or tag devices? For example, if say there are multiple TRVs in a room, can we identify specific TRVs by name?
Yes. You have friendly-names, & use the config file to make that happen.

The one feature that would be helpful to me is a callback that lets me know once a send command has completed
Are you interested only in getters, or setters too?

Are you interested only in getters, or setters too?

For the moment, getters should be enough.

I have been trying to - essentially - turn UDP into TCP... In short: there is an architecture; client -> Msgs -> Pkts -> serial port or (read/only) file. There is QoS - callbacks and re-transmits - built into the Pkt layer..

Thanks and this is helpful background.

you can just hook into them at the Pkt layer

Great, will look into this.

Question: what would you prefer: callbacks, or an async wait? I think I could do either, but I was thinking of the latter.

Ok with either.

So, you would then have code like this:

This would be perfect, and save me a whole bunch of coding. At the moment, I have a build_send_string() function to do that sort of pre-processing to simplify send commands to the app. Would be great if you could add in durations/until etc (e.g. for setpoints, or mode change) into these.

Yes. You have friendly-names, & use the config file to make that happen.

Yes, I found that this required the devices to be specified in the allowlist. Again, maybe one for the wiki :)
One further request (sorry!) - could you make HGI_DEV_ID user configurable.
Thanks again for all your help, and of course the great library you have developed!

Sorry - callbacks in Msg layer, QoS (e.g. re-transmits) in Pkt layer.

Would be great if you could add in durations/until etc

I'm adding in nothing which isn't already there. But it is is all there anyway.
This is command 2309 (zone_setpoint):
cmd = Command.zone_setpoint(ctl_id, zone_idx, setpoint)
msg = await gwy.send_cmd(cmd)

... and command 2349 (zone_mode):
cmd = Command.zone_mode(ctl_id, zone_idx, mode, setpoint, until)
msg = await gwy.send_cmd(cmd)

I have a build_send_string() function

You won't need it. This is an example of a Command constructor:
    @classmethod  # constructor for 000A  # TODO
    def zone_config(
        cls,
        ctl_id,
        zone_idx,
        min_temp: int = 5,
        max_temp: int = 35,
        local_override: bool = False,
        openwindow_function: bool = False,
        multiroom_mode: bool = False,
        **kwargs,
    ):
        """Constructor to set the config of a zone (c.f. parser_000a)."""

        payload = f"{zone_idx:02X}" if isinstance(zone_idx, int) else zone_idx

        assert 5 <= min_temp <= 30, min_temp
        assert 0 <= max_temp <= 35, max_temp
        assert isinstance(local_override, bool), local_override
        assert isinstance(openwindow_function, bool), openwindow_function
        assert isinstance(multiroom_mode, bool), multiroom_mode

        bitmap = 0 if local_override else 1
        bitmap |= 0 if openwindow_function else 2
        bitmap |= 0 if multiroom_mode else 16

        payload += f"{bitmap}"
        payload += temp_to_hex(min_temp)
        payload += temp_to_hex(max_temp)

        return cls(" W", ctl_id, "000A", payload, **kwargs)

One further request (sorry!) - could you make HGI_DEV_ID user configurable.

Before I say no (it is a constant), what is your use-case?

Thanks for the above on the command constructor etc - all very helpful, and clearly still a lot more for me to discover in your code!

One further request (sorry!) - could you make HGI_DEV_ID user configurable.

Before I say no (it is a constant), what is your use-case?

I understood from a comment in the evofw3 repo that the ID is hardcoded in the firmware to be 18:056026 (see https://github.com/ghoti57/evofw3/issues/2#issuecomment-681130505 and the subsequent comment). The ID that you are using in your const file  18:000730, hence the request.

My understanding is that the latest evofw3 FW has been changed, so that each is very likely to have a unique address.
You send a packet using the source address 18:000730, and the HGI80/evofw3 then substitutes its own address in that field. That is:
RQ --- 18:000730 13:237335 --:------ 3EF0 001 00

becomes:
RQ --- 18:xxxxxx 13:237335 --:------ 3EF0 001 00

My understanding is that the latest evofw3 FW has been changed, so that each is very likely to have a unique address.
Are you happy that it doesn't need changing?

Oh ok. I wasn't aware of that. In that case, no need to change anything! BTW, do you have a link to that change in the ID, just out of interest?

Sorry for the tardy reply - I didn't see your question.
The nanoCULs ID should never change, even across restarts.
The issue is that RF is unreliable, bit-flips are common (I see about 2 a day, where the resultant packet is still valid, but the address has changed) and it is difficult for me to know how to treat packets with a new address (this issue is not limited to 18:xxxxxx addresses).
Oddly, I have just had a conversation with the evofw3 dev about this - we are hoping to mitigate this issue with evofw3.
I have means to do something about it in evohome_rf

schemas & whitelisting of device_ids
validation of device_type vs packet/payload
and (coming soon) stateful inspection of packets

You only have an nanoCUL? If you have an HGI80, is it doing it with that too?

Thanks for the update and makes sense (in fact, I had originally assumed it was just bit flips as well). Unfortunately I don't have an HGI80 but have two nanoCULs, and so have been testing on both.

I have means to do something about it in evohome_rf

schemas & whitelisting of device_ids

Is this the allowlist?

Is this the allowlist?
Yes.

Just pushed to master branch - you  may want to use that for a while - or bleeding_edge - up to you.

You're using a different mechanism to specify an allow list, but these notes will still be useful to read: https://github.com/zxdavb/evohome_cc/wiki/Tips-for-the-configuration.yaml-file#allow-list

Just pushed to master branch - you may want to use that for a while - or bleeding_edge - up to you.

Thanks for the headsup.

You're using a different mechanism to specify an allow list, but these notes will still be useful to read: https://github.com/zxdavb/evohome_cc/wiki/Tips-for-the-configuration.yaml-file#allow-list

Thanks, that's helpful.
On a seperate matter, I was trying out the async_send_cmd  and got the following timeout:
Traceback (most recent call last):
  File "evorf.py", line 550, in mqtt_process_msg
    resp = asyncio.run(GWY.async_send_cmd(gw_cmd))
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.6-py3.7.egg/evohome_rf/__init__.py", line 308, in async_send_cmd
    return await self.msg_protocol.send_data(cmd, awaitable=awaitable, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.6-py3.7.egg/evohome_rf/protocol.py", line 426, in send_data
    result = await awaitable(timeout=4)  # may: raise TimeoutError
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.6-py3.7.egg/evohome_rf/protocol.py", line 51, in getter
    raise TimeoutError
TimeoutError

As this is my first foray into asyncio, is the timeout due to the way I'm calling the coroutine?

YMMV with 3.7 - should be OK, but I'm developing on 3.8
The TimeoutError simply says that it didn't get the response is was hoping for - it can't wait forever, so only waited 4 seconds & then gave up:
result = await awaitable(timeout=4)

The awaitable() is this code (i.e. is getter(), earlier up in protocol.py:
        async def getter(timeout=None) -> Tuple:
            dt_expired = dt.now() + td(seconds=3 if timeout is None else timeout)
            while dt.now() < dt_expired:
                try:
                    self._result = self._queue.get_nowait()
                except Empty:
                    await asyncio.sleep(0.005)
                else:
                    return self._result
            raise TimeoutError

What parameters did you pass to:
self.msg_protocol.send_data(cmd, awaitable=awaitable, **kwargs)

Specifically, what was the cmd, and did you use any kwargs?

What parameters did you pass to:
self.msg_protocol.send_data(cmd, awaitable=awaitable, **kwargs)

Specifically, what was the cmd, and did you use any kwargs?

cmd is RQ --- 18:000730 01:139901 --:------ 313F 001 00, and I did not pass any kwargs. The same cmd works perfectly on the synchronous send_cmd.
Noted on the python version you are developing on. I will switch to the same version shortly.

How you getting on - I'm making various changes - I hope not breaking too much for you.

Getting on quite well actually - have a working system where I've replaced all my legacy encoding/decoding with your framework. I'll probably upload it to github in the next day or so for anyone that wants to play with it.
Really like what you've done, and look forward to seeing your changes each time you commit!

BTW, did you get a chance to look at that logging.basicConfig in packet.py?

looking now

OK, I'll move it to my client.py - the obvious place.

Really like what you've done, and look forward to seeing your changes each time you commit!

If you have any more suggestions, just ask!

If you have any more suggestions, just ask!

Thanks and will do!

@smar000 I have made some change to schema, etc - likely will break your code - sorry.
Of course, you have a PiPy package you're using - so stick with that, but there'll be many bugs / new features coming too - it just may be worth hanging on before you jump ship to a later version of eh package...

@smar000 I have made some change to schema, etc - likely will break your code - sorry.
Of course, you have a PiPy package you're using - so stick with that, but there'll be many bugs / new features coming too - it just may be worth hanging on before you jump ship to a later version of eh package...

Thanks for the heads up. I will leave off installing newer builds for the moment then.

Hi @zxdavb
Hope all is well at your end.
I've been working on updating my system to use your latest build and in particular getting my send commands integrated in. You have clearly been very busy with the whole framework :)
One question/request - is there a callback for when sending commands fail? I am seeing log messages such as the following, but was looking for same way to trap so that they can be published to the mqtt broker:
PktProtocolQos.send_data(RQ|01:139901|0418|00): boff=1, want=RQ|01:139901|0418|00, tout=2021-05-19 16:29:07.773286: RE-SENT (1/3)
PktProtocolQos.send_data(RQ|01:139901|0418|00): boff=2, want=RQ|01:139901|0418|00, tout=2021-05-19 16:29:07.980920: RE-SENT (2/3)
PktProtocolQos.send_data(RQ|01:139901|0418|00): boff=3, want=RQ|01:139901|0418|00, tout=2021-05-19 16:29:08.391702: RE-SENT (3/3)
PktProtocolQos.send_data(RQ|01:139901|0418|00): boff=3, want=RQ|01:139901|0418|00, tout=2021-05-19 16:29:08.391702: EXPIRED

Thanks!

Welcome back!
In init.py, there is a send_cmd() that can wrap any Command into a callback, and an async_send_cmd() that can wrap any Command as an awaitable (by using a callback internally).
Let me know if they don't work.

Thanks for the quick reply!
I am in fact using that very same send_cmd() with a callback. The callback appears to be only called when the send is successful (and this is working consistently), but doesn't appear to be called if the send is Expired as in the above log. Is this expected behaviour?

Is this expected behaviour?

No, if the callback expires, it should return False.

        for (
            hdr,
            callback,
        ) in self._callbacks.items():  # 1st, notify all expired callbacks
            if callback.get(EXPIRES, dt.max) < pkt._dtm:
                _LOGGER.error("MsgTransport._pkt_receiver(%s): Expired callback", hdr)
                callback[FUNC](False, *callback.get(ARGS, tuple()))

Is this expected behaviour?

No, if the callback expires, it should return False.

I haven't been seeing this. The callback doesn't seem to have been called on any of my expired commands that I could see.

Let me know what Command you're calling? & I'll take a look.

It isn't any specific Command, though more often with the more "complex" commands such as set_dhw_mode with an until. On occasion, some commands fail and have to be resent - I think it may be due to the distance between my arduino and the Controller (they are in different rooms). This doesn't always happen, but happens often enough that it would be helpful to know. In my earlier log paste, the expired command was actually a simple get_system_log_entry.

OK, will have a look.

Try 0.9.7 - only the above issue remains.

Working! Thanks again for the quick turn-around.

I have pushed an update to master that will make callbacks for failed Command much faster. Something like:
2021-05-20T21:36:15.727873  # evofw3 0.7.0
21:36:15.757 || HGI:006402 | NUL:------ |  I | puzzle_packet    | 01763... || {'ramses_rf': 'v0.9.7'}
21:36:18.766 || HGI:006402 | CTL:145039 | RQ | zone_name        | 0000     || {'zone_idx': '00'}
21:36:18.828 PktProtocolQos.send_data(RQ|01:145039|0004|00): boff=1, want=RQ|01:145039|0004|00, tout=2021-05-20 21:36:18.928404: RE-SENT (1/2)
21:36:18.850 || HGI:006402 | CTL:145039 | RQ | zone_name        | 0000     || {'zone_idx': '00'}
21:36:18.961 PktProtocolQos.send_data(RQ|01:145039|0004|00): boff=2, want=RQ|01:145039|0004|00, tout=2021-05-20 21:36:19.161328: RE-SENT (2/2)
21:36:18.982 || HGI:006402 | CTL:145039 | RQ | zone_name        | 0000     || {'zone_idx': '00'}
21:36:19.195 PktProtocolQos.send_data(RQ|01:145039|0004|00): boff=2, want=RP|01:145039|0004|00, tout=2021-05-20 21:36:19.183045: EXPIRED
21:36:19.195 PktProtocolQos.send_data(RQ|01:145039|0004|00): Expired callback
Callback has expired
21:36:42.583 || STA:205645 |            |  I | temperature      | 00077D   || {'temperature': 19.17}

Hi again @zxdavb
One thing to clarify - is it still the recommendation that the schema should only contain the controller ID and allow_list, and leave everything else for auto-discovery?

Yes, Controller ID only, and an allow_list - there are a few exceptions:

OTB can't be discovered, only specified in schema or eavesdropped
some TRVs, 00:, can't be discovered (so, same as above)

Eavesdropping is not generally recommended, but is needed to confirm a schema.

Yes, Controller ID only, and an allow_list - there are a few exceptions:

OTB can't be discovered, only specified in schema or eavesdropped
some TRVs, 00:, can't be discovered (so, same as above)

Eavesdropping is not generally recommended, but is needed to confirm a schema.

Ok thanks and noted.

Yes - may need allow_list off & eavesdropping on to get full schema - then make a schema, use an allow_list & turn off eavesdropping for ongoing, until any changes are made at some future time.
TBH, you can manage without a full schema in most circumstances, but an allow_list is virtually required.
I use the allow list in ramses_rf, but also in evohome_cc too - that it, you may want to consider using it in evohome_listener.

Thanks and that makes sense. I will look to do it the same way.

TBH, you can manage without a full schema in most circumstances, but an allow_list is virtually required.

For my own understanding, the allow_list is required to prevent messages appearing from (a) neighbouring evohome setups and (b) messages getting garbled in reception thus appearing to be from incorrect device ids?

The issue is that valid packets can - as you say - have corrupted (but still valid) devices addresses.

Understood! Thanks.

@zxdavb Looking at client.py, is it sufficient to pass the --discover argument in order to enable eavesdropping for the full schema?
Also when should we use the SCAN_DISC, SCAN_FULL, SCAN_HARD and SCAN_XXXX?
Thanks.

(not sure if I understand your question 100%, so here goes)

Also when should we use the SCAN_DISC, SCAN_FULL, SCAN_HARD and SCAN_XXXX?

These are used for probing new device types or for development - not for production - discovery is a separate thing, and is built into the library.
The client.py tool's main reason for being is to: a) perform ad-hoc stuff / learn about the protocol (e.g. SCAN_FULL), or to demonstrate how the library can be used (e.g. evohome-listener, or evohome_cc). I see you've pulled some ideas from client.py (which the idea), but you may also want to look at evohome_cc\__init__.py (although it is a little more obtuse).
All the CLI switches are a 'cheat' - all that matters is passing the config/schema/device_list kwargs to the library - you can get those parameters from any source you like - from CLI/yaml/json/etc. - or a config file in your case.
Generally, eavesdropping is to be avoided, especially without an allow_list (although it may be the only way to get your allow_list, but don't eavesdrop after that.)
I use voluptuous roughly the same way you use configparser, and so see my schema.py:
CONFIG_SCHEMA = vol.Schema(
    {
        vol.Optional(DISABLE_DISCOVERY, default=False): bool,
        vol.Optional(DISABLE_SENDING, default=False): bool,
        vol.Optional(ENABLE_EAVESDROP, default=False): bool,
        vol.Optional(REDUCE_PROCESSING, default=0): vol.All(
            int, vol.Range(min=0, max=DONT_CREATE_MESSAGES)
        ),
        vol.Optional(MAX_ZONES, default=DEFAULT_MAX_ZONES): vol.All(
            int, vol.Range(min=1, max=16)
        ),
        vol.Optional(USE_SCHEMA, default=True): vol.Any(None, bool),
        vol.Optional(ENFORCE_ALLOWLIST, default=None): vol.Any(None, bool),
        vol.Optional(ENFORCE_BLOCKLIST, default=None): vol.Any(None, bool),
        vol.Optional(USE_NAMES, default=None): vol.Any(None, bool),
        vol.Optional(EVOFW_FLAG, default=None): vol.Any(None, str),
    },
    extra=vol.ALLOW_EXTRA,  # TODO: remove for production
)

There is one thing that ramses_rf needs to do, and does not yet do well - that is periodically re-discover...
This would be most useful or OTBs (R8810As) - I will ad this soon enough.
HTH, ask again if not.

Great, that clears up quite a bit. Will also have a look at your evohome_cc_init.py_.
Thanks again.

Hi David
I noticed that zone names are available in .evo.params but not in the schema's zone collection. Should /can these param zone names also be loaded into the framework in some way at start up, to save having to wait for the system to populate itself from eavesdropping (which can take some time)?

I am happy with the split between schema (effectively static, unless bindings are destroyed/created), params (configurable), and status (variable, but read-only) - the only possible
I am happy to push in (impose) a (hand-crafted) schema, only because it is so static - doing so with params would be problematic, and inappropriate with status...
... however, the latter two can be managed with _get_state(), _clear_state() and _set_state() - have a look at those - although they are currently private methods, they will eventually become canon.
There are two bugs presently affecting _set_state():

some packets are being expired too early
packets are not being refreshed when they expire

HTH

BTW, you make a good point - eavesdropping of schema if problematic & disabled by default, but eavesdropping of params/state is normal practice.

however, the latter two can be managed with _get_state(), _clear_state() and _set_state() - have a look at those - although they are currently private methods, they will eventually become canon.

Thanks, will check these out. Noted on the bugs.

but eavesdropping of params/state is normal practice.

Ok!

Hi David
Thanks for all your help (and patience!) throughout the migration. I am almost there I think. One final (hopefully!) question for now WRT the allow_list. Is this supposed to only permit  exactly those devices listed or will it automatically allow any "18:" HGI device types through?
I ask as I have a number of spurious messages purporting to be from HGI devices (e.g. I am seeing messages from the following HGI IDs now as I type: 18:000030, 18:000070, 18:007030, 18:056026, 18:136712; these are all  in addition to my actual ID of 18:000730).
My allow_list (as shown at initialisation) appears to be correctly configured and active and ENFORCE_ALLOWLIST  also set to true:
Using an allow_list: ['01:139901', '02:043392', '04:000868', '04:001048', '04:143922', '04:143924', '04:143926', '04:143928', '04:227936', '04:227952', '04:227954', '04:228032', '04:228042', '04:228044', '04:228048', '04:228094', '04:228096', '04:228104', '07:033469', '07:043555', '10:142568', '13:230917', '13:102710', '13:133904', '13:127198', '34:015243', '34:112193', '34:214769', '34:231315', '18:000730']
Anyway, thanks again!

Sorry, yet another query/clarification. I noticed that zone/dhw mode "until" datetimes are formatted (in the helper.) without the T separating date and time.
Should the T be included as part of the isoformat (e.g. https://en.wikipedia.org/wiki/ISO_8601)?

Hi again @zxdavb
One further query - I noticed that you have the default HGI device address in the consts.py file as 18:000730. However, the address of my nanoCUL appears to be 18:136712.  I don't see this address in the eavesdropped schema. Is there any way to identify the nanoCUL by the framework?
e.g. some messages from my log file:
2021-06-02 07:19:17,441 [299] || HGI:136712         | OpenTherm Bridge   | RQ | opentherm_msg    | 00000... || {'msg_id': '0x03', 'msg_name': 'SlaveConfigFlags', 'msg_type': 'Read-Data', 'description': 'Slave configuration'}
2021-06-02 07:19:17,760 [299] || HGI:136712         | Controller         | RQ | zone_devices     | 0208     || {'zone_idx': '02', 'device_class': 'rad_actuators'}
2021-06-02 07:19:17,776 [299] || Controller         | HGI:136712         | RP | zone_devices     | 02080... || {'zone_idx': '02', 'device_class': 'rad_actuators', 'devices': ['04:228042']}
2021-06-02 07:19:17,856 [299] || HGI:136712         | DHW Relay          | RQ | tpi_params       | 00       || {}
2021-06-02 07:19:17,873 [299] || DHW Relay          | HGI:136712         | RP | tpi_params       | 00300... || {'cycle_rate': 12, 'min_on_time': 1.0, 'min_off_time': 0.0, '_unknown_0': '00', 'proportional_band_width': None, '_unknown_1': '01'}

Currently. 18: devices are never filtered - this causes issues, and needs to change.
Because you don't know the HGI80s address before you send a packet, you use 18:000730, and it 'translates' it to it's actual device address, 18:136712 in your case.

Is there any way to identify the nanoCUL by the framework?

This is planned.

Sorry, yet another query/clarification. I noticed that zone/dhw mode "until" datetimes are formatted (in the helper.) without the T separating date and time.

I am willing to take a view on this (the issue is whether it will be a breaking change), but is is kind of 'optional', see: https://docs.python.org/3/library/datetime.html#datetime.datetime.fromisoformat

Currently. 18: devices are never filtered - this causes issues, and needs to change.
Because you don't know the HGI80s address before you send a packet, you use 18:000730, and it 'translates' it to it's actual device address, 18:136712 in your case.

Understood!

I am willing to take a view on this (the issue is whether it will be a breaking change), but is is kind of 'optional'

Ah ok, I wasn't aware that it was optional. It's not that much of a problem, but openHAB's mqtt binding requires the "T" out of the box. Without it, additional formatting steps need to be carried out which has to be repeated for each datetime item etc. For now, I've added a small conversion before publishing to the mqtt broker. Maybe make it optional via config If you do re-consider it in the future?

Maybe make it optional via config If you do re-consider it in the future?

Making it a config option is not useful - I'll change the format if I can.

Making it a config option is not useful - I'll change the format if I can.

Thanks. As I said, not a significant issue from my side.
Anyway, my migration to your framework seems to be complete for now, and so I will close this thread. Sincere thanks once again for all your support throughout.


GitHub's ci-status check is limited to the first page of results

The hub ci-status does not do pagination, so when there are more than 30 statuses/checks, some of them are excluded.

Either await a resolution of https://github.com/github/hub/issues/3024 or use another approach than hub.


Question about react-json-to-table component

Hi Javier,
I just start to learn React. And I found react-json-to-table component written by you. I have a question for you. In order to simplify my question, I created a separate project from my original project.
See the attached JsonToTableExample.js. When I hard code the json string in setReturnedJson function, the table displays. But when I parse the variable called cleanedJson string which is the same string as hard coded string into setReturnedJson(cleanedJson) (I commented out in the code ), the table is not displayed at all and there is no error. This may not be an issue on react-json-to-table component. I may have something wrong. I don't know why. Would you please help me out? I appreciate your help very much.
JsonToTableExample.zip

I figured out that the variable cleanedJson in my code is a string but not a json object. I have to convert it to json object by using  JSON.parse(cleanedJson), and then setReturnedJson(JSON.parse(cleanedJson)) works with ts-react-json-table component and the table displays well.


Fixed logic flaw in the secrets mounts

Tested on a REHL box and found out that the mounts were not showing up
Had a logic flaw, where if the mount was "host:container"
Was setting the mount source to "host" and destination to "ctrRunDir/container"
When instead, the mount source should be "ctrRunDir/container" and destination "container"
with the data copied from "host" to "ctrRunDir/container"
Signed-off-by: umohnani8<EMAIL_ADDRESS>

@rhatdan PTAL

@umohnani8 Travis CI isn't happy.

LGTM
Don't know why travis is complaining.

LGTM, if possible, would be nice for @nalind to take a peak.

Yeah, Travis looks like some kind of build system bogosity.  It can't resolve https://golang.org/x/sys/unix?go-get=1 for some reason.

LGTM great catch

/test all

@rhatdan @mrunalp Tests pass.

LGTM


Unit test and integration test in .netcore

How do I create unit test for react application in .netcore? And how to run it in VS?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a3154dec-545a-c749-76ac-1f4581c2f922
Version Independent ID: cd33922a-0feb-2a37-4e8e-6d7856617a31
Content: Use the React project template with ASP.NET Core
Content Source: aspnetcore/spa/react.md
Product: aspnet-core
GitHub Login: @SteveSandersonMS
Microsoft Alias: scaddie

@nnofa Are you trying to test the JavaScript or the C#?

sorry I was not clear, it's for javascript testing

Thanks for the question. You'll probably get a faster answer on a support forum, such as Stack Overflow, or a support chat, such as Slack or Gitter.


What do we do with our partial FreeBSD support?

We have lost our FreeBSD maintainer. Partial FreeBSD support exists in the codebase but it is not mentioned in the README as a supported platform.
Should we just leave it at that and see if another FreeBSD enthusiast shows up? From a quick glimpse it seemed like adjusting the pthreads support might be quite annoying, since the types are smaller on FreeBSD so we'd have to adjust the offsets where we store our stuff. I have no clue about any other FreeBSD-specific APIs libstd might use. So I won't work on completing this support. But for now there is also little cost in just keeping it around; we can always remove it if/when it becomes a problem.

FWIW Android is pretty much in the same situation -- we have just enough support to run 'hello world', and that's about it.
This is a good foundation for someone to come in and add the missing shims for proper support.

Yea, I'd rather keep the support as long as it's not a maintenance burden (which afaict it is not).
Having a minimal thing to start from seems great. Maybe all we need is a table of supported targets and their tier/level of support


Not sensitive enough on 6s

I have an issue where the microphone on my 6s will only give me an output if I am right in front of the microphone. I'd like to change the sensitivity so that it will read a frequency even from a couple of feet distance.
This is how I initialize the AKMicrophone:
AKSettings.audioInputEnabled = true
        mic = AKMicrophone()
        tracker = AKFrequencyTracker.init(mic)
        silence = AKBooster(tracker, gain: 0)

I'm guessing it has something to do with the booster, but changing the gain value did not yield any result.

This is not really an issue, we prefer support to go to StackOverflow.  If you post the question there, I have a response ready.  Thanks!

https://stackoverflow.com/questions/46840362/audiokit-issue-mic-not-sensitive-enough


Added Game class

Redacted the changes that weren't necessary. Incorporated the relevant changes under PR 120.


Advanced Ip filtering

Description:
Implementation of automatic IP ban in case of multiple failed login attempts. Partially related to this request.
File ip_bans.yaml automatically created in config folder.
Pull request in home-assistant.github.io with documentation (if applicable): home-assistant/home-assistant.github.io#1450
Example entry for configuration.yaml (if applicable):
Two new params in http component:
http:
  ip_ban_enabled: True
  login_attempts_threshold: 2

Checklist:
If user exposed functionality or configuration variables are added/changed:

[x] Documentation added/updated in home-assistant.github.io

If the code does not interact with devices:

[x] Local tests with tox run successfully. Your PR cannot be merged unless tests pass
[x] Tests have been added to verify that the new code works.

:+1: from me. @balloob ?

Nice feature.
Is the ip_ban_enabled  parameter necessary?
login_attempts_threshold < 1  -> disable ip ban ?

@Danielhiversen, ip_ban enabled responsible for rejecting all request from ip_bans.yaml file.
If login_attempts_threshold >= 1 it will automatically ban any new failed IPs.

I just setup external access to my home assistant and exempted my home network.. I was thinking of setting up fail2ban but a native solution sounds great.
Two question:

Is this change compatible with the X-forwarded for option?
I think there should be a parameter / default expiry of banned IPs.. From experience a lot of late nite scans happen from many random cloud hosted IPs. I am afraid that without an automatic clean up the IP ban file could get very very large.

@bagobones,

Yes
Didn't think about it, basically haven't notice such problems yet. Can implement something after this PR will be merged as another PR.

@vkorn
Sounds great... Home assistant uses a web form for authentication but it probably wont be too long before a bot starts filling it in and failing.. working with fail2ban and other IP ban scripts the cleanup eventually becomes important.
Another reason as well is to not permanently lock your self out, some systems have a soft ban and a hard ban... IE such as X consecutive attempts results in a non-permanent 5-10min ban, however if there are additional attempts it moves into a permanent / long term ban.
You may also want an IP ban whitelist for users that have the API password enabled at all times, as they may locally lock them self out otherwise.


webvr: touchstart is not sufficient as a user gesture

Hi @mosra !
I finally solved the issue of the webvr example on mobile devices.
Also make sure to enable webvr support in your browser (docs are updated). Even on mobile devices this is required, except if a API token from google has been aquired for your origin (e.g. magnum.graphics, see https://docs.google.com/forms/d/e/1FAIpQLSfO0_ptFl8r8G0UFhT0xhV17eabG-erUWBDiKSRDTqEZ_9ULQ/viewform)
Cheers, Jonathan.

Thanks!
Will merge it as soon as I can verify that it works on my phone (and will bother you again in case it doesn't, hehe).
The form says:

we will create a token for your site that will unlock these new features for the next 6 weeks.

I'm not sure if I want to renew this every 6 weeks, heh. Why is it controlled by a central authority like this?

Yaaas! I see a red cube! And a blue cube!
Thanks! :heart_eyes: :beer:  Merged in bedfe74c34d17f9afe2ab90076f602b8a26ef1f6.


python310Packages.whitenoise: 6.4.0 -> 6.5.0

https://github.com/evansd/whitenoise/blob/6.5.0/docs/changelog.rst
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 251173 run on x86_64-linux 1

11 packages built:

healthchecks
paperless-ngx
python310Packages.flask-common
python310Packages.flask-common.dist
python310Packages.whitenoise
python310Packages.whitenoise.dist
python311Packages.flask-common
python311Packages.flask-common.dist
python311Packages.whitenoise
python311Packages.whitenoise.dist
tandoor-recipes


Add travis-ci

It would be nice to have an automated builder hooked up to this repo.

Good point, hm... I just want to get this next release out and will look into this when its out. Thanks for the input!

Just wanted to let you know that this will be part of release 0.13

Cool. Looking forward to it. Thanks for all the hard work!
On Tuesday, April 14, 2015, Christian Alfoni<EMAIL_ADDRESS>wrote:

Just wanted to let you know that this will be part of release 0.13
—
Reply to this email directly or view it on GitHub
https://github.com/christianalfoni/formsy-react/issues/54#issuecomment-92976931
.

This is now added! :-)

Seems like there is an issue with this feature, first noted in #195.
Also noticed in #197.
Possibly related to this update in the travis-ci project.

I am having a full review of testing before next release. With Node 0.4 release I have to change the way testing is done. It will be better :-)


DS-2654: Add necessary prefix to commented out config

Tiny, obvious PR.
solrauthority.cfg has a commented out config which doesn't have the necessary prefix (per #1104).
Here's where it is used, and referenced as solrauthority.auto-update-items:
https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/authority/UpdateAuthorities.java#L147

Merging this immediately as a very obvious change. Plus, it only changes a commented out setting.


add some apparent type hintings

Description of changes
Summarize the changes made by this PR.

Improvements & Bug fixes

Added some missing type hints to arguments and responses that their type is apparent for mainly local and fastapi.

Test plan

No need for test I believe since there is no logic change, and just added type hints.

Documentation Changes

No

Thank you!

Thanks for the changes, just one more comment (Sorry! Just noticed) and this looks great. Thank you so much!


Fixed typo in pull request template

Pull Request Checklist

[X] Make sure to read the contributor guide here.
[X] Make sure you are making a pull request against the staging branch.
[X] Make sure your pull request title matches the requested format.
[X] Make sure to lint, type-check, and run unit and API tests.

Description
Meant it to be "main" branch, not "staging" branch.

:tada: This PR is included in version 1.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Asyncio component txaio

Follow-up for https://github.com/crossbario/autobahn-python/pull/872 which re-writes the asyncio stuff using pure txaio (avoiding e.g. yield from).
Also fixes some statistics-counting and proper disconnect-on-error in the Twisted component as well.

Codecov Report

Merging #873 into master will decrease coverage by 0.92%.
The diff coverage is 1.85%.

@@            Coverage Diff             @@
##           master     #873      +/-   ##
==========================================
- Coverage   61.81%   60.88%   -0.93%
==========================================
  Files          74       74
  Lines       12870    12880      +10
  Branches     2043     2045       +2
==========================================
- Hits         7955     7842     -113
- Misses       4257     4356      +99
- Partials      658      682      +24

Impacted Files
Coverage Δ

autobahn/twisted/component.py
35.18% <ø> (-0.44%)
:arrow_down:

autobahn/wamp/component.py
45.36% <0%> (-0.32%)
:arrow_down:

autobahn/asyncio/component.py
0.45% <1.92%> (+0.45%)
:arrow_up:

autobahn/twisted/test/test_endpoint_plugins.py
43.75% <0%> (-56.25%)
:arrow_down:

autobahn/websocket/utf8validator.py
41.33% <0%> (-32.01%)
:arrow_down:

autobahn/wamp/test/test_exception.py
84.61% <0%> (-15.39%)
:arrow_down:

autobahn/websocket/xormasker.py
78.94% <0%> (-13.16%)
:arrow_down:

autobahn/wamp/auth.py
50.58% <0%> (-9.31%)
:arrow_down:

autobahn/wamp/test/test_serializer.py
84.5% <0%> (-8.46%)
:arrow_down:

autobahn/asyncio/test/test_asyncio_websocket.py
77.35% <0%> (-7.55%)
:arrow_down:

... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0af895b...aebdc2c. Read the comment docs.


Add table for WinRateCallback

What does this PR do?

Closes #2099 to produce a nice table of prompts / completions / winner index like this:

Wandb reference run: https://wandb.ai/huggingface/huggingface/runs/n1eodgm9/workspace?nw=nwuserlewtun
Gist of code: https://gist.github.com/lewtun/e5e85b8ca4869c3e29f54f20eb0d62c6
Run on a single node of 8 x H100s with:
accelerate launch --config_file=examples/accelerate_configs/multi_gpu.yaml examples/scripts/dpo.py     --dataset_name trl-lib/ultrafeedback_binarized     --model_name_or_path Qwen/Qwen2-0.5B-Instruct     --learning_rate 5.0e-7     --num_train_epochs 1     --per_device_train_batch_size 4 --per_device_eval_batch_size 4     --gradient_accumulation_steps 4     --gradient_checkpointing     --logging_steps 25     --eval_strategy steps     --eval_steps 25     --output_dir Qwen2-0.5B-DPO     --no_remove_unused_columns --dataset_num_proc 48 --bf16 --max_length 2048 --logging_first_step

Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,
Pull Request section?
[ ] Was this discussed/approved via a GitHub issue? Please add a link
to it if that's the case.
[ ] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines.
[ ] Did you write any new necessary tests?

Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@edbeeching @qgallouedec I've iterated to remove the strict wandb requirement and also tweaked the callback to log on_train_begin to compute a reference value of the win rate / view initial completions. Let me know what you think!

LGTM, just apply the minor doc change and we're good to merge!


[Feature request] Add Bcc field

While trying if I got nmail working with Exchange servers on Solus after following your guide in #22, I wanted to add one of my secondary email accounts to the recipients of my work emails and check if my important emails were indeed successfully sent and received, but I don't want people to know about it so Bcc would be ideal. This made me realize nmail doesn't offer a Bcc field yet. There are of course other situations where people would want Bcc.

Right, this is a basic email capability. I believe my backend code already caters for it. I just never put it in the UI, so it should be relatively easy to fix.

Hi @Kabouik - thanks for reporting this issue! I've implemented something now. The UI is similar to pine/alpine - by default the bcc field is hidden, but by pressing Ctrl-R the bcc field is shown (nmail will remember this for future sessions). Let me know if you encounter any issues. Thanks!


Why ChunkExtractor is required?

Hi there!
createLoadable function includes a condition that checks props.__chunkExtractor
https://github.com/gregberge/loadable-components/blob/main/packages/component/src/createLoadable.js#L149
What if i don't want to use ChunkExtractor at all? To be more correctly, i can't add ChunkExtractor, because i use ReactJs.NET on my server.

Without chunk extractor active Loadable will not be able to properly hydrate App as it will not know which chunks are required to be loaded first.
I am not sure how exactly ReactJs.NET works and why you cannot use ChunkExtractor and it's more about webpack rather than platform, but if you still can use Babel try a sibling solution - https://github.com/theKashey/react-imported-component, which does not use anything except Babel

As far as I understand, ChunkExtractor is executed during request processing in node.js. It receives loadableStats.json as chunks information. And then a ChunkExtractor instance is passed to the client components using React Context.
So. In this case, the source of truth is loadableStats.json and i can make required scripts by myself based on loadableStats.json. It seems to me that ChunkExtractor is only a helper to create script tags.
Am i wrong?

Correct, it's only a bearer of mapping between import and chunk. This information is generated by a webpack plugin, but can be generated in any other way.


Add scrape interval to all metric sources

Fixes #84

IIRC the scrape config as it exists out in the wild right now is 30s?

Actually, this effectively is a no-op. Grafana agent's default scrape interval is 60s (https://grafana.com/docs/agent/latest/flow/reference/components/prometheus.scrape/#arguments), not 30s. We're just pulling that variable up a layer to make it easier to override.

Ah ok, maybe I was thinking about something else that caused our scrapes to increase. Sounds good.

@petewall would it be possible to run the create-release ci? - Looks like although the chart ver was updated with this PR a release wasn't cut

Thanks for the poke. Yep, it's done now.


[BUG]: Input/Output error on linux

Operating System
Linux
App Version
3.0.4 Release
Detailed Description
i just tried to run it and it gave me this:
jason@turtle:~/Downloads$ ./Simple-RPC-Editor-3.0.4.AppImage
Bus error (core dumped)
jason@turtle:~/Downloads$ [0128/173602.661344:ERROR:icu_util.cc(249)] Couldn't mmap icu data file
[0128/173602.667671:ERROR:icu_util.cc(249)] Couldn't mmap icu data file
^C
jason@turtle:~/Downloads$ ./Simple-RPC-Editor-3.0.4.AppImage
-bash: ./Simple-RPC-Editor-3.0.4.AppImage: Input/output error

Reproducing the issue
run the appimage
Any additional info
No response

Could it be that the download was corrupted?
Just tested on my Arch Desktop and it launched fine


Update MD file on new branch 2017-12-07_11-43-10

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestCon1.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


race condition running sasc_deploy in parallel

I think the other instance was writing it while this one was processing.
Likely just document not to run in parallel. I don't think there's much else to do.
Working on host <IP_ADDRESS>
Traceback (most recent call last):
  File "/opt/sasc/bin/sasc_deploy", line 251, in <module>
    check_connection(ip)
  File "/opt/sasc/bin/sasc_deploy", line 159, in check_connection
    known_hosts = paramiko.hostkeys.HostKeys(KNOWN_HOSTS_FILE)
  File "/usr/lib/python2.7/dist-packages/paramiko/hostkeys.py", line 63, in __init__
    self.load(filename)
  File "/usr/lib/python2.7/dist-packages/paramiko/hostkeys.py", line 101, in load
    e = HostKeyEntry.from_line(line, lineno)
  File "/usr/lib/python2.7/dist-packages/paramiko/hostkeys.py", line 335, in from_line
    key = ECDSAKey(data=decodebytes(key), validate_point=False)
  File "/usr/lib/python2.7/dist-packages/paramiko/ecdsakey.py", line 70, in __init__
    validate_point=validate_point)
  File "/usr/lib/python2.7/dist-packages/ecdsa/keys.py", line 47, in from_string
    point = ellipticcurve.Point(curve.curve, x, y, order)
  File "/usr/lib/python2.7/dist-packages/ecdsa/ellipticcurve.py", line 73, in __init__
    if self.__curve: assert self.__curve.contains_point( x, y )
AssertionError

It looks like a file lock on the known hosts file would be the right thing to do.
The best library for this looks to be at: https://fasteners.readthedocs.io/en/latest/examples.html#interprocess-locks


make sure AlcaHarvest CMSSW jobs only use a single core

The SiPixelAli AlcaHarvest jobs are assigned 4 cores since they need lots of disk space. The AlcaHarvest CMSSW job itself can only run single-threaded though, so make sure it only uses one core.

Code looks good. However, I'd suggest making a single handleNumberOfCores method for if funcname in ['merge', 'repack', 'alcaHarvesting']. Then from this if` you make another call for the repack tweak (without the multicore code).
Eventually we should fix it in the source (spec file) and set it to 1 for any job types or tasks that need to run as single core.

We can't "fix" it in the spec. The spec has them as single core jobs, but that is overridden in the job splitting, from that point on they are real multicore jobs. Multicore jobs whose CMSSW step only uses a single core, hence the needed override in the runtime CMSSW code.
Also, since you suggest a functional change/refactoring, can you open another issue on it later ? Or I can just make the change later. I do not want to change this version since I already pointed Tier0Ops at it and AFAIK it is actively being used.

Ahh, this alcaHarvesting is one step in a multi-steps job? Or is it a single job but one that you want to deceive gWMS as multicore?
About this patch, isn't the change simple enough that you can make it right from the beginning? Sorry for being picky, but the more I look at the WMCore code, the more I want to get rid of complications, duplications, very specific exceptions and patches everywhere.

Correct on the latter. I want gWMS to schedule it multicore, but the CMSSW step can only run single-threaded so far.
This is complicated, but it can't be done any other way at the moment.  Problem is that disk space estimates are so unreliable that this would be dangerous. For our workflows to function at all we rely on them having plenty of local disk space and that is allocated by the famous 20GB per core rule we ask of the sites. I just don't see how we can fit single core jobs that ask for and use 100GB of local disk into our job mix without total screwups and messes.

Which effectively means if you have a job that you know needs 100GB of local disk, you need to submit it as a 5 core gWMS job, no matter what the actual payload uses in terms of cores. Only way to do it.

@ticoann this seems to have been forgotten somewhat ?

I think Alan just wanted to make duplicated code to simple since the 3 methods are almost the same. (But I might be wrong). For me, I was just wondering
following part
exception:
    pass

Is there nothing need to be done when exception occurs?

If there is an exception it means that self.step.data.application.multicore.numberOfCores doesn't exist, which is fine, since then the job is already singlecore (that parameter not existing is historically interpreted as singlecore in our code).

We can wait merging this until Monday, I can review this and see if I can simplify the methods...

test this please

Dirk, do you need a new tag with this?

Ok, looked at the code and I can simplify this. I'd rather do it in a new commit and pull request though since I don't want to modify this one (we are already using it in the Tier0). Which means this should get merged and I'll need a new tag.

Hi Dirk, I will merge it and create the new tag then. In general we freeze the merge during testbed validation since code might affect in applications in cmsweb. (reqmgr, etc)
But in this case, it won't. Anyway, I think we should maintain Tier0 branch for T0 agent upgrade as we briefly discussed in the release meeting.


EJB request load balancing issue

Hello team,
We have a stateful EJB request load balancing issue, the EJB clients in cluster1/domain2 connect EJB servers in squada-n1/domain1 using clusterIP service (e.g t3://sample-domain1-cluster-squada-n1.sample-domain1-ns.svc.cluster.local:9011), we found most of the request were sent to EJB server 1, why not sent to other servers? Attach the snapshot of thread in different managed servers. Please let me know if you need more information.

AB#1335575

Slack link.

Johnny Shum  2 minutes ago
@Edward Burns It depends on the nature of the conversation.   If you get a client and repeatedly make request, it will go to the same server.  After you dispose it and recreate another client, it repeats the same behavior.  So, it depends on how many ejb client do you have and how long is the conversation.   Is there any reason, you are not using stateless ejb ?
https://docs.oracle.com/middleware/1213/wls/CLUST/best.htm#CLUST514
Also, there is a settings for load balancing ejb home in the weblogic-ejb.xml descriptor.
https://docs.oracle.com/cd/E24329_01/web.1211/e24973/ejb_jar_ref.htm#EJBPG582
Check on stateful-session-descriptor,  home-load-algorithm    Once a client is obtained, it will stick to it’s instance on a particular server and maintain the conversation until it ends or failed over.
Also, there is a settings in the weblogic-ejb.xml
docs.oracle.comdocs.oracle.com
Clustering Best Practices
This chapter recommends design and deployment practices that maximize the scalability, reliability, and performance of applications hosted by a cluster in WebLogic Server 12.
docs.oracle.comdocs.oracle.com
B weblogic-ejb-jar.xml Deployment Descriptor Reference
The following sections describe the EJB 2.1 elements in the weblogic-ejb-jar.xml file, the WebLogic-specific XML Schema-based (XSD) deployment descriptor file.


[Issue]:  微调promot的参数改了吗

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have checked #657 to validate if my issue is covered by community support

Describe the issue
usage: python -m graphrag.prompt_tune [-h] --config CONFIG [--root ROOT] [--domain DOMAIN] [--selection-method {all,random,top,auto}] [--n_subset_max N_SUBSET_MAX] [--k K] [--limit LIMIT]
                                      [--max-tokens MAX_TOKENS] [--min-examples-required MIN_EXAMPLES_REQUIRED] [--chunk-size CHUNK_SIZE] [--language LANGUAGE] [--no-entity-types] [--output OUTPUT]
python -m graphrag.prompt_tune: error: the following arguments are required: --config

Steps to reproduce
No response
GraphRAG Config Used
# Paste your config here

Logs and screenshots
No response
Additional Information

GraphRAG Version:
Operating System:
Python Version:
Related Issues:

确实官方改了 要看看参数是啥了


[local_auth]add biometrics support for iOS < 11.0.1

iOS versions below 11.0.+ has TouchID support but not FaceID.
With current implementation it just always return an empty list for those devices.
If you want to support it, you need to add "fingerprint" in a biometrics list in a case if there was no error.

A small comment -- but it's not exactly true that iOS versions < 11 have TouchID.  TouchID was introduced first in iOS 7 .. and "only" iPhones 5s or newer have it .. Not sure how worthwhile putting these checks (or feasibility even) to tighten it up ..

But if device will not have TouchID it will return an error and goes in other condition, I'm processing a case when check for biometrics was successful.

Does anyone know how to fix Setup step in a build-ipas?

I'm going through some old PRs to see what we can do with them. First, sorry for dropping this on the floor, we do appreciate your contributions though apparently we don't do a good job of showing it.
For this PR specifically, I'm going to close it because it is lacking tests and all our PRs need to have tests. Please don't hesitate to reach out on the #hackers-plugins channel on our chat server if you'd like help with this.


Added Algo/longest_increasing_subsequence [C++]

Fixes #488
By submitting this pull request I confirm I've read and complied with the below declarations.

[x] I have read the Contribution guidelines and I am confident that my PR reflects them.
[x] I have followed the coding guidelines for this project.
[x] My code follows the skeleton code structure.
[x] This pull request has a descriptive title. For example, Added {Algorithm/DS name} [{Language}], not Update README.md or Added new code.
[x] This pull request will be closed if I fail to update it even once in a continuous time span of 7 days.

Hi! Thanks for showing interest in contributing to this repository.
Please make sure that you have ticked the points mentioned in PR description correctly.
Thanks again!

@aashutoshrathi @singhpratyush

Please add an entry to README.

@manish1997 Please make the requested changes.

Closing due to inactivity.


Issue while processing Variable block length files - each block has specific number of records

Hi, I am having a variable block length file. Structure of my file below
Block_length field which is occurring after every 5 records in my case.
Value of Block_length is 28049. Each record length is 5609 bytes plus Block_length is 4 bytes  So each block has 5 records.  ((28049 - 4)/5609)
Record_header contains record_length value.
01 Record(1st record)
    05 Block_length  4 bytes
    05 Record_header   2 bytes
    05 remaining fields  5607 bytes
01 Record(2nd record)
    05 Record_header   2 bytes
    05 remaining fields  5607 bytes
01 Record(3rd record)
    05 Record_header   2 bytes
    05 remaining fields  5607 bytes
01 Record(4th record)
    05 Record_header   2 bytes
    05 remaining fields  5607 bytes
01 Record(5th record)
   05 Record_header   2 bytes
   05 remaining fields  5607 bytes
01 Record(6th record)
   05 Block_length  4 bytes
   05 Record_header   2 bytes
   05 remaining fields  5607 bytes
01 Record(7th record)
   05 Record_header   2 bytes
   05 remaining fields  5607 bytes
01 Record(8th record)
   05 Record_header   2 bytes
   05 remaining fields  5607 bytes
01 Record(9th record)
   05 Record_header   2 bytes
   05 remaining fields  5607 bytes
01 Record(10th record)
   05 Record_header   2 bytes
   05 remaining fields  5607 bytes

I don't know how to deal with Block_length field which is occurring after every 5 records. Is it possible to calculate the number of records per each block and split number of records based on block_length/record_length
Thanks!

I am able to write a simple python code to read block_length and copy (block_length-4) bytes to separate file. By doing this in loop, I am able to remove the extra bytes. Issue is resolved.


Add renew certificate github action

Runs the renew-certificate Juju action on the certbot-k8s charm every 2
months and on command.

Depends On: https://github.com/pedroleaoc/certbot-k8s/pull/7

Looks good, we can merge once the necessary secrets have been added.

The integration tests seem to fail because of microk8s, and is unrelated to this PR.


allow creation on remote (X11) displays

with X11 it is possible to open a window on a remote display.
i have a some users who are using this, allowing them to display the control interface on one computer, and the openGL output window on another computer - all from within one application. and they might even decide to switch the target display during runtime (obviously destroying the window first, then re-creating it on another DISPLAY)
now that we have switched our default windowing backend from GLX to GLFW, this is no longer possible.
i did a quick check to see whether I could trick glfw by setting the DISPLAY variable before calling glfwCreateWindow(), but alas! this does not work (as glfw requests the display only once, during glfwInit(), which can only be called once).
would it be possible to add a (possibly X11 specific) Window creation hint (e.g. glfwWindowHintString(GLFW_DISPLAY, ":0")) to allow setting a target display?
i'm aware that currently glfw maintains a global display for all its windows....

GLFW has a number of platform specific hints already, so a PR for this would likely be acceptable.

i'm aware that currently glfw maintains a global display for all its windows....

given the above, should a PR try to keep the current behavior (so selecting a display can only be done before the first window is created) or maintain a display for each window (which might get tricky if the different displays have different capabilities).
i tend towards the former, but then I think the proper function would be glfwInitHintString() (which does not exist yet!).
(note that i haven't checked the code recently, so i might have gotten things wrong)

Out of interest can you rework the application to call init after changing the DISPLAY environment variable?
A change to add per window displays would be pretty large, since the GLFW init uses the display variable to initialize a large number of variables used by all the systems in GLFW. So if a change to add this as an init hint would work for your use case then I think this would be the best way to go.

Out of interest can you rework the application to call init after changing the DISPLAY environment variable?

probably.
so do I understand correctly, that a PR should add a new glfwInitHintString() function (because the existing glfwInitHint() cannot take strings and is therefore not sufficient)

Yes, a new function glfwInitHintString(int hint, const char* value) similar to glfwWindowHintString(int hint, const char* value) looks like the way to do this.


Add new user type of reserved.

This allows for usernames, and emails connected to them to be reserved and not reused.
Use case, I manage an instance with open registration, and sometimes when users are deleted for spam (or other purposes), their usernames are freed up and they sign up again with the same information.
This could also be used to reserve usernames, and block them from being registered (in case an instance would like to block certain things without hardcoding the list in code and compiling from scratch).
This is an MVP, that will allow for future work where you can set something as reserved via the interface.

Codecov Report

Merging #24026 (994830b) into main (f521e88) will decrease coverage by 0.08%.
The diff coverage is 30.85%.

@@            Coverage Diff             @@
##             main   #24026      +/-   ##
==========================================
- Coverage   47.14%   47.06%   -0.08%
==========================================
  Files        1149     1163      +14
  Lines      151446   153738    +2292
==========================================
+ Hits        71397    72355     +958
- Misses      71611    72871    +1260
- Partials     8438     8512      +74

Impacted Files
Coverage Δ

cmd/dump.go
0.66% <0.00%> (-0.01%)
:arrow_down:

cmd/mailer.go
0.00% <0.00%> (ø)

cmd/manager.go
0.00% <0.00%> (ø)

cmd/manager_logging.go
0.00% <0.00%> (ø)

cmd/migrate_storage.go
5.76% <0.00%> (-0.12%)
:arrow_down:

cmd/restore_repo.go
0.00% <0.00%> (ø)

cmd/web.go
0.00% <0.00%> (ø)

models/actions/run.go
1.63% <0.00%> (-0.10%)
:arrow_down:

models/actions/runner.go
1.44% <ø> (ø)

models/organization/org_user.go
80.51% <0.00%> (-12.02%)
:arrow_down:

... and 64 more

... and 112 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

But how did you know it's a user or an org originally if the type is reserved?

Doesn't matter.
It can still be converted to a user or organization later on.

Have you seen what the content of this PR is at the moment?

Yes, that would be the easiest approach.

Yes, that would be the easiest approach.

added :)

Okay, we meant totally different things.
My

Yes, that would be the easiest approach.

was intended regarding the

To de-reserve you'd likely just delete the reservation to open it up again so it could be used again

…

Ah, I see. I have a test with my database without this change but just change the type to 2 from 0 of one account. But the account can still login.

@lunny I've now added a change to block that

🤖 🎺


Policy Based VPN gateway can be still created using PowerShell and Az CLI

The verbatim on this FAQ  should match as per verbatim on this document here.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 98d7fd30-80a8-f690-6191-d97ae4cac174
Version Independent ID: 3e15f99b-2f2d-7adc-a2d9-0b2184989db7
Content: Azure VPN Gateway FAQ
Content Source: articles/vpn-gateway/vpn-gateway-vpn-faq.md
Service: vpn-gateway
GitHub Login: @cherylmc
Microsoft Alias: cherylmc

Working on a PR


Fixed Prettier using plugins

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#1035  👈
#1034
#1033
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @CatChen and the rest of your teammates on  Graphite