[Device Support Request] TY0201 by _TZ3000_bjawzodf

Problem description
Round white temperature and humidity sensor with display purchased from Temu.
Only temperature and battery shows.
I attempted using the ts0201.py and replacing ("_TZ3000_qaaysllp", "TS0201") with ("_TZ3000_bjawzodf", "TY0201") and it seems to be working.
Solution description
Temperature, Humidity and Battery level support.
Screenshots/Video
Screenshots/Video
Before Quirk

After Quirk

Device signature
Device signature
{
"node_descriptor": "NodeDescriptor(logical_type=<LogicalType.EndDevice: 2>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.AllocateAddress: 128>, manufacturer_code=4417, maximum_buffer_size=66, maximum_incoming_transfer_size=66, server_mask=10752, maximum_outgoing_transfer_size=66, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=True, *is_full_function_device=False, *is_mains_powered=False, *is_receiver_on_when_idle=False, *is_router=False, *is_security_capable=False)",
"endpoints": {
"1": {
"profile_id": "0x0104",
"device_type": "0x0302",
"input_clusters": [
"0x0000",
"0x0001",
"0x0003",
"0x0402",
"0x0405"
],
"output_clusters": [
"0x0019"
]
}
},
"manufacturer": "_TZ3000_bjawzodf",
"model": "TY0201",
"class": "zigpy.device.Device"
}

Diagnostic information
Diagnostic information
{
"home_assistant": {
"installation_type": "Home Assistant OS",
"version": "2023.12.3",
"dev": false,
"hassio": true,
"virtualenv": false,
"python_version": "3.11.6",
"docker": true,
"arch": "x86_64",
"timezone": "America/New_York",
"os_name": "Linux",
"os_version": "6.1.63-haos",
"supervisor": "2023.12.0",
"host_os": "Home Assistant OS 11.2",
"docker_version": "24.0.7",
"chassis": "vm",
"run_as_root": true
},
"custom_components": {
"meross_lan": {
"version": "3.0.0",
"requirements": []
},
"hacs": {
"version": "1.33.0",
"requirements": [
"aiogithubapi>=22.10.1"
]
}
},
"integration_manifest": {
"domain": "zha",
"name": "Zigbee Home Automation",
"after_dependencies": [
"onboarding",
"usb"
],
"codeowners": [
"@dmulcahey",
"@adminiuga",
"@puddly",
"@TheJulianJES"
],
"config_flow": true,
"dependencies": [
"file_upload"
],
"documentation": "https://www.home-assistant.io/integrations/zha",
"iot_class": "local_polling",
"loggers": [
"aiosqlite",
"bellows",
"crccheck",
"pure_pcapy3",
"zhaquirks",
"zigpy",
"zigpy_deconz",
"zigpy_xbee",
"zigpy_zigate",
"zigpy_znp",
"universal_silabs_flasher"
],
"requirements": [
"bellows==0.37.3",
"pyserial==3.5",
"pyserial-asyncio==0.6",
"zha-quirks==0.0.107",
"zigpy-deconz==0.22.2",
"zigpy==0.60.1",
"zigpy-xbee==0.20.1",
"zigpy-zigate==0.12.0",
"zigpy-znp==0.12.0",
"universal-silabs-flasher==0.0.15",
"pyserial-asyncio-fast==0.11"
],
"usb": [
{
"vid": "10C4",
"pid": "EA60",
"description": "2652",
"known_devices": [
"slae.sh cc2652rb stick"
]
},
{
"vid": "1A86",
"pid": "55D4",
"description": "sonoffplus*",
"known_devices": [
"sonoff zigbee dongle plus v2"
]
},
{
"vid": "10C4",
"pid": "EA60",
"description": "sonoffplus*",
"known_devices": [
"sonoff zigbee dongle plus"
]
},
{
"vid": "10C4",
"pid": "EA60",
"description": "tubeszb",
"known_devices": [
"TubesZB Coordinator"
]
},
{
"vid": "1A86",
"pid": "7523",
"description": "tubeszb",
"known_devices": [
"TubesZB Coordinator"
]
},
{
"vid": "1A86",
"pid": "7523",
"description": "zigstar",
"known_devices": [
"ZigStar Coordinators"
]
},
{
"vid": "1CF1",
"pid": "0030",
"description": "conbee",
"known_devices": [
"Conbee II"
]
},
{
"vid": "0403",
"pid": "6015",
"description": "conbee",
"known_devices": [
"Conbee III"
]
},
{
"vid": "10C4",
"pid": "8A2A",
"description": "zigbee",
"known_devices": [
"Nortek HUSBZB-1"
]
},
{
"vid": "0403",
"pid": "6015",
"description": "zigate",
"known_devices": [
"ZiGate+"
]
},
{
"vid": "10C4",
"pid": "EA60",
"description": "zigate",
"known_devices": [
"ZiGate"
]
},
{
"vid": "10C4",
"pid": "8B34",
"description": "bv 2010/10",
"known_devices": [
"Bitron Video AV2010/10"
]
}
],
"zeroconf": [
{
"type": "_esphomelib._tcp.local.",
"name": "tube*"
},
{
"type": "_zigate-zigbee-gateway._tcp.local.",
"name": "zigate"
},
{
"type": "_zigstar_gw._tcp.local.",
"name": "zigstar"
},
{
"type": "_uzg-01._tcp.local.",
"name": "uzg-01*"
},
{
"type": "_slzb-06._tcp.local.",
"name": "slzb-06*"
}
],
"is_built_in": true
},
"data": {
"ieee": "REDACTED",
"nwk": 63935,
"manufacturer": "_TZ3000_bjawzodf",
"model": "TY0201",
"name": "_TZ3000_bjawzodf TY0201",
"quirk_applied": false,
"quirk_class": "zigpy.device.Device",
"quirk_id": null,
"manufacturer_code": 4417,
"power_source": "Battery or Unknown",
"lqi": 87,
"rssi": null,
"last_seen": "2023-12-25T18:54:54",
"available": true,
"device_type": "EndDevice",
"signature": {
"node_descriptor": "NodeDescriptor(logical_type=<LogicalType.EndDevice: 2>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.AllocateAddress: 128>, manufacturer_code=4417, maximum_buffer_size=66, maximum_incoming_transfer_size=66, server_mask=10752, maximum_outgoing_transfer_size=66, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=True, *is_full_function_device=False, *is_mains_powered=False, *is_receiver_on_when_idle=False, *is_router=False, *is_security_capable=False)",
"endpoints": {
"1": {
"profile_id": "0x0104",
"device_type": "0x0302",
"input_clusters": [
"0x0000",
"0x0001",
"0x0003",
"0x0402",
"0x0405"
],
"output_clusters": [
"0x0019"
]
}
},
"manufacturer": "_TZ3000_bjawzodf",
"model": "TY0201"
},
"active_coordinator": false,
"entities": [
{
"entity_id": "button.tz3000_bjawzodf_ty0201_identify",
"name": "_TZ3000_bjawzodf TY0201"
},
{
"entity_id": "sensor.tz3000_bjawzodf_ty0201_battery",
"name": "_TZ3000_bjawzodf TY0201"
},
{
"entity_id": "sensor.tz3000_bjawzodf_ty0201_temperature",
"name": "_TZ3000_bjawzodf TY0201"
}
],
"neighbors": [],
"routes": [],
"endpoint_names": [
{
"name": "TEMPERATURE_SENSOR"
}
],
"user_given_name": null,
"device_reg_id": "388644d1eb87a058dbda55635d46ea92",
"area_id": null,
"cluster_details": {
"1": {
"device_type": {
"name": "TEMPERATURE_SENSOR",
"id": 770
},
"profile_id": 260,
"in_clusters": {
"0x0000": {
"endpoint_attribute": "basic",
"attributes": {
"0x0004": {
"attribute_name": "manufacturer",
"value": "_TZ3000_bjawzodf"
},
"0x0005": {
"attribute_name": "model",
"value": "TY0201"
}
},
"unsupported_attributes": {}
},
"0x0001": {
"endpoint_attribute": "power",
"attributes": {
"0x0021": {
"attribute_name": "battery_percentage_remaining",
"value": 200
}
},
"unsupported_attributes": {}
},
"0x0003": {
"endpoint_attribute": "identify",
"attributes": {},
"unsupported_attributes": {}
},
"0x0402": {
"endpoint_attribute": "temperature",
"attributes": {
"0x0000": {
"attribute_name": "measured_value",
"value": 2360
}
},
"unsupported_attributes": {}
},
"0x0405": {
"endpoint_attribute": "humidity",
"attributes": {
"0x0000": {
"attribute_name": "measured_value",
"value": 3870
}
},
"unsupported_attributes": {
"0x0000": {
"attribute_name": "measured_value"
}
}
}
},
"out_clusters": {
"0x0019": {
"endpoint_attribute": "ota",
"attributes": {},
"unsupported_attributes": {}
}
}
}
}
}
}

Logs
Logs
[Paste the logs here]

Custom quirk
Custom quirk
"""Tuya TS201 temperature, humidity and optional illumination sensors."""
from zigpy.profiles import zha
from zigpy.profiles.zha import DeviceType
from zigpy.quirks import CustomCluster, CustomDevice
import zigpy.types as t
from zigpy.zcl.clusters.general import Basic, Identify, Ota, PowerConfiguration, Time
from zigpy.zcl.clusters.measurement import (
IlluminanceMeasurement,
RelativeHumidity,
TemperatureMeasurement,
)
from zigpy.zdo.types import NodeDescriptor
from zhaquirks.const import (
DEVICE_TYPE,
ENDPOINTS,
INPUT_CLUSTERS,
MODELS_INFO,
NODE_DESCRIPTOR,
OUTPUT_CLUSTERS,
PROFILE_ID,
)
class ValueAlarm(t.enum8):
"""Temperature and humidity alarm values."""
ALARM_OFF = 0x02
MAX_ALARM_ON = 0x01
MIN_ALARM_ON = 0x00

class TuyaTemperatureHumidityAlarmCluster(CustomCluster):
"""Tuya temperature and humidity alarm cluster (0xE002)."""
name = "Tuya Temperature and Humidity Alarm Cluster"
cluster_id = 0xE002

attributes = {
    # Alarm settings
    0xD00A: ("alarm_temperature_max", t.uint16_t, True),
    0xD00B: ("alarm_temperature_min", t.uint16_t, True),
    0xD00C: ("alarm_humidity_max", t.uint16_t, True),
    0xD00E: ("alarm_humidity_min", t.uint16_t, True),
    # Alarm information
    0xD00F: ("alarm_humidity", ValueAlarm, True),
    0xD006: ("temperature_humidity", ValueAlarm, True),
    # Unknown
    0xD010: ("unknown", t.uint8_t, True),
}

class NeoTemperatureHumidtyIlluminanceSensor(CustomDevice):
"""Neo temperature, humidity and illumination sensor."""
signature = {
    #  <SimpleDescriptor endpoint=1, profile=260, device_type=262
    #  device_version=1
    #  input_clusters=[0, 1, 1024, 57346]
    #  output_clusters=[25, 10]>
    MODELS_INFO: [("_TZ3000_bjawzodf", "TY0201")],
    ENDPOINTS: {
        1: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: DeviceType.LIGHT_SENSOR,
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                PowerConfiguration.cluster_id,
                IlluminanceMeasurement.cluster_id,
                TuyaTemperatureHumidityAlarmCluster.cluster_id,
            ],
            OUTPUT_CLUSTERS: [
                Time.cluster_id,
                Ota.cluster_id,
            ],
        },
    },
}

replacement = {
    ENDPOINTS: {
        1: {
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                PowerConfiguration.cluster_id,
                IlluminanceMeasurement.cluster_id,
                TuyaTemperatureHumidityAlarmCluster,
            ],
            OUTPUT_CLUSTERS: [
                Time.cluster_id,
                Ota.cluster_id,
            ],
        },
        2: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: DeviceType.TEMPERATURE_SENSOR,
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                TemperatureMeasurement.cluster_id,
                RelativeHumidity.cluster_id,
            ],
        },
    },
}

class ZemismartTemperatureHumidtySensor(CustomDevice):
"""Zemismart temperature and humidity sensor."""
signature = {
    #  <SimpleDescriptor endpoint=1 profile=260 device_type=770
    # device_version=1
    # input_clusters=[0, 1, 3, 1029, 1026, 61183]
    # output_clusters=[3, 25]>
    MODELS_INFO: [("_TZ3000_lfa05ajd", "TS0201")],
    ENDPOINTS: {
        1: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: zha.DeviceType.TEMPERATURE_SENSOR,
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                PowerConfiguration.cluster_id,
                Identify.cluster_id,
                RelativeHumidity.cluster_id,
                TemperatureMeasurement.cluster_id,
                0xEEFF,  # Unknown
            ],
            OUTPUT_CLUSTERS: [Identify.cluster_id, Ota.cluster_id],
        }
    },
}

replacement = {
    NODE_DESCRIPTOR: NodeDescriptor(
        0x02,
        0x40,
        0x80,
        0x1037,
        0x7F,
        0x0064,
        0x2C00,
        0x0064,
        0x00,  # Forcing capability 0x80 instead of 0x84 so AC Power = false
    ),
    ENDPOINTS: {
        1: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: zha.DeviceType.TEMPERATURE_SENSOR,
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                PowerConfiguration.cluster_id,
                Identify.cluster_id,
                RelativeHumidity.cluster_id,
                TemperatureMeasurement.cluster_id,
            ],
            OUTPUT_CLUSTERS: [Identify.cluster_id, Ota.cluster_id],
        }
    },
}

class RelativeHumidityX10(CustomCluster, RelativeHumidity):
"""Handles invalid humidity values."""
def _update_attribute(self, attrid, value):
    # x10 factor in measured_value`(attrid=0)
    if attrid == 0:
        value = value * 10
    super()._update_attribute(attrid, value)

class MoesTemperatureHumidtySensorWithScreen(CustomDevice):
"""Moes temperature and humidity sensor with screen."""
signature = {
    #  <SimpleDescriptor endpoint=1, profile=260, device_type="0x0302"
    #  input_clusters=["0x0000", "0x0001", "0x0003", "0x0402", "0x0405", "0xe002"]
    #  output_clusters=["0x0003", "0x000a", "0x0019"]>
    ENDPOINTS: {
        1: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: DeviceType.TEMPERATURE_SENSOR,
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                PowerConfiguration.cluster_id,
                Identify.cluster_id,
                TemperatureMeasurement.cluster_id,
                RelativeHumidity.cluster_id,
                TuyaTemperatureHumidityAlarmCluster.cluster_id,
            ],
            OUTPUT_CLUSTERS: [
                Identify.cluster_id,
                Time.cluster_id,
                Ota.cluster_id,
            ],
        },
    },
}

replacement = {
    ENDPOINTS: {
        1: {
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                PowerConfiguration.cluster_id,
                Identify.cluster_id,
                TemperatureMeasurement.cluster_id,
                RelativeHumidityX10,
                TuyaTemperatureHumidityAlarmCluster,
            ],
            OUTPUT_CLUSTERS: [
                Identify.cluster_id,
                Time.cluster_id,
                Ota.cluster_id,
            ],
        },
    },
}

Additional information
https://www.temu.com/tuya-smart-zigbee-temperature-and-humidity-sensor-smart-life-app-control-indoor-humidity-detector-works-with-zigbee-gateway-hub-g-601099531358811.html

Hi. My equipment works the same. Only temperature and battery shows.  TY0201, _TZ3000_bjawzodf

Try the custom quirk I listed on the original post. It should work for you as it is working for me.

I take it back. The humidity stopped working for whatever reason.

Hi. My custom quirk works.
ty0201.zip

Hi. My custom quirk works. ty0201.zip

Hello,
I just received one TY0201 by _TZ3000_bjawzodf device today and encountered the same situation as described in this issue. I could add the device to HA in ZHA, but I only got battery and temperature, but not humidity.
My HA is running on a raspberry PI 4 with ConBee II Zigbee stick.
After some search, I found this issue and tried the ty0201.zip custom quirk from @LukaszJedynowicz .
I think i managed to activate it ( I have no experience with custom quirks until now, I followed this description) correctly as I have an additional line in the Zigbee info of the device including the typo with the missing "i" from the ty0201.py (see screenshots).
But still the humidity is missing (also see screenshots).
I deleted and newly added the device to be sure.
Did you do anything additionally to get it running with humidity?
Thank you in advance!

It didn’t work for me at first either. Try restarting HA after applying the quirk and then adding the device again.

It didn’t work for me at first either. Try restarting HA after applying the quirk and then adding the device again.

Thanks for your answer.
I'm sorry I wasn't precise enough what I did, unfortunately I already did these steps.
In detail:

applied quirk
checked configuration (just to be sure...)
restarted HA
deleted device
newly added device
restarted HA
--> Device is running but still without humidity.

I have the latest HA core update and some other updates open, so I will take the time to do all updates now.
I will give an update tomorrow if that changes anything.

Update: I did not succeed in getting the humidity in home assistant. I will leave it as it is for the moment, maybe there will be some activity here in the future that helps me getting it running.
Thank you @idumzaes for you support!

I don't know if this helps but here are a couple of internal pictures and a link to all the endpoints on the Tuya website as I too am experiencing the same problem.

https://developer.tuya.com/en/docs/connect-subdevices-to-gateways/zigbee-sensor?id=K9ik6zvmhrfh6

Hi, I have two identical thermometers. The second thermometer only shows the temperature, but both provide humidity information in their diagnostics.
Diagnostics are identicaly.
Check your diagnostic information to see if there's humidity information available.
"0x0405": {
"endpoint_attribute": "humidity",
"attributes": {
"0x0000": {
"attribute_name": "measured_value",
"value": 4140
}
},
"unsupported_attributes": {
"0x0000": {
"attribute_name": "measured_value"
}
}
}
The thermometers are of very poor quality.

Hello,
I got the same device today TY0201  _TZ3000_bjawzodf firmware: 0x10013001
The problem with ZHA is a little bit different. I can see the humidity but not the temperature. Currently I have no quirks installed. I am working with the current HA Version

I am very interested in the temperature not so much in the humidity. Because of this I am looking forward for a fix. Below a screenshot from my device configuration.

Hi. My custom quirk works. ty0201.zip

I can confirm this custom quirk works. I have been using it for 2 weeks without any issues.

Unfortunately, this still didn't work for me.

Same here. The custom quirk doesn't work for me either.

How did you manage to pair the device? I tried to pair with PhosCon several times, also tried different devices, it never succeeded. Usually I'm using ioBroker with the Deconz adapter and RaspBee, but for testing I tried to pair using the Zigbee adapter in ioBroker and the Sonoff USB stick - that worked immediately.

I’ve been using ZHA with a Sonoff Zigbee 3.0 USB Dongle Plus.  I’ve never used PhisCon, Deconz, ioBroker or RaspBee. On Jan 27, 2024, at 3:10 PM, ThorstenDtk @.***> wrote:
How did you manage to pair the device? I tried to pair with PhosCon several times, also tried different devices, it never succeeded. Usually I'm using ioBroker with the Deconz adapter and RaspBee, but for testing I tried to pair using the Zigbee adapter in ioBroker and the Sonoff USB stick - that worked immediately.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

I paired this device using  ZHA and SkyConnect as dongle.

The custom quirk mentioned does not work for me either. I'm not experienced with quirk development, but I also don't know why it works for anyone (please someone explain if they know, I'm legit curious, not criticizing) - since on the surface it appears to make the replacement template identical to the original.

For me the quirk posted here didn't work either so I changed it a bit and now it works for me.
this Is my quirk.
`"""Tuya TY0201 temperature, humidity and optional illumination sensors."""
from zigpy.profiles import zha
from zigpy.profiles.zha import DeviceType
from zigpy.quirks import CustomCluster, CustomDevice
import zigpy.types as t
from zigpy.zcl.clusters.general import Basic, Identify, Ota, PowerConfiguration, Time
from zigpy.zcl.clusters.measurement import (
IlluminanceMeasurement,
RelativeHumidity,
TemperatureMeasurement,
)
from zigpy.zdo.types import NodeDescriptor
from zhaquirks.const import (
DEVICE_TYPE,
ENDPOINTS,
INPUT_CLUSTERS,
MODELS_INFO,
NODE_DESCRIPTOR,
OUTPUT_CLUSTERS,
PROFILE_ID,
)
class TemperatureHumidtySensorWithScreen(CustomDevice):
"""Temu temperature and humidity sensor with screen."""
signature = {
    #  <SimpleDescriptor endpoint=1, profile=260, device_type="0x0302"
    #  input_clusters=["0x0000", "0x0001", "0x0003", "0x0402", "0x0405", "0xe002]
    #  output_clusters=["0x0003", "0x000a", "0x0019"]>
    MODELS_INFO: [("_TZ3000_bjawzodf", "TY0201")],
    ENDPOINTS: {
        1: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: DeviceType.TEMPERATURE_SENSOR,
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                PowerConfiguration.cluster_id,
                Identify.cluster_id,
                TemperatureMeasurement.cluster_id,
                RelativeHumidity.cluster_id,
              ],
            OUTPUT_CLUSTERS: [
                Ota.cluster_id,
            ],
        },
    },
}

replacement = {
    ENDPOINTS: {
        1: {
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                PowerConfiguration.cluster_id,
                Identify.cluster_id,
                TemperatureMeasurement.cluster_id,
                RelativeHumidity.cluster_id,
            ],
            OUTPUT_CLUSTERS: [
                Ota.cluster_id,
            ],
        },
    },
}`

@gielie could you post it as a file? I think formatting broke it.

@gielie  I have the same problem: "Only temperature shown". I had a look at the code posted above; it's the same as in the quirk files above, but I don't see any difference. What did you actually change?

I had have the same issue: had no humidity only temperature in HA
Today I have upgraded to latest (2024.1.6) HA and added the custom quirk.
After that no measurement at all. Even the temperature and battery disappeared.
Then I stopped HA (docker container), removed the quirk and the cache in the custom quirk folder.
Started HA Docker container (on Raspberry Pi 4).
And it wokrs:

@gielie I have the same problem: "Only temperature shown". I had a look at the code posted above; it's the same as in the quirk files above, but I don't see any difference. What did you actually change?

I changed this part
input_clusters=["0x0000", "0x0001", "0x0003", "0x0402", "0x0405"]
output_clusters=["0x0019"]>
into this
input_clusters=["0x0000", "0x0001", "0x0003", "0x0402", "0x0405", "0xe002]
output_clusters=["0x0003", "0x000a", "0x0019"]>

Hi @gielie !
Thanks for sharing your config.
Let me ask two questions!

These modified lines are only source comments. How do these effect the configuration. There is also a typo in the first comment line "0xe002 missing closing quoutes.

input_clusters=["0x0000", "0x0001", "0x0003", "0x0402", "0x0405", "0xe002]
output_clusters=["0x0003", "0x000a", "0x0019"]>

Shouldn't the modification be made on the following lists two? How do you do it?

INPUT_CLUSTERS: [
Basic.cluster_id,
PowerConfiguration.cluster_id,
Identify.cluster_id,
TemperatureMeasurement.cluster_id,
RelativeHumidity.cluster_id,
],
OUTPUT_CLUSTERS: [
Ota.cluster_id,
],

Regards,
Gábor

This is very strange. Following the change to the quirks file, one of the sensors has started showing temperature and humidity but another is only showing temperature still.
I have tried repeatedly restarting HA and removing the cache file but no further change.

Hi, Iam trying to get this working also, I got all the same results as above except a working device.
I cannot find how the clear the cache for the quirks, can anyone point me in the right direction?
My feeling is that any change I make to the quirk is not loaded when I pair the device.
Thanks in advance.
Danny

@koningdde your quirks cache is in config\zha_quirks_pycache_. You can also see if your quirk is being used by looking at the device info

@koningdde your quirks cache is in config\zha_quirks\__pycache__. You can also see if your quirk is being used by looking at the device info

Thank you,
I should have been looking a little closer as it is states obvious in the folder.
I will restart this adventure now and see if the quirks is used again, indeed the quirk was not loaded at my configuation.

To add to the confusion I unpaired the working device, re-paired and the sensor returned to only showing temperature. It seems there is something more at play. I now have no idea how I got it fully working.

Weird results here also, i removed the devices, after pairing one is showing temperature, the other one has no data.
When pairing the endpoints looks fine.

Data is available on one device, but only temperature:

Seems to me we are on the wrong track or the device doen's communicate as i should, maybe a timing problem in the RF system.
I think the devices are rubbish :)

Weird results here also, i removed the devices, after pairing one is showing temperature, the other one has no data. When pairing the endpoints looks fine.

Data is available on one device, but only temperature:
Seems to me we are on the wrong track or the device doen's communicate as i should, maybe a timing problem in the RF system.
I think the devices are rubbish :)

Don’t give up, with my Quirk I have 5 devices working, I can remove them and add them without any problem.
But the main thing is that it is an ZHA problem in Z2MQTT they are all working out of the box.

Thanks, i don't feel like migrating to Z2MQTT yet, because i need to pair everything again,
Although i like the idea that i can manipulate the communication more.

After repeatedly deleting the device and re-adding worked using the custom quirk given on this thread.

After repeatedly deleting the device and re-adding worked using the custom quirk given on this thread.

Pretty much what I did to get mine working with ZHA and it’s been working ever since. Though I only have one of these sensors.

I have four of these devices, the same, and they act randomly - some of them register humidity, some of them temperature. I never got two readings at once. Suprisingly, they are ultra reliable under zigbee2mqtt, in contrast to other meters...

Here's the ty0201.py.zip that I ended up using which seems to work for me. I don't know if it continues to work, as of now, it does.
This is the file mentioned earlier in the thread, and with the changes from @gielie.   I made no other chances to it, I'm just posting it here as a file to make it easier to download :)

Mine stopt working after a couple of days, can’t get them to include anymore in zha, I switched to z2mqtt, no more troubles.
Someone needs to pick this up why zha doesn’t work but z2mqtt has no problem with this device.

Mine stopt working after a couple of days, can’t get them to include anymore in zha, I switched to z2mqtt, no more troubles.

Well that doesn't bode well.
Also, I can't get my 2nd of these devices to give me humidity, despite the same quirk is being applied as for the 1st one.
Weird.

This seems to be something very elementary at the protocol level query - as those particular meters doesn't work well here, while they work perfect in z2mqtt, but on the contrary - meters from Sonoff (eg. TH01) doesn't work well in z2mqtt while they are perfect here (ZHA) - i whish I have skills to compare differences, when the devices "talks" and how to debug that...

On damn. My single one of these stopped reporting humidity so I’m back to square one.
I’m thinking about switching to Z2MQTT but that’s a drastic move for just one device. I’ll probably just end up not using this sensor.

The quirk did not immediately work for me. But after some other tweaking and restarting home assistant it finally got the humidity. I did notice the same thing as @theusen, which is that for some reason the humidity gets marked as unsupported. Thus I thought to maybe map it to something Tuya specific... No idea if this is even sane, as it is entirely new to me.
`
"""Tuya TY0201 temperature, humidity and optional illumination sensors."""
from zigpy.profiles import zha
from zigpy.profiles.zha import DeviceType
from zigpy.quirks import CustomCluster, CustomDevice
import zigpy.types as t
from zigpy.zcl.clusters.general import Basic, Identify, Ota, PowerConfiguration, Time
from zigpy.zcl.clusters.measurement import (
IlluminanceMeasurement,
RelativeHumidity,
TemperatureMeasurement,
)
from zigpy.zdo.types import NodeDescriptor
from zhaquirks.const import (
DEVICE_TYPE,
ENDPOINTS,
INPUT_CLUSTERS,
MODELS_INFO,
NODE_DESCRIPTOR,
OUTPUT_CLUSTERS,
PROFILE_ID,
)
from zhaquirks.tuya.air import (
TuyaAirQualityTemperature,
TuyaAirQualityHumidity,
)
class TemperatureHumiditySensorWithScreen(CustomDevice):
"""Temu temperature and humidity sensor with screen."""
signature = {
    #  <SimpleDescriptor endpoint=1, profile=260, device_type="0x0302"
    #  input_clusters=["0x000", "0x0001", "0x0003", "0x0402", "0x0405"]
    #  output_clusters=["0x0019"]>
    MODELS_INFO: [("_TZ3000_bjawzodf", "TY0201")],
    ENDPOINTS: {
        1: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: DeviceType.TEMPERATURE_SENSOR,
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                PowerConfiguration.cluster_id,
                Identify.cluster_id,
                TemperatureMeasurement.cluster_id,
                RelativeHumidity.cluster_id,
              ],
            OUTPUT_CLUSTERS: [
                Ota.cluster_id,
            ],
        },
    },
}

replacement = {
    ENDPOINTS: {
        1: {
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                PowerConfiguration.cluster_id,
                Identify.cluster_id,
                TuyaAirQualityTemperature,
                TuyaAirQualityHumidity,
            ],
            OUTPUT_CLUSTERS: [
                Ota.cluster_id,
            ],
        },
    },
}

`

Hi @philippedeswert, I was just trying to make this work and saw your comment and tried it out, and I can confirm that it works for me too! I couldn't get the original quirk to work at all, even after several repairs and restarts.
I had to figure out the correct file format because your comment formatting is messed up, but after applying the changes, restarting and re-pairing I finally got the humidity to show! :tada:

Hi @philippedeswert, I was just trying to make this work and saw your comment and tried it out, and I can confirm that it works for me too! I couldn't get the original quirk to work at all, even after several repairs and restarts.
I had to figure out the correct file format because your comment formatting is messed up, but after applying the changes, restarting and re-pairing I finally got the humidity to show! 🎉

Could you share the file u used? Many thanks.

Hi @philippedeswert, I was just trying to make this work and saw your comment and tried it out, and I can confirm that it works for me too! I couldn't get the original quirk to work at all, even after several repairs and restarts.
I had to figure out the correct file format because your comment formatting is messed up, but after applying the changes, restarting and re-pairing I finally got the humidity to show! 🎉

Could you share the file u used? Many thanks.

I apologize for the bad formatting, the quoting/code block just did not want to work correctly. Had to rename the file to .txt, but I guess it should be ok.
Anyway I hope it helps. I suspect there is still some Tuya specific thing with the clusters/power as reconfiguring now works for the temperature and humidity but not for the battery,
ty0201.txt

philippedeswert

Great...with this querk, humidity is finally working for me after newly adding the device.
Thanks a lot!

Hi @philippedeswert, I was just trying to make this work and saw your comment and tried it out, and I can confirm that it works for me too! I couldn't get the original quirk to work at all, even after several repairs and restarts.
I had to figure out the correct file format because your comment formatting is messed up, but after applying the changes, restarting and re-pairing I finally got the humidity to show! 🎉

Could you share the file u used? Many thanks.

I apologize for the bad formatting, the quoting/code block just did not want to work correctly. Had to rename the file to .txt, but I guess it should be ok.
Anyway I hope it helps. I suspect there is still some Tuya specific thing with the clusters/power as reconfiguring now works for the temperature and humidity but not for the battery,
ty0201.txt

Great...with this querk, humidity is finally working for me after newly adding the device.
Thanks a lot!

Hi @philippedeswert, I was just trying to make this work and saw your comment and tried it out, and I can confirm that it works for me too! I couldn't get the original quirk to work at all, even after several repairs and restarts.
I had to figure out the correct file format because your comment formatting is messed up, but after applying the changes, restarting and re-pairing I finally got the humidity to show! 🎉

Could you share the file u used? Many thanks.

I apologize for the bad formatting, the quoting/code block just did not want to work correctly. Had to rename the file to .txt, but I guess it should be ok.
Anyway I hope it helps. I suspect there is still some Tuya specific thing with the clusters/power as reconfiguring now works for the temperature and humidity but not for the battery,
ty0201.txt

great work, this works for all my devices, many thanks.

How do you re-add it / factory reset it?
I can't seem to figure out how to get it work. Whenever I remove it from ZHA, and I start to pair it again, it just re-appears in the list.
It seems if I hold down the top-button on the device for more than 10 seconds, it seems to forget its connected to my ZHA. But once I add it again, it gets the old quirk added (I believe), despite I removed it and only use the new one.

Absolutely excellent @gielie  that did it. Now showing humidity. I hadn't realised that I had to remove the device and unpair the sensor by holding the button in until the signal strength on the display disappeared. I then paired as usual and voila it works.

I changed the network to Z2MQTT an it is working without problems, so i cannot test this anymore, but i hope for you guys this is working now.

Hi @philippedeswert, I was just trying to make this work and saw your comment and tried it out, and I can confirm that it works for me too! I couldn't get the original quirk to work at all, even after several repairs and restarts.
I had to figure out the correct file format because your comment formatting is messed up, but after applying the changes, restarting and re-pairing I finally got the humidity to show! 🎉

Could you share the file u used? Many thanks.

I apologize for the bad formatting, the quoting/code block just did not want to work correctly. Had to rename the file to .txt, but I guess it should be ok.
Anyway I hope it helps. I suspect there is still some Tuya specific thing with the clusters/power as reconfiguring now works for the temperature and humidity but not for the battery,
ty0201.txt

After applying this quick, my ZHA shows nothing - no temp, no humidity.

Someone wrote this morning that:

After applying this quick, my ZHA shows nothing - no temp, no humidity.

But that message is now gone. If that someone managed to fix that issue, please enlighten us :D I'm stuck at the same, after finally getting the device removed from ZHA (I think).

Usually restarting ZHA helps with that, but only after you added the device back... I had some cases when trying things out that the device would not report temp and/or humidity but would be fine after a ZHA restart.

Thank you philippe and gielie for your suggestions. I was not aware of the pycache folder, File Editor was hiding it. I tried to clear it, restart ZHA, HA, re-pair etc etc., but it keeps appearing with only "Firmware", no temperature or humidity.
I guess I'll let it be for now. Maybe it will change its mind eventually.

Thank you philippe and gielie for your suggestions. I was not aware of the pycache folder, File Editor was hiding it. I tried to clear it, restart ZHA, HA, re-pair etc etc., but it keeps appearing with only "Firmware", no temperature or humidity.
I guess I'll let it be for now. Maybe it will change its mind eventually.

Same here with @philippedeswert quirk I now don't even see temperature, which was only thing working out of the box.
I used txt file provided, I see that this quirk was applied but temperature and humidity is missing. (Did restart, remove device and cache, etc )

Thank you philippe and gielie for your suggestions. I was not aware of the pycache folder, File Editor was hiding it. I tried to clear it, restart ZHA, HA, re-pair etc etc., but it keeps appearing with only "Firmware", no temperature or humidity.
I guess I'll let it be for now. Maybe it will change its mind eventually.

Same here with @philippedeswert quirk I now don't even see temperature, which was only thing working out of the box.
I used txt file provided, I see that this quirk was applied but temperature and humidity is missing. (Did restart, remove device and cache, etc )

I hope you changed the txt extension back to .py?

Thank you philippe and gielie for your suggestions. I was not aware of the pycache folder, File Editor was hiding it. I tried to clear it, restart ZHA, HA, re-pair etc etc., but it keeps appearing with only "Firmware", no temperature or humidity.
I guess I'll let it be for now. Maybe it will change its mind eventually.

Same here with @philippedeswert quirk I now don't even see temperature, which was only thing working out of the box.
I used txt file provided, I see that this quirk was applied but temperature and humidity is missing. (Did restart, remove device and cache, etc )

I hope you changed the txt extension back to .py?

Yes I did as, I don't think it would be applied otherwise.

Thank you philippe and gielie for your suggestions. I was not aware of the pycache folder, File Editor was hiding it. I tried to clear it, restart ZHA, HA, re-pair etc etc., but it keeps appearing with only "Firmware", no temperature or humidity.
I guess I'll let it be for now. Maybe it will change its mind eventually.

You are not alone :) Same store, diagnostic shows that now temperature is not supported, but show nothing for both )))

Thank you philippe and gielie for your suggestions. I was not aware of the pycache folder, File Editor was hiding it. I tried to clear it, restart ZHA, HA, re-pair etc etc., but it keeps appearing with only "Firmware", no temperature or humidity.
I guess I'll let it be for now. Maybe it will change its mind eventually.

You are not alone :) Same story, diagnostic shows that now temperature is not supported, but show nothing for both )))
UPD: after yet another restart the humidity back to live, but not temperature )) Magic... I can read both values using Manage Zigbee Device, btw...
UPD2: hm, now Download Diagnostic shows that all *value attributes supported, I did restart of the HA and guess what... it works... May be my manual execution of the Read Values in the Manage Zigbee Device did the trick, not sure..
UPD3: similar comment here, so may be really reading values manually do the magic...

UPD3 seems to do it...same procedure as described, manual reading, temperature and humidity could be read but were not available as entities in the device, then restarting HA and - MAGIC - they're here and updating!
Let's see if it stays this way, if you compare my comment some days ago I was somehow able to get both values directly after adding the device with the custom quirk, but then the values stopped both updating.

Thank you philippe and gielie for your suggestions. I was not aware of the pycache folder, File Editor was hiding it. I tried to clear it, restart ZHA, HA, re-pair etc etc., but it keeps appearing with only "Firmware", no temperature or humidity.
I guess I'll let it be for now. Maybe it will change its mind eventually.

You are not alone :) Same story, diagnostic shows that now temperature is not supported, but show nothing for both )))
UPD: after yet another restart the humidity back to live, but not temperature )) Magic... I can read both values using Manage Zigbee Device, btw...
UPD2: hm, now Download Diagnostic shows that all *value attributes supported, I did restart of the HA and guess what... it works... May be my manual execution of the Read Values in the Manage Zigbee Device did the trick, not sure..
UPD3: similar comment here, so may be really reading values manually do the magic...

UPD3 seems to do it...same procedure as described, manual reading, temperature and humidity could be read but were not available as entities in the device, then restarting HA and - MAGIC - they're here and updating!
Let's see if it stays this way, if you compare my comment some days ago I was somehow able to get both values directly after adding the device with the custom quirk, but then the values stopped both updating.

Do you use any quirk? If yes which one?

Thank you philippe and gielie for your suggestions. I was not aware of the pycache folder, File Editor was hiding it. I tried to clear it, restart ZHA, HA, re-pair etc etc., but it keeps appearing with only "Firmware", no temperature or humidity.
I guess I'll let it be for now. Maybe it will change its mind eventually.

You are not alone :) Same story, diagnostic shows that now temperature is not supported, but show nothing for both )))
UPD: after yet another restart the humidity back to live, but not temperature )) Magic... I can read both values using Manage Zigbee Device, btw...
UPD2: hm, now Download Diagnostic shows that all *value attributes supported, I did restart of the HA and guess what... it works... May be my manual execution of the Read Values in the Manage Zigbee Device did the trick, not sure..
UPD3: similar comment here, so may be really reading values manually do the magic...

UPD3 seems to do it...same procedure as described, manual reading, temperature and humidity could be read but were not available as entities in the device, then restarting HA and - MAGIC - they're here and updating!
Let's see if it stays this way, if you compare my comment some days ago I was somehow able to get both values directly after adding the device with the custom quirk, but then the values stopped both updating.

Do you use any quirk? If yes which one?
Edit: yes it looks like it is finally working also for me without any quirk. Just read those values and restart the HA ( I recently also updated HA to newest version 2024.3.1, if anybody interested, but I am not sure if that has anything to do with this, so upgrade on your own risk)

I was using this querk from philipedeswert above https://github.com/zigpy/zha-device-handlers/issues/2862#issuecomment-1975062539. I am running HA 2024.2.5 at the moment. The values are still updating this morning :-)
I will try without querk after your comment and give an update later

Worked me too, doing the manual reading. Afterwards, humidity and temperature appears and seems to update. Hope it continues to work :)
Still using the quirk. Don't know if it will work without the quirk.

Worked for me too, doing the manual reading. Afterwards, humidity and temperature appears and seems to update. Hope it continues to work :) Thanks for bringing that idea out there!
Still using the quirk. Don't know if it will work without the quirk.

No, it doesn't work without quirk, even can't read values manually.

The version in this comment works for me ( needed to re-pair the device, then restart HA, as initially I had no temperature and no humidity)

I do have _TZ3000_zl1kmjqx and with quirk in this comment it is working for me.
I just did a reconfig + restart of HA, than it started working. Thank you!

The file from @philippedeswert worked for me too, thanks!

Summary of steps I took to get _TZ3000_bjawzodf working, using the above comments:

Download the quirk by @philippedeswert.
Move the file to your Home Assistant ZHA quirks directory (in my case /config/zha_quirks) and change the file extension to .py (since it is written in Python) so that you end up with /config/zha_quirks/ty0201.py.
Ensure your ZHA quirks directory is in your HA configuration.yaml:zha:
  custom_quirks_path: /config/zha_quirks

Remove the thermometer from HA if it was previously added.
Restart Home Assistant.
Add the thermometer to HA.
Verify under the device's 'Zigbee info' section in HA that the quirk is being used. Should say:Quirk: tuya_ty0201_thermometer.TemperatureHumiditySensorWithScreen

If the temperature and humidity values are not showing, go to the device in HA and select Triple Dots -> Manage zigbee device -> select TuyaAirQualityHumidity in clusters -> measured_value in attributes -> read attribute. Repeat the same for the TuyaAirQualityTemperature cluster. Restart HA again.

Thanks @holocronweaver and @philippedeswert for the file and the step-by-step summary of how to solve this problem.
Finally this worked for me!

Recently got to play with this device and I see the fix is already out there. Thanks to @philippedeswert  in https://github.com/zigpy/zha-device-handlers/issues/2862#issuecomment-1975062539
This is not the only request for support, there's also #2851 and #2701 so many people are using this sensor. Digging through github issues and installing custom quirks might be too much for some however.
@philippedeswert would you be so kind as to send a PR to add the quirk to the repo and help any future users? Otherwise I can do that for you.

@philippedeswert would you be so kind as to send a PR to add the quirk to the repo and help any future users? Otherwise I can do that for you.

I could, I just don't feel like the quirk is actually 100% correct. OTOH a PR might get some eyes on it that could make it work 100%. I'll see what I can do later tonight.

I looked over the thread and no solution really worked, so I worked on my own.
@philippedeswert I got this Tuya sensor from AliExpress, maybe its a different one with same signature?
Anyways this is my ty0201.py (custom quirk):
ty0201.py.zip

I looked over this thread and no solution really worked, so I worked on my own. @philippedeswert I got this Tuya sensor from AliExpress, maybe its a different one with same signature? Anyways this is my ty0201.py (custom quirk): ty0201.py.zip
It uses the TuyaLocalCluster class which the other Tuya thermometer quirks utilizes and it worked flawlessly from the beginning.

Good you got it working. I was not 100% sure of my fix and this weekend I finally should have some time to play around with getting the quirk updated to v2.

Maybe something changed in the firmware of these things. Last year I ordered three of these. Works perfect, all sensor are reporting. Last month I bought two more (same make/model and same device ID in Zigbee but the new ones won't display temp/hum (they did day one by the way). Re-adding does not show the sensors. Real weird.

This needs to be a quirk less thing, if it works. I only just bought these devices, and the quirk worked. I'd have thought after 6 months, this would have hit core.

Quirks are going to be there no matter what. If you manually load one written by a community member, help write tests and make sure it's in a state to merge into the library!

This quirk was already merged: https://github.com/zigpy/zha-device-handlers/pull/3201
It is in release 0.0.117 from June 26th: https://github.com/zigpy/zha-device-handlers/releases/tag/0.0.117
However it looks like Home Assistant hasn't last updated zigpy since June 14th: https://github.com/home-assistant/core/commit/dc0fc318b813795900e798053adfaecb0548a735
I am not sure how zigpy / device handler updates are distributed to Home Assistant, but I suspect it will show up in a future HA release.

I just used the instructions of @holocronweaver and it works but only after doing step 8.
I also created a zha quirks directory and created the ty0201.py but i thought it would be in the Home Assistant release allready?
Is it possible that the ty0201.py is updated so that an inexperienced user (meaning myself :) ) doesnot have to mess with the "Manage zigbee device" settings?
Just wanted to share my experience and thanks @holocronweaver for the detailed instructions!


somethin strange with colors

coronavirus :) ?
I understand that it happened because of SVG, but maybe somebody can help me to understand - why?
link to my SVG https://help.hrupkie.ru/assets/images/svg-icons/girl.svg

Hi @MariskaS ,
Maybe this is helping you.
If you want to overwrite the color add 'svg2sprite sourcefolder distinationFile --stripAttrs fill --stripAttrs stroke --stripAttrs id'
Else just use svg2sprite sourcefolder distinationFile

@MariskaS, is the issue solved? If yes, i will close this Issue...

@MariskaS, is the issue solved? If yes, I will close this Issue...

Hey) actually not(the first time I do not correctly understand @sbala69gmail .
I can send you the video with a problem and SVG if you want it) It Looks like this SVG generator cannot work with pictures with different colors. Maybe... I'm not sure) my email<EMAIL_ADDRESS>let me know if you wont to see a problem about that I talk

@jannicz  if I'm removing --stripAttrs fill --stripAttrs stroke --stripAttrs id that help's me to save colors of my SVG... I think we can close this issue, it was my mistake
I mean if I have different colors in SVG and I want to save these colors, I need to write:
"generate:svg-sprite": "svg2sprite ./src/assets/icons ./src/assets/sprites/sprite.svg --stripAttrs id",

Yes, the IDs are necessary if there are inline styles. I recommend not stripping any attributes if you want to preserve all styles. See here https://github.com/jannicz/ng-svg-icon-sprite#dealing-with-multi-color-svgs-containing-inline-styles
Happy to help


tokyonight-day where is my cursor now?

if activated tokyonight-day finding the cursor on the page becomes a real test, please make it a little brighter

you need to properly configure your terminal. Cursor colors are managed by the terminal

you need to properly configure your terminal. Cursor colors are managed by the terminal

thank you


High memory usage when the worker is updating data

The implementation with the Parallel gem uses multiple child processes, but because there are several levels of maps running within each other, and we're specifying 4 processes at each level, we're actually spawning something like 4x4x4 processes, each with it's own copy of the original process memory.
As a result, memory requirements are unnecessarily large when the worker is updating data, this needs to be addressed in a future release.

This is fixes with the use of threads instead of spawning new processes.
https://github.com/SpringerPE/cf-light-api/commit/59cd6185ced1f5d36cffbe48d04cb80caedde1dc

Nice one, thanks @simonjohansson. I had issues initially when trying threads, which is why I went for processes, so I'm glad it works now!


Verifier does not receive info if QR code is refreshed before information is sent over

Description of problem
Using the verifier feature, the verifier does not receive the prover's info if QR code is refreshed before information is sent over.
Expected behavior
If information is sent over, the screen should change to reflect the information sent over.

Verifier presents a QR code
Prover scans QR code
Verifier and Prover wait for connection to get stablished

Verifier waits for presentation (this screen is missing)
Prover waits for proof request

Prover sends information
Verifier receives information

Steps to reproduce

Verifier presents a QR code
Prover scans QR code and wait
Verifier refreshes the QR code
Prover sends information

Screenshots and/or log output
Environment

[ ] Occurs on Android (did not test)
[X] Occurs on iOS

Build #: 975
Android Device Model:
iOS Device Model: iOS iPhone SE
Workaround
Prover rescans new generated QR code
Severity

[ ] High
[X] Medium
[ ] Low

Checked the flow, the information is sent but is not opened automatically. The verifier will have to review the information sent over from the Contact's chat screen.


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target_49381d0f-a192-47d3-be06-841ece342cbe/pull/1 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-archive-test-target_49381d0f-a192-47d3-be06-841ece342cbe.git

Docs Build status updates of commit bc296af:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error: GitBranchDeletedOrForcePushed] Cannot sync git repo to specified commit because branch Release_Archive_main_2022-05-30-03-33-55 has been deleted or has been force pushed

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


New MySQL Backends

Given that Emysql seems to have reached end-of-life, and may never support transactions, it's time to start looking at alternative backends.
There are two newcomers to the mysql driver world:

https://github.com/mysql-otp/mysql-otp (which seems pretty actively developed)
https://github.com/gar1t/erlang-mysql (which is made by Garrett, who's a pretty okay dude)

I'm not sure how pooling goes with Garrett's , but with mysql-otp supporting poolboy, that makes for a pretty simple abstracted backend to use poolboy for both it, and then opening up the door to supporting postgresql.

New version uses mysql-otp with poolboy.


Failed to instantiate module when multiple apps in page (e.g., via Chrome Extension).

I have an angular chrome extension (content script) that gets injected into a page that already contains a different angular app (e.g., calendar.sunrise.am).
I'm using angular.bootstrap to manually run my app. But this causes the following familiar error -- I'm guessing because Angular is trying to automatically instantiate the existing ng-app directive:
Uncaught Error: [$injector:modulerr] Failed to instantiate module App due to:
Error: [$injector:nomod] Module 'App' is not available! You either misspelled the module name or forgot to load it.
If registering a module ensure that you specify the dependencies as the second argument.

My extension loads and runs fine, but I'd like to remove this error.
Is there a mechanism to suppress angular from inspecting the DOM and looking for "ng-app"?

I onced answered a similar (actually I think it's the same) question on SO: answer
It's been almost 2 years, so things might have changed a bit (but the general concept should still apply).
Wondering if @kentcdodds has some insights on this (since he's been there, done that :smiley:).

In any case, this is a general support question and as such would have been a btter fit for one of the appropriate support channels.
GitHub issues are reserved for bug reports and feature requests.
--
Closing as this isn't actionable, but feel free to continue discussion below.
If you think you ran into a bug or want to request a feature, please open a new issue with a detailed description.

Here's the extension if you want a working example. I think that @gkalpak's answer to my question two years ago outlines it quite nicely (thanks again @gkalpak!). Good luck!


Network field empty when editing a vSphere Node Profile

What kind of request is this (question/bug/enhancement/feature request): Bug
Steps to reproduce (least amount of steps as possible):

Create a vSphere Node Profile
Fill all info including Network field in 3. Scheduling section
Save the profile
Edit the profile

Result:

The Network field in 3. Scheduling section appears blank
The profile can not be saved without setting the Network field again

Environment information

Rancher version:  v2.1.5
Installation option (single install/HA): Single

Version - master 4/8, UI v2.3.0
Verified fixed


API as a record?

Is it possible to use this library with Servant's "API as a record" approach? And if not, do you have an idea of how difficult it would be to support it? At a glance, the types don't seem to fit, but I don't understand Servant well enough to gauge what the patch would look like.

Just discovered https://hackage.haskell.org/package/servant-0.20.1/docs/Servant-API.html#t:ToServant which seems to do the trick.


Typo in CustomIC energy function

there is a extra "s" in the IClist variable. The enrgy function should be read as:
energyIC = ("step(d-dinit)IClist(d)step(dend -d)fstep(r-lim);"
"f=0.5(1. + tanh(mu(rc - r)));"
"d=abs(idx2-idx1)")

This is fixed in the Dev version and should be available in release 1.0.1.


Crash whenever the signature of a static method on a class_name class with no extends can be compiled

Tested versions

Tested on v4.4 dev 1 mono (v4.4.dev1.mono.official [28a72fa43])
I did not get this crash before that version

System information
Godot v4.4.dev1.mono - Windows 10.0.22631 - GLES3 (Compatibility) - NVIDIA GeForce RTX 3060 Laptop GPU (NVIDIA; <IP_ADDRESS>23) - Intel(R) Core(TM) i5-10300H CPU @ 2.50GHz (8 Threads)
Issue description
It's a crash, also title
Steps to reproduce

Create a class with the class_name keyword
Give it a static method
Fill the signature of the static method from a different class (in this case extending Node)
See the entire editor crash silently

Minimal reproduction project (MRP)
bug.zip
Open the project, go into the script (ScriptBNode) that doesn't compile and type true as the function's parameter to try to get it to compile

This bug easily fkn happens whenever anyone uses a Utils class...
Please fix this quickly as it is extremely reliable to produce (on my machine anyways) and it's obnoxiously annoying to work with. Using an external editor doesn't fix the problem and only when saved with the external editor and after opening Godot with the saved file, does it not crash anymore, i.e. only when Godot doesn't try to recompile that specific line...

You'll find that you get more help if you're polite and less demanding
Also I can't get this to happen, following your instructions, nothing crashes, so this doesn't "easily fkn happen"
Please provide more details as this isn't enough to replicate this bug

This is a duplicate of:

https://github.com/godotengine/godot/issues/96239

Which has already been fixed, the fix will be available in dev2 or if you build from source

That's why I haven't seen it reported xd
(has it really been fixed this fast??? 👏🏼👏🏼👏🏼)

We have a very fast fix time some times! Always good to search closed issues as well! And no problem!


Update createUiDefinition.json - AlertIQ

Change(s):

Update some names of data connector
Reason for Change(s):
Some data connector names have changed from the old version

@v-spadarthi Please look into this.

Hi @sreedharande , can you please check and provide feedback on above. Thanks

Hi @sreedharande , can you please check and provide feedback on above. Thanks

Hi @sreedharande , can you please check and provide feedback on above. Thanks

Hi @sreedharande , can you please check and provide feedback on above. Thanks

Hi @sreedharande , can you please check and provide feedback on above. Thanks

Hi @sreedharande , can you please check and provide feedback on above. Thanks

Hi @v-rucdu
Can you please review and provide your feedback or assign it to someone who can review it.
Thanks


Update CommonAnnotatedKey fields scope to be accessed by derived classes

Description

Update CommonAnnotatedKey private fields bytes and base64Key to properties with private setter, so their values can be accessed by derived classes.

close for now, if needed in future will revisit


feat: Extend WebAuthnIdentity with AuthenticatorAttachment

This PR extends the WebAuthnIdentity with authenticatorAttachment. In addition, the constructor is made public to allow downstream projects to construct it more easily.
Description
This is required for Internet Identity to record the authenticatorAttachment on devices being registered.
Fixes # (issue)
How Has This Been Tested?
There are no tests for WebAuthnIdentity. However, similar code has been tested in II code.
Checklist:

[x] My changes follow the guidelines in CONTRIBUTING.md.
[x] The title of this PR complies with Conventional Commits.
[x] I have edited the CHANGELOG accordingly.
[x] I have made corresponding changes to the documentation.

Approved - you can squash and merge when you're ready, or I can handle it if you don't have the correct permissions


Add Boba Avax, Boba BNB and Boba BNB testnet

As per: https://github.com/safe-global/safe-singleton-factory/pull/79/files
Boba on testnet Avax is sadly broken.

Hey @InoMurko, could you create an issue to add new chain? some information is required to review.

Hey @InoMurko, could you create an issue to add new chain? some information is required to review.

Sure. Does this prevent merging this PR?

Hey @InoMurko, could you create an issue to add new chain? some information is required to review.

Sure. Does this prevent merging this PR?
It is a helpful information to review this PR like RPC URLs and this type of things. I can see that you have three chains in this PR, then you can add the other chains info in additional information form. I'll review this PR after the issue, ping me when ready.

Hey @InoMurko, could you create an issue to add new chain? some information is required to review.

Sure. Does this prevent merging this PR?

It is a helpful information to review this PR like RPC URLs and this type of things. I can see that you have three chains in this PR, then you can add the other chains info in additional information form. I'll review this PR after the issue, ping me when ready.

Issues open!


Update CreateApp docs (rebased onto develop)

This is the same as gh-1055 but rebased onto develop.

Updates the web CreateApp page to address a couple of small issues:

previous instructions specified that the app directory itself was on the PYTHONPATH whereas it should be the parent directory that is on the PYTHONPATH.
import statements in the urls.py example don't work with 5.0 OMERO

To test, follow the CreateApp instructions until you have the "Welcome" message at a new app /index.html.

--rebased-from #1055


feat: add self-join button and api to cases

You can join dedicated case channels, if it is not closed. You cannot join non-dedicated case channels and there is no concept of 'subscribe'

@mvilanova Ready for review now if you'd like to take another look, I took a pass.


Compojure support

The following middleware adds route information to spans when using Compojure:
(ns steffan-westcott.clj-otel.api.trace.http
  (:require
   [compojure.core :as compojure]))

(defn wrap-compojure-route
  "Ring middleware to add matched Compojure route to the server span and Ring
   request map."
  [handler]
  (compojure/wrap-routes handler (fn [route-handler]
                                   (trace-http/wrap-route route-handler (comp second :compojure/route)))))

It can be used in the same way as steffan-westcott.clj-otel.api.trace.http/wrap-reitit-route.
To avoid adding a dependency on Compojure for projects that do not use it, some optional-require mechanism would be needed. I don't know how people commonly do that, but the following seems to work:
(ns steffan-westcott.clj-otel.api.trace.http)

(def wrap-compojure-route
  "Ring middleware to add matched Compojure route to the server span and Ring
   request map."
  (try
    (when-let [compojure-core-wrap-routes (requiring-resolve 'compojure.core/wrap-routes)]
      (fn [handler]
        (compojure-core-wrap-routes handler (fn [route-handler]
                                              (trace-http/wrap-route route-handler (comp second :compojure/route))))))
    ;; `requiring-resolve` throws `java.io.FileNotFoundException` when the namespace cannot be required:
    (catch Exception _)))

The above code does not capture the full route if compojure/route is used: https://github.com/weavejester/compojure/issues/192#issuecomment-1850993709
The following code works:
(defn- extract-compojure-route
  [request]
  (let [{:compojure/keys [route route-context]} request]
    (str route-context (second route))))

(defn- wrap-compojure-route
  "Ring middleware to add matched Compojure route to the server span and Ring
   request map."
  [handler]
  (compojure/wrap-routes handler (fn [route-handler]
                                   (trace-http/wrap-route route-handler extract-compojure-route))))

@devurandom Thank you for your help. Building on your insight, I've added support for Compojure route data and an example showing its use.


BLU10 - Exercise Notebook - Q5 is expecting a subsetted matrix

BLU10 - Non Personalized Recommender Systems
Exercise Q5 is expecting a subsetted matrix
Q5 asks for an array with the Top-7 items with at least 10 ratings. The indices are expected to refer only to items with more than 10 ratings, thus subsetting the original matrix. Answers referring the full ratings matrix will not pass the assert.
Possible Fixes
The question should either change its assert to contain both scenarios or add details in the PyDoc description.
Addtional Details
As mentioned by @minhhoang1023, @ccosta03 and @JTPenedo.

The PyDoc already mentions the need to subset the matrix.


VBA vs. Visual Basic

VBA != vb
I can only search for Visual Basic. It returns mostly .net code and that is just noise.
VBA should be separate.

Do they use different sets of file extensions?

VBA is embedded in ms office apps.
If files are exported then default is .bas
Sometimes .mod for modules or
You can make up your own.
Distribution is often inside an office app
So extensions could be
Xls, xlsx, mdb, accdb, doc, docx etc. Ad nauseum...
Someone searching for VBA is typically
Not interested in .net
Thanks
Peter
E.g.
https://www.bing.com/search?q=peterennis+github&PC=U316&FORM=CHROMN
These are VBA repos, not visual basic, so the default github
Description is false
Sent from Yahoo Mail on Android
From:"Paul Chaignon"<EMAIL_ADDRESS>Date:Sat, May 23, 2015 at 8:33 AM
Subject:Re: [linguist] VBA vs. Visual Basic (#2418)
Do they use different sets of file extensions?
—
Reply to this email directly or view it on GitHub.￼

Linguist currently associates the following extensions with Visual Basic:

.vb
.bas
.cls
.frm
.frx
.vba
.vbhtml
.vbs

Which of these are VBA? Visual Basic?

Not .vb
Not .frx
Not .vbhtml
Not .vbs
However, I could create all them if I wanted :)
Removing those would at least reduce the search spam.
In the spirit of cross platform development, ms
Honest embracement of os(?), and github expansion,
Perhaps you could have "Microsoft Office VBA"
As the 'language' selection?
Thanks,
Peter
Sent from Yahoo Mail on Android
From:"Paul Chaignon"<EMAIL_ADDRESS>Date:Sun, May 24, 2015 at 3:37 AM
Subject:Re: [linguist] VBA vs. Visual Basic (#2418)
Linguist currently associates the following extensions with Visual Basic:
.vb.bas.cls.frm.frx.vba.vbhtml.vbs
Which of these are VBA? Visual Basic?
—
Reply to this email directly or view it on GitHub.￼

VBA is actually just embedded classic VB. Modern VB.NET uses .vb, but classic VB will use the older extensions like .bas and .frm.
.vbs is VBScript, which is its own language altogether.

Yes. The issue us that search for VBA returns too much github spam as it brings back all results for VB. Ironically I find better results using google while searching the universe in comparison to the "targeted" subset of github.

I confirm :+1:
.vb => VB.NET
.bas, .cls, .frm => VBA (or to be more precise, it's originally VB6 extensions)
.frx => binary file associated with .frm
.vbs => VB Script, sort of sub-langage based on VB6
.vba and .vbhtml => no idea
If we could search for VBA code only, it will be very helpfull for VBA coders, have mercy for them ;)

I agree with chadebeauche - also .vbs means VB Script what it's scripting language - it's came form VB family but is different from VB.NET and VBA.
We can consider VBS as an old, but still is in use, and should be counted as separated language.
BTW similar extension .svb it's also VB Script file but it's used internally in TestComplete for creating automatic test scripts - I'm not sure but probably should by considered as separate category. Typically .svb file contains 70-90% of VBS and 10-30% of TestComplete specific features.

This issue persists. Some VBA repos are even being labeled as TeX. Go figure.

I have passed this issue/proposal to the Office documentation team - asking them to take over.

The situation hasn't changed to my knowledge.

I have renewed my request to the "doc people" of Office at Microsoft to take care of this.

AFAIK, this is still an issue.

It is, but priority at MS seems low.

Could this be reopened? As far as I can tell, this is still unresolved and it applies to latest linguist version.

My MS contact must have gone cold; no response yet.
Could someone else perhaps pick this up? Like @linda-editor ?

@GustavBrock I'll see what I can do about finding the right person to address this.

I agree this VBA-vs-VB-vs-VB.net confusion is a problem, for Linguist as well as search engines in general, but it's not clear what is being requested here. As covered in the readme, Linguist ignores blobs, and XLSX/XLSM files (where most VBA code resides) are zipped blobs. Is there a specific change to Linguist that people are looking to Microsoft submit a PR for?

@dmahugh : Browse the thread from the start. And VBA does not reside in xlsx files. However, most code has been exported to .bas or .cls text files for the benefit of the GitHub parser and for easy perusing by the visitors. If .frm files exist in a repository, it is VB6, not VBA.

Actually, you can have .frm files in a VBA project, if you have a "form" (basiclly, a GUI). BTW, it's by far the less common use-case.
+1 for the answer : the point here is to make a distinction between :

.vb => .NET world
.cls/.bas/.frm => VB6/VBA world

To all of you on this thread, if you have an actionable suggestion, please submit a PR. If you're looking for documentation on VBA, please see https://docs.microsoft.com/en-us/office/vba/api/overview/.

I see a similar pattern:

.vb => VB.NET
.vbs => VBScript
(.bas and/or .cls) but no .frm => VBA
(.bas and/or .cls) and .frm => VB6 or VBA (Word, Excel)

I've tried to write a PR, but have run into some issues. Any help would be greatly appreciated.

@zspitz I have no Ruby or linguist experience. Thanks for taking this on :+1:

@peterennis That makes two of us 😊.

@zspitz That makes three of us. Sorry.

ping @Linda-Editor

Question: should the language name be VBA or Visual Basic for Applications?

The name is Visual Basic for Applications, and VBA is the abbreviation for this.

@GustavBrock The name of SQL is Structured Query Language, and yet the name in languages.yml is listed as SQL. Same for eXtensible Markup Language, HyperText Markup Language,  ANother Tool for Language Recognition, Comma-Separated Values and Cascading Style Sheets.
IMO, as long as the acronym has wide acceptance, I think makes more sense to use the acronym instead of the alternative mouthful.

Well, you only asked for the name, not what to use in languages.yml - where VBA will be fine in my opinion.

How to make a distinction between VB6 and VBA and VB.NET? The .vbp and .vbw extensions are unique to the VB6 language. Thus, if any VBA files are accompanied by files with .vbp and .vbw extensions, than all repository is Visual Basic 6.0. Just look on any of my repositories on VB6.
In short:
VB6 - if (.bas and/or .cls and/or .frm and/or .frx) and .vbp and .vbw exist.
VBA - if (.bas and/or .cls and/or .frm and/or .frx) but no .vbp and .vbw exist.
VB.NET - if .vb exists.
VBScript - .vbs
Note also that any .xlsm file it implies the presence of VBA. Also note that VB6 is the only programming language among these (it is fully compiled).


Update the RFC to transition to Step 2. Does not include update ALL RFCs to change from did:sov...

This step SHOULD include changing (almost) all references from "did:sov..." to "https://didcomm.org/".  Almost, because at least this RFC has some references that need to stay the same.

Should all of the other RFCs be updated in this PR or a follow up?

Vote taken on community call on 23 Sep 2020: agreed to merge.


Tidy up analyse_gee_data.py output

[ ] Write out a single JSON file containing network centrality values for all sub-images
[x] Write sub-images to a sub-directory
[ ] Only write color and un-thresholded NDVI images if necessary (or put them in a sub-directory)

On top of this, should we download and unzip GEE results to the output directory, rather than the tmp folder?
Do we actually want to keep the tif files?

See changes/comments in #107


[zh]Update content/zh/blog/_posts/2018-10-01-health-checking-grpc.md

please squash your commits into one.

Please help to review this PR

/lgtm

/approve


Internal driver error   No enum constant org.jkiss.dbeaver.ext.greenplum.model.GreenplumExternalTable.FormatType.p   No enum constant org.jkiss.dbeaver.ext.greenplum.model.GreenplumExternalTable.FormatType.p

System information:

Operating system (distribution) and version: Windows 10 Enterprise Version:1809
DBeaver version: Version <IP_ADDRESS>008180601
Java version: 1.8.0_131-b11
Additional extensions

Connection specification:

Database name and version: Greenplum Database 5.26.0
Driver name: Greenplum
Do you use tunnels or proxies (SSH, SOCKS, etc)?: None

Describe the problem you're observing:
When attempting to browse specific schemas in the database navigator, dbeaver throws the error below:
Internal driver error
No enum constant org.jkiss.dbeaver.ext.greenplum.model.GreenplumExternalTable.FormatType.p

Steps to reproduce, if exist:
Open the specific schema again.
Include any warning/errors/backtraces from the logs

Fixed

verified


api input field hidden by keyboard

The OSK does hide the API key input field. Not good for manual input. Device Xperia X.

A, you already released the fix. Good. I can confirm this is fixed. So lets close this issue. Thanks for the work.

Thanks for your help!


Update test and build deps

Coveralls is taking forever again ... no idea why that happens sometimes now.

Coverage remained the same at 100.0% when pulling 915bc926e7a442431738eccb916ff4dd9903e8f8 on update-dev-deps into 9e0ef633bc6df27aaea8046b7aa2dbc4b1540af6 on master.


WIP refactor(@embark/blockchain): consistently merge config accounts and node accounts

This is most definitely a WIP.
Main points:
eth_accounts and personal_listAccounts do exactly the same things in geth's source code (see here and here in the same file; parity should be reviewed too) and therefore web3.eth.getAccounts() and web3.eth.personal.getAccounts() are the same thing. The web3 docs website is not up-to-date and the published docs are misleading and incorrect. Compare these:
https://web3js.readthedocs.io/en/1.0/web3-eth.html#getaccounts
https://github.com/ethereum/web3.js/pull/2124
So getAccounts(), either eth or personal, simply returns a list of accounts controlled by the node.
What we're really doing in our blockchain connector's provider and the blockchain proxy is making config accounts look as if they're node accounts.
In the case of the provider, the accounts list was always being overwritten by the initial output of getAccounts combined with config accounts.
In the case of the proxy, we were concat'ing the config accounts onto a live result of eth_accounts RPC.
So this WIP PR is trying to make the operations consistent, and it does seem successful. I added something like this to a fresh embark_demo:
accounts: [
      {
        nodeAccounts: true, // Accounts use for the node
        numAddresses: "2", // Number of addresses/accounts (defaults to 1)
        password: "config/development/password" // Password file for the accounts
      },
      // Below are additional accounts that will count as `nodeAccounts` in the `deployment` section of your contract config
      // Those will not be unlocked in the node itself
      {
        privateKey: "<some valid key>"
      }
]

When connecting from a node REPL in a fresh project (npm init -y, not an embark dapp) that only has<EMAIL_ADDRESS>installed, I compared the output of getAccounts when connecting to 8545 vs 8555 of embark blockchain. I did the same in a browser with web3 beta.37. And I compared those with the output in embark run cli dashboard. I made sure to use the web3.eth.personal.newAccount() method and ran getAccounts() in all the environments and I got consistent and expected results, i.e. [eth|personal].getAccounts() is always up-to-date after adding node accounts with personal.newAccount(), and 8555 lacks the { privateKey: "<some valid key>"} account.
I'm not 100% sure yet that the output of the account parser is being handled in exactly the same way when comparing the blockchain connector's provider with embark-blockchain-process/src/{blockchain,proxy}. Also, seems like some more refactoring could be done to reduce duplicated logic.
This PR doesn't doesn't directly address the decisions we're making as a team re: account config semantics, but hopefully it sheds some light on things that are easily misunderstood. I'm 100% sure I was totally confused before I dove into the code and started messing with web3, our provider, and our proxy. I believe these changes would fix @emizzle's problem re: lightchain and the manual creation of node accounts with web3.

Really nice work, I love the descriptiveness and the lengths you went through to really try to tackle this issue!!
Unfortunately, I do not think this will fix the issue and would still require a restart, because web3.eth.getAccounts will not call the eth_accounts RPC method unless it has no wallet accounts.  And with Embark, we always have wallet accounts.
Embark is adding all node accounts plus wallet accounts to its web3 wallet, and thus will always return those accounts for any subsequent web3.eth.getAccounts calls. Because we are adding merged accounts to the wallet, we are forced to keep that wallet up-to-date with node account changes. Therefore, I believe what we could be doing here (IMO, this is still a bandaid approach and could be solved with broader strokes), is that we could be intercept any personal_createAccount RPC calls, and append the new account to Embark's web3 wallet. I have no idea if this is actually possible, mainly because I'm not sure if we have access to the PK at that point.

Really nice work, I love the descriptiveness and the lengths you went through to really try to tackle this issue!!

Thanks!

Unfortunately, I do not think this will fix the issue and would still require a restart, because web3.eth.getAccounts will not call the eth_accounts RPC method unless it has no wallet accounts (web3 PR that implemented these changes). And with Embark, we always have wallet accounts.

Despite appearances, i.e. the branch names involved in the web3 PR you linked to, those changes aren't and won't be part of web3 1.0.0. Based on community feedback, the maintainer decided that 1.x releases will be based off 1.0.0-beta.37 and all the refactoring and features, etc. made after beta.37 will become part of 2.x. See:
https://github.com/ethereum/web3.js/blob/1.x/packages/web3-eth/src/index.js#L226
https://github.com/ethereum/web3.js/blob/release/1.0/packages/web3-eth/src/index.js#L226
What makes the true behavior of eth.getAccounts even more confusing is that web3's docs website provides wrong information about how it works. Fixes to the docs are already merged in web3's repo but they haven't been deployed yet.
Summary:  eth.getAccounts of web3 beta.37 and 1.x going forward always makes the eth_accounts RPC call. Also, it never mixes the list of accounts returned from the RPC call with accounts (if any) in web3.eth.accounts.wallet.

Embark is adding all node accounts plus wallet accounts to its web3 wallet, and thus will always return those accounts for any subsequent web3.eth.getAccounts calls. Because we are adding merged accounts to the wallet, we are forced to keep that wallet up-to-date with node account changes. Therefore, I believe what we could be doing here (IMO, this is still a bandaid approach and could be solved with broader strokes), is that we could be intercept any personal_newAccount RPC calls, and append the new account to Embark's web3 wallet. I have no idea if this is actually possible, mainly because I'm not sure if we have access to the PK at that point.

This PR already fixes it: there's no need to intercept personal_newAccount and a restart is not required after calling web3.eth.personal.newAccount.
Let's back up a bit...
What embark is actually doing, effectively, is making the accounts from the dapp config behave like unlocked node accounts.
The reason the list of accounts returned by eth.getAccounts wasn't getting updated is that we were always overwriting the results of the call with a never-updated list. See on the master branch:
https://github.com/embark-framework/embark/blob/master/packages/embark-blockchain-connector/src/provider.js#L136
Once the misunderstanding about the behavior of eth.getAccounts is cleared up it's easier to understand what has to change. That's the realization I made, that realization informed the changes in this PR (please have another look), and those changes do solve the problem.
While it may seem like a bandaid approach, I don't think there's another good solution. In fact, making all of the accounts seem like node accounts from the perspective of browsers and the cli dashboard — which is exactly what we were doing and this PR doesn't change that, just improves it — is probably the best and simplest thing we can do, i.e. for the sake of a smooth development experience. Dapp authors simply want to use their config accounts without having to worry about wallet management during development.

Thanks for clearing that up, I will take another look. The web3 branching names definitely were causing the confusion here. It makes complete sense that the provider could be the source of error.
Can we assume that what I described will be included in web3 2.0? If it is, we may have this problem again as soon as we make the switch.

While it may seem like a bandaid approach, I don't think there's another good solution. In fact, making all of the accounts seem like unlocked node accounts from the perspective of browsers and the cli dashboard — which is exactly what we were doing and this PR doesn't change that, just improves it — is probably the best and simplest thing we can do, i.e. for the sake of a smooth development experience. Dapp authors simply want to configure accounts and then use them without having to worry about wallet management during development.

This is another point of confusion, because when the developer adds wallet accounts to the config (ie via mnemonic), these accounts are added to embark's wallet, they appear in DApp calls to web3.eth.getAccounts, but they cannot be used to sign txs from the DApp (while they can be used to sign Embark txs, ie contract deployment and console commands). An update for this can be added in another PR if needed and is outside the scope of this PR, but one of my concerns is that if we blur the line between node accounts and wallet accounts, and there are some fundamental differences between the two, it will cause more issues and confusion down the line. The developer will need to develop with the idea of using Metamask at some point, so why add this in when we already have a developer node account at our disposal?

but they cannot be used to sign txs from the DApp (while they can be used to sign Embark txs, ie contract deployment and console commands).

@emizzle can you give an example of a web3 call that I could run in the browser console of embark_demo (augmented with a wallet account via config) that would work for a true unlocked node account but not for a config-wallet account?

Before I get to your last question, I wanted to follow up and confirm that you are correct in that<EMAIL_ADDRESS>does always calls eth_accounts. I confirmed this by:

new folder with no parent node_modules
In term 1, geth --wsapi=eth,web3,net,debug,pubsub,personal --ws --wsorigins=* --datadir=./geth/datadir
In term 2:node
const Web3 = require('web3');
const web3 = new Web3(new Web3.providers.WebsocketProvider("ws://localhost:8546"));
web3.eth.getAccounts().then(console.log)
# [] // no default accounts

// create a node account
web3.eth.personal.newAccount("eric");
web3.eth.getAccounts().then(console.log);
# ['0x517d83a71e3981E148a0eEbffF291264758533a6']

// create a wallet account
web3.eth.accounts.create().then(console.log)
# { address: '0xf29C7AC5fce635997654930fcc4F1B5fFa971482',
privateKey:
'0x2cded5a23c0737d94c7b3d3cab87b672895e92628f2285e2b02780a15b29e4c2',
signTransaction: [Function: signTransaction],
sign: [Function: sign],
encrypt: [Function: encrypt],
index: 0 }

// add wallet account to wallet
web3.eth.accounts.wallet.add("0x2cded5a23c0737d94c7b3d3cab87b672895e92628f2285e2b02780a15b29e4c2")

// check which accounts are returned to us
web3.eth.getAccounts().then(console.log)
# [ '0x517d83a71e3981E148a0eEbffF291264758533a6' ] // this is the original node account, and **NOT** the wallet accounts

Using the v5 config updates branch as a base, I created a new demo. Then, I added the following to config/blockchain.js:
development: {
    clientConfig: {
      miningMode: 'dev' // Mode in which the node mines. Options: dev, auto, always, off
    },
    accounts: [
      {
        nodeAccounts: true,
        password: "config/development/password"
      },
      {
        mnemonic: "carpet tank that sugar clerk cross youth night reform current high rifle",
        hdpath: "m/44'/60'/0'/0/",
        numAddresses: "1"
      }
    ]
  },

After that, I ran embark run.
In the browser, I ran
await web3.eth.getAccounts()
# ["0x4d6C8b5662154C72Aa36cd456a356CF5b728A056", "0x7ebba0510f478c739a7d20A93D2fdB9333aA78A0"]

You can see the first account is the unlocked --dev node account (specified in the blockchain config as nodeAccounts:true. The second account is the wallet account (specified by the mnemonic in the config).
Running
await web3.eth.sendTransaction({ from: "0x4d6C8b5662154C72Aa36cd456a356CF5b728A056", value:0, to:"0x4d6C8b5662154C72Aa36cd456a356CF5b728A056" })
# {blockHash: "0x05202672eb9cb044737e3be6bd67fd0ffa484f2af24627794f84b1e8c74c754b", blockNumber: 16, contractAddress: null, cumulativeGasUsed: 21000, from: "0x4d6c8b5662154c72aa36cd456a356cf5b728a056", …}

works because the from account is the unlocked --dev node account.
Running
await web3.eth.sendTransaction({ from: "0x7ebba0510f478c739a7d20A93D2fdB9333aA78A0", value:0, to:"0x7ebba0510f478c739a7d20A93D2fdB9333aA78A0" })
# Uncaught (in promise) Error: Returned error: unknown account Error: Returned error: unknown account
    at Object.ErrorResponse (errors.js:29)
    at Object.<anonymous> (index.js:140)
    at index.js:121
    at Array.forEach (<anonymous>)
    at WebSocket.WebsocketProvider.connection.onmessage (index.js:98)
ErrorResponse @ errors.js:29
(anonymous) @ index.js:140
(anonymous) @ index.js:121
WebsocketProvider.connection.onmessage @ index.js:98
setTimeout (async)
_fireError @ index.js:71
sendTxCallback @ index.js:491
(anonymous) @ index.js:140
(anonymous) @ index.js:121
WebsocketProvider.connection.onmessage @ index.js:98
errors.js:29 Uncaught (in promise) Error: Returned error: unknown account
    at Object.ErrorResponse (errors.js:29)
    at Object.<anonymous> (index.js:140)
    at index.js:121
    at Array.forEach (<anonymous>)
    at WebSocket.WebsocketProvider.connection.onmessage (index.js:98)

fails because the from account exists only in the wallet in Embark and not in the browser's web3 wallet. Therefore this account is not unlocked.

Under the hood, the browser is sending eth_sendTransaction RPC call, which assumes the tx has been signed by the from account using its PK. We can't do that in the browser in this case because the PK of the account is in Embark, not in the browser. This can be "fixed" by intercepting eth_sendTransaction RPC calls in Embark's proxy, and if the tx is from a wallet account that Embark controls, sign the tx using the accounts PK, and forward the tx on to the network using eth_sendRawTransaction. This is essentially what Metamask does.

Under the hood, the browser is sending eth_sendTransaction RPC call, which assumes the tx has been signed by the from account using its PK. We can't do that in the browser in this case because the PK of the account is in Embark, not in the browser. This can be "fixed" by intercepting eth_sendTransaction RPC calls in Embark's proxy, and if the tx is from a wallet account that Embark controls, sign the tx using the accounts PK, and forward the tx on to the network using eth_sendRawTransaction. This is essentially what Metamask does.

Thanks @emizzle, that makes a lot of sense.
It seems we are intercepting eth_sendRawTransaction: https://github.com/embark-framework/embark/blob/master/packages/embark-blockchain-connector/src/provider.js#L140-L142.
But I guess we need to generalize that for all RPC calls that require signing, i.e. if there are others besides eth_sendTransaction and eth_sendRawTransaction.


Migrate to the JsonSchemaElement API

Migrate to the JsonSchemaElement API.
Closes #1054

I'm pretty sure I got everything! Let's make sure CI passes

Looks like I need to fix
https://github.com/quarkiverse/quarkus-langchain4j/blob/2103f0748aed983262a41dd08d1ea6c6e0aa06c3/model-providers/watsonx/runtime/src/main/java/io/quarkiverse/langchain4j/watsonx/bean/TextChatMessage.java#L225-L232

Ops, I'm starting to work on the same topic. You can see my changes in my repo.

I'm sorry about crossing each other. There was already an issue I had created a few weeks ago (#1054) that I had assigned to myself.
I started working on this a few weeks ago but then ran into a wall and needed to wait for a LangChain4j upgrade.

Yeah, I noticed it after I started. I wasn't paying attention 😄 .

From what I can tell (only looking at this on my phone) we came up with similar solutions :)
Good sanity check!

Very nice, thanks @edeandrea! I added a small comment on this

Once this is in, I'll release a 0.22.0.CR1


NSwag generates incorrect serialization for date parameters

According to the Swagger 2.0 spec, "type": "string", "format": "date" should be formatted according to full-date here. (E.g. "2018-01-01") However, even when a request parameter is specified this way, NSwag serializes the date with a timestamp appended, like "2018-01-03T00:00:00".
Here's a minimal Swagger specification that exhibits this behavior:
{
    "swagger": "2.0",
    "info": {
        "version": "<IP_ADDRESS>",
        "title": "API"
    },
    "paths": {
        "/api/endpoint": {
            "post": {
                "consumes": [ "application/json" ],
                "parameters": [
                    { "name": "date",
                      "in": "body",
                      "schema": {
                          "type": "string",
                          "format": "date"
                      }
                    }
                ],
                "responses": {
                    "200": {
                    }
                }
            }
        }
    }
}

After generating a client from this specification using NSwag -> C# and running it against a local server, the parameter has a timestamp attached to the date.
NSwag version: 11.15.3

Good point, we should probably generate a custom json converter and add it to the date properties.
https://stackoverflow.com/a/18639829/876814
Do you know if there is a way to correctly serialize these dates without having to generate the custom json converter?

Do you know if there is a way to correctly serialize these dates without having to generate the custom json converter?

Not that I know of. With the built-in date conversion base class in Newtonsoft, it doesn't seem like that much extra generated code to add though.
I remember having issues dealing with nested DateTimes when trying to customize Newtonsoft's JSON serialization in the past though; let me see if I can repro that, cuz getting all usages of a date formatted string in Swagger might be more complicated than at first glance.

Yup, if the DateTimes are inside another data structure, things get ugly. Here's some example C# where the dates are in a List; compiles fine, errors at runtime:
using System;
using System.Collections.Generic;

using Newtonsoft.Json;
using Newtonsoft.Json.Converters;

namespace Program {
  public class CustomDateTimeConverter : IsoDateTimeConverter {
    public CustomDateTimeConverter() {
      base.DateTimeFormat = "yyyy-MM-dd";
    }
  }

  public class JSON1 {
    [JsonConverter( typeof(CustomDateTimeConverter) )]
    [JsonProperty( "date" )]
    public DateTime Date { get; set; }
  }

  public class JSON2 {
    [JsonConverter( typeof(CustomDateTimeConverter) )]
    [JsonProperty( "dates" )]
    public List<DateTime> Dates { get; set; }
  }

  public class Program {
    public static void Main(String[] args) {
      var json1 = new JSON1() { Date = new DateTime(2018, 01, 03) };
      var json2 = new JSON2() {
        Dates = new List<DateTime> {
          new DateTime(2018, 01, 03),
          new DateTime(2018, 01, 17),
          new DateTime(2018, 01, 31)
        }
      };

      Console.WriteLine(JsonConvert.SerializeObject(json1));
      Console.WriteLine(JsonConvert.SerializeObject(json2));
    }
  }
}

So the nice, simple converter can't be used, say, when the Swagger spec contains an array of date-formatted strings, and another, more complicated converter would be needed for that case. Ugh.
There is this example of serializing a date by specifying the DateFormatString directly in the serializer settings. Unfortunately, that would have collision problems if a Swagger spec contained both dates without timestamps and dates with timestamps...

Yes, it should be a property "local" converter and not a global converter. Maybe we can implement a whole new converter which is not based IsoDateTimeConverter and can handle date, date array and date dictionary properties?

Hmm... but that wouldn't work if there were multiple nested collections; for example, maybe someone wants to represent date ranges, and uses an array of two-element subarrays, each containing a date. In the general recursive case, the level of possible nesting is infinite, and we'd need an infinite amount of generated converters... obviously, in any given specification, there's only going to be a finite amount of expressed nestings, but still. Having to recursively generate the code in each converter feels... wrong.
Grr, dammit Newtonsoft. There should really just be a "here's a converter for a given type; within this class member, recursively serialize any instances of this type using this converter" property.

Yes, but we could implement a recursive converter? :-) Or we just support the most common scenarios...

There's the same problem with dateTime.
OpenAPI dateTime should be formatted according to RFC 3339 date-time which, if you look closely, makes the timezone mandatory.
Sadly, the generated DTOs and controller classes accept values without timezone just fine.
I have tried suggesting making it easier for every ASP.NET based framework by implementing this on Newtonsoft.JSON but no success for now. The lib author indeed suggests writing a date time converter (I've asked whether a PR would be considered).

I think we can address both here...

@RicoSuter: I would consider this a major bug, not an enhancement, as it violates the OpenAPI spec and there is no workaround which doesn't break date-time parameters.
I guess there should also be an additional config paramter parameterDateFormat  besides parameterDateTimeFormat to allow "customization" which is compliant to OpenAPI:
 // for "date"
DateTime.Now.ToString("yyyy-MM-dd");

// for "date-time"
DateTime.Now.ToString("o");

This should also be possible in JSON-Serialization, but that's another issue I guess.

We have the same problem. (See   #2084  )
An additional parameter "parameterDateFormat" would indeed solve the problem.