2 HP cascaded / 2 WP in Kaskade

Before submitting a new issue

[X] I'm not using the older Vaillant sensoAPP (which is unsupported by this integration)
[X] I redownloaded the latest release and restarted Home Assistant
[X] I've enabled debug logs and attached the relevant logs to this ticket

Problem description
Dear GitHub Community,
I am using the MyVaillant integration for Home Assistant, and I have two heat pumps at home that are cascaded (Systemschema 8). The issue I'm facing is that I can only retrieve data for one of the heat pumps in the integration, specifically the energy data. I cannot access the data for the second heat pump, and the second heat pump is not displayed in the integration.
Has anyone of you had similar experiences or found a solution to this problem? I would appreciate your help or advice on how to successfully integrate both heat pumps into Home Assistant.
Thank you in advance for your support!

Hat jemand von euch ähnliche Erfahrungen gemacht oder eine Lösung für dieses Problem? Ich würde mich über eure Hilfe oder Ratschläge freuen, wie ich beide Wärmepumpen erfolgreich in Home Assistant integrieren kann.
Vielen Dank im Voraus für eure Unterstützung!
Beste Grüße,

I have the same issue. A solution would be really nice!

I tried to fix #15 with #64 today and I think it shouldn't be too hard to differentiate between the two. Are you still interested in a fix and are willing to test it?

Good Morning,
yes I'm still interested. I would be great.

I am still interested and willing to test it.

A sensors for the energy data and the heating energy efficiency of the overall system would be sufficient.

Sensors for the energy data and the heating energy efficiency of the overall system would be sufficient.

I second that.

How does it look with https://github.com/ml1nk/mypyllant-component.git? There will surely some changes (if it's working at all) until this is merged, see #64, but feedback is welcome.
Please note, that you need to remove the existing integration before testing.

I unistalled the component from ha and reinstalled it over hacs, but for me, there are no changes. did i make something wrong?

@ml1nk I can also create a beta release from your PR, let me know when you think it's in a testable state.

@andyx87 Have you added my repository and installed it from there? You should at least see the names changed, even if it isn't working for you.
@signalkraft If it isn't too much work, that would be nice.

https://github.com/signalkraft/mypyllant-component/releases/tag/v0.7.0b0
Here's a pre-release with the changes that @ml1nk contributed! Feedback would be appreciated, but make sure to read the warning in the release notes.

That’s with 0.7

Thanks for testing. As it splits your aroTHERM and the hydraulic station the changes are somewhat working and it now shows all devices separately.
There are some possibilities i can think of why it isn't working as expected:

The device_uuid isn't unique in this case and it overwrites the device/entities like before
There is only one device with all the data of both heat pump

does the data looks like it is the sum of both or clearly one of them?

The data isn't there at all (most probably)

If you provide some test data (https://github.com/signalkraft/myPyllant?tab=readme-ov-file#contributing-test-data) i will try to fix it.
@signalkraft Do you have more ideas?

The aerotherm device shows only the energy data of one heat pump. It is not the sum of both.

@ml1nk test data would probably help, I agree. Maybe the mapping of device data to entities is breaking with multiple systems.

Nether the library nor the component looks suspicious to me. The only thing left without test data is to manipulate my own by guessing how it could look from vaillants side, but i would rather use yours (@andyx87) :)

Can I generate test data with HA OS?

Not that it is know how, but you can try to look up "/config/logs" and see if you see some errors from mypyllant when syncing.

Vielleicht hilft das?

To help with this I added a new service generate_test_data in this release: https://github.com/signalkraft/mypyllant-component/releases/tag/v0.7.0b2
After install you can go to https://my.home-assistant.io/redirect/developer_services/ and select mypyllant: generate_test_data. Please post the output here. It's already anonymized, but please double check that no sensitive data is in there.

When i controlled my testdata i would expect, that the missing devices are somewhere in "secondary_heat_generators" which is an empty array in my system, which wouldn't be checked if i read models.py#L342-L350 correctly:
    @property
    def raw_devices(self) -> Iterator[tuple[str, dict]]:
        for key, device in self.current_system.items():
            if isinstance(device, dict) and "device_uuid" in device:
                dtc = self.diagnostic_trouble_codes_by_serial_number(
                    device["device_serial_number"]
                )
                device["diagnostic_trouble_codes"] = dtc
                yield key, device

There probably more auxiliary heat generators somewhere in the testdata of @andyx87 inside "secondary_heat_generators", so I''m excited how it looks like.

homes:

homeName: d2daff4a69005f9639a0531648352f9a4f7fd57d
serialNumber: 7181f45161f0efcf33aeafaf1c3c54f81ec4f450
systemId: 7571042b8af21b829bda040425f77a76706b03d7
productMetadata:
productType: VR921
productionYear: "23"
productionWeek: "19"
articleNumber: "0020260962"
state: CLAIMED
migrationState: FINISHED
firmware:
version: 0357.40.32
updateEnabled: true
updateRequired: false
nomenclature: VR 921
cag: false
countryCode: DE
productInformation: VR921
firmwareVersion: 0357.40.32
7571042b8af21b829bda040425f77a76706b03d7:
control_identifier:
controlIdentifier: tli
time_zone:
timeZone: Europe/Berlin
diagnostic_trouble_codes:

serialNumber: 7181f45161f0efcf33aeafaf1c3c54f81ec4f450
articleNumber: "0020260962"
codes: []
serialNumber: 6393da755ab2c3dc3bbae88ae6043a4ff06f9864
articleNumber: "0010021118"
codes: []
serialNumber: cac1a4f1cee23448eef68cc847022ca801a4a3b3
articleNumber: "0020260914"
codes: []
serialNumber: e4ae823e5c0721e8ae0abb6477fc0730bc522c73
articleNumber: "0020184846"
codes: []
serialNumber: 6393da755ab2c3dc3bbae88ae6043a4ff06f9864
articleNumber: "0010021118"
codes: []
serialNumber: b8ed8afd17ba811e9a6f6c90c628013685d91b4d
articleNumber: "0010023609"
codes: []
connection_status:
connected: true
system:
state:
system:
outdoorTemperature: 7.6914062
systemFlowTemperature: 37.625
systemWaterPressure: 1.5
outdoorTemperatureAverage24h: 9.2109375
zones:

index: 0
desiredRoomTemperatureSetpointHeating: 21.5
desiredRoomTemperatureSetpoint: 21.5
currentRoomTemperature: 15.975
currentRoomHumidity: 60
currentSpecialFunction: NONE
heatingState: HEATING_UP
circuits:
index: 0
circuitState: HEATING
currentCircuitFlowTemperature: 36.8125
heatingCircuitFlowSetpoint: 37.098827
calculatedEnergyManagerState: HEATING_ACTIVE
dhw:
index: 255
currentSpecialFunction: REGULAR
currentDhwTemperature: 46.5
properties:
system:
controllerType: VRC720
systemScheme: 8
backupHeaterType: CONDENSING
backupHeaterAllowedFor: DHW_AND_HEATING
moduleConfigurationVR71: 3
energyProvidePowerCutBehavior: DISABLE_HEATPUMP_AND_BACKUP_HEATER
zones:
index: 0
isActive: true
zoneBinding: CENTRAL_CONTROL
isCoolingAllowed: false
associatedCircuitIndex: 0
circuits:
index: 0
mixerCircuitTypeExternal: HEATING
heatingCircuitType: MIXER_CIRCUIT_EXTERNAL
isCoolingAllowed: false
index: 1
mixerCircuitTypeExternal: FIX_VALUE
heatingCircuitType: DIRECT_HEATING_CIRCUIT
isCoolingAllowed: false
index: 2
mixerCircuitTypeExternal: FIX_VALUE
heatingCircuitType: DIRECT_HEATING_CIRCUIT
isCoolingAllowed: false
dhw:
index: 255
minSetpoint: 35
maxSetpoint: 70
configuration:
system:
continuousHeatingStartSetpoint: -26
alternativePoint: -21
heatingCircuitBivalencePoint: -5
dhwBivalencePoint: -7
automaticCoolingOnOff: false
adaptiveHeatingCurve: false
dhwMaximumLoadingTime: 90
dhwHysteresis: 5
dhwFlowSetpointOffset: 2
continuousHeatingRoomSetpoint: 20
hybridControlStrategy: BIVALENCE_POINT
maxFlowSetpointHpError: 20
dhwMaximumTemperature: 80
maximumPreheatingTime: 0
paralellTankLoadingAllowed: true
zones:
index: 0
general:
name: Zuhause
heating:
operationModeHeating: TIME_CONTROLLED
setBackTemperature: 17
manualModeSetpointHeating: 21
timeProgramHeating:
metaInfo:
minSlotsPerDay: 0
maxSlotsPerDay: 12
setpointRequiredPerSlot: true
monday:
- startTime: 0
endTime: 1440
setpoint: 21.5
tuesday:
- startTime: 0
endTime: 1440
setpoint: 21.5
wednesday:
- startTime: 0
endTime: 1440
setpoint: 21.5
thursday:
- startTime: 0
endTime: 1440
setpoint: 21.5
friday:
- startTime: 0
endTime: 1440
setpoint: 21.5
saturday:
- startTime: 0
endTime: 1440
setpoint: 21.5
sunday:
- startTime: 0
endTime: 1440
setpoint: 21.5
circuits:
index: 0
heatingCurve: 0.75000006
heatingFlowTemperatureMinimumSetpoint: 20
heatingFlowTemperatureMaximumSetpoint: 55
heatDemandLimitedByOutsideTemperature: 20
heatingCircuitFlowSetpointExcessOffset: 0
coolingFlowTemperatureMinimumSetpoint: 20
setBackModeEnabled: false
roomTemperatureControlMode: NON
dhw:
index: 255
operationModeDhw: TIME_CONTROLLED
tappingSetpoint: 50
timeProgramDhw:
metaInfo:
minSlotsPerDay: 0
maxSlotsPerDay: 3
setpointRequiredPerSlot: false
monday:
- startTime: 120
endTime: 360
- startTime: 660
endTime: 960
tuesday:
- startTime: 120
endTime: 360
- startTime: 660
endTime: 960
wednesday:
- startTime: 120
endTime: 360
- startTime: 660
endTime: 960
thursday:
- startTime: 120
endTime: 360
- startTime: 660
endTime: 960
friday:
- startTime: 120
endTime: 360
- startTime: 660
endTime: 960
saturday:
- startTime: 120
endTime: 360
- startTime: 660
endTime: 960
sunday:
- startTime: 120
endTime: 360
- startTime: 660
endTime: 960
timeProgramCirculationPump:
metaInfo:
minSlotsPerDay: 0
maxSlotsPerDay: 3
setpointRequiredPerSlot: false
monday:
- startTime: 270
endTime: 480
- startTime: 600
endTime: 900
- startTime: 1080
endTime: 1260
tuesday:
- startTime: 270
endTime: 480
- startTime: 600
endTime: 900
- startTime: 1080
endTime: 1260
wednesday:
- startTime: 270
endTime: 480
- startTime: 600
endTime: 900
- startTime: 1080
endTime: 1260
thursday:
- startTime: 270
endTime: 480
- startTime: 600
endTime: 900
- startTime: 1080
endTime: 1260
friday:
- startTime: 270
endTime: 480
- startTime: 600
endTime: 900
- startTime: 1080
endTime: 1260
saturday:
- startTime: 270
endTime: 480
- startTime: 600
endTime: 900
- startTime: 1080
endTime: 1260
sunday:
- startTime: 270
endTime: 480
- startTime: 600
endTime: 900
- startTime: 1080
endTime: 1260
current_system:
system_type: HEATPUMP_CASCADE
has_emf_capable_devices: true
primary_heat_generator:
device_uuid: 4f334b0a6fb2b942f84ad28334cfd68b880a8a72
ebus_id: HMU03
spn: 351
bus_coupler_address: 0
article_number: "0010021118"
emfValid: true
device_serial_number: 2633e35b2e3ce9aa37d1d853fa61612efd5f3805
device_type: HEATPUMP
first_data: "2023-10-11T03:46:18Z"
last_data: "2023-12-30T04:16:27Z"
data:
operation_mode: DOMESTIC_HOT_WATER
value_type: CONSUMED_ELECTRICAL_ENERGY
calculated: false
from: "2023-10-11T03:46:19Z"
to: "2023-12-30T04:16:23Z"
operation_mode: HEATING
value_type: CONSUMED_ELECTRICAL_ENERGY
calculated: false
from: "2023-10-11T03:46:18Z"
to: "2023-12-30T04:16:23Z"
operation_mode: DOMESTIC_HOT_WATER
value_type: EARNED_ENVIRONMENT_ENERGY
calculated: false
from: "2023-10-11T03:46:20Z"
to: "2023-12-30T04:16:26Z"
operation_mode: HEATING
value_type: EARNED_ENVIRONMENT_ENERGY
calculated: false
from: "2023-10-11T03:46:19Z"
to: "2023-12-30T04:16:26Z"
operation_mode: HEATING
value_type: HEAT_GENERATED
calculated: true
from: "2023-10-11T03:46:19Z"
to: "2023-12-30T04:16:23Z"
operation_mode: DOMESTIC_HOT_WATER
value_type: HEAT_GENERATED
calculated: true
from: "2023-10-11T03:46:20Z"
to: "2023-12-30T04:16:23Z"
product_name: aroTHERM plus
secondary_heat_generators:
device_uuid: bc84fcd6f80d327bcc66f69a2a78a921eeca0959
ebus_id: HMU03
spn: 351
bus_coupler_address: 24
article_number: "0010021118"
emfValid: true
device_serial_number: 6393da755ab2c3dc3bbae88ae6043a4ff06f9864
device_type: HEATPUMP
first_data: "2023-10-11T03:46:25Z"
last_data: "2023-12-30T04:16:38Z"
data:

operation_mode: DOMESTIC_HOT_WATER
value_type: CONSUMED_ELECTRICAL_ENERGY
calculated: false
from: "2023-10-11T03:46:25Z"
to: "2023-12-30T04:16:34Z"
operation_mode: HEATING
value_type: CONSUMED_ELECTRICAL_ENERGY
calculated: false
from: "2023-10-11T03:46:25Z"
to: "2023-12-30T04:16:33Z"
operation_mode: DOMESTIC_HOT_WATER
value_type: EARNED_ENVIRONMENT_ENERGY
calculated: false
from: "2023-10-11T03:46:26Z"
to: "2023-12-30T04:16:36Z"
operation_mode: HEATING
value_type: EARNED_ENVIRONMENT_ENERGY
calculated: false
from: "2023-10-11T03:46:26Z"
to: "2023-12-30T04:16:35Z"
operation_mode: DOMESTIC_HOT_WATER
value_type: HEAT_GENERATED
calculated: false
from: "2023-10-11T03:46:27Z"
to: "2023-12-30T04:16:37Z"
operation_mode: HEATING
value_type: HEAT_GENERATED
calculated: false
from: "2023-10-11T03:46:27Z"
to: "2023-12-30T04:16:37Z"
product_name: aroTHERM plus
electric_backup_heater:
device_uuid: 09594d943eb7923ed1f5b9d856856ce518ac924a
ebus_id: VWZ02
spn: 351
bus_coupler_address: 0
article_number: "0010023609"
emfValid: true
device_serial_number: b8ed8afd17ba811e9a6f6c90c628013685d91b4d
device_type: ELECTRIC_AUXILIARY_HEATER
first_data: "2023-10-11T03:46:22Z"
last_data: "2023-12-30T04:16:30Z"
data:
operation_mode: DOMESTIC_HOT_WATER
value_type: CONSUMED_ELECTRICAL_ENERGY
calculated: false
from: "2023-10-11T03:46:23Z"
to: "2023-12-30T04:16:30Z"
operation_mode: HEATING
value_type: CONSUMED_ELECTRICAL_ENERGY
calculated: false
from: "2023-10-11T03:46:22Z"
to: "2023-12-30T04:16:29Z"
operation_mode: DOMESTIC_HOT_WATER
value_type: HEAT_GENERATED
calculated: false
from: "2023-10-11T03:46:23Z"
to: "2023-12-30T04:16:30Z"
operation_mode: HEATING
value_type: HEAT_GENERATED
calculated: false
from: "2023-10-11T03:46:23Z"
to: "2023-12-30T04:16:30Z"
product_name: hydraulic station
solar_station: null
ventilation: null
gateway: null
mpc:
devices: []
device_buckets:
startDate: "2023-12-29T00:00:00Z"
endDate: "2023-12-30T00:00:00Z"
resolution: HOUR
operationMode: DOMESTIC_HOT_WATER
energyType: CONSUMED_ELECTRICAL_ENERGY
totalConsumption: 5000
data:

startDate: "2023-12-29T00:00:00Z"
endDate: "2023-12-29T01:00:00Z"
value:<PHONE_NUMBER>444445
startDate: "2023-12-29T01:00:00Z"
endDate: "2023-12-29T02:00:00Z"
value: 1000
startDate: "2023-12-29T02:00:00Z"
endDate: "2023-12-29T03:00:00Z"
value:<PHONE_NUMBER>5555554
startDate: "2023-12-29T03:00:00Z"
endDate: "2023-12-29T04:00:00Z"
value: 0
startDate: "2023-12-29T04:00:00Z"
endDate: "2023-12-29T05:00:00Z"
value: 0
startDate: "2023-12-29T05:00:00Z"
endDate: "2023-12-29T06:00:00Z"
value: 0
startDate: "2023-12-29T06:00:00Z"
endDate: "2023-12-29T07:00:00Z"
value: 0
startDate: "2023-12-29T07:00:00Z"
endDate: "2023-12-29T08:00:00Z"
value: 0
startDate: "2023-12-29T08:00:00Z"
endDate: "2023-12-29T09:00:00Z"
value: 0
startDate: "2023-12-29T09:00:00Z"
endDate: "2023-12-29T10:00:00Z"
value:<PHONE_NUMBER>444445
startDate: "2023-12-29T10:00:00Z"
endDate: "2023-12-29T11:00:00Z"
value: 1726.9444444444443
startDate: "2023-12-29T11:00:00Z"
endDate: "2023-12-29T12:00:00Z"
value:<PHONE_NUMBER>111111
startDate: "2023-12-29T12:00:00Z"
endDate: "2023-12-29T13:00:00Z"
value: 0
startDate: "2023-12-29T13:00:00Z"
endDate: "2023-12-29T14:00:00Z"
value: 0
startDate: "2023-12-29T14:00:00Z"
endDate: "2023-12-29T15:00:00Z"
value: 0
startDate: "2023-12-29T15:00:00Z"
endDate: "2023-12-29T16:00:00Z"
value: 0
startDate: "2023-12-29T16:00:00Z"
endDate: "2023-12-29T17:00:00Z"
value: 0
startDate: "2023-12-29T17:00:00Z"
endDate: "2023-12-29T18:00:00Z"
value: 0
startDate: "2023-12-29T18:00:00Z"
endDate: "2023-12-29T19:00:00Z"
value: 0
startDate: "2023-12-29T19:00:00Z"
endDate: "2023-12-29T20:00:00Z"
value: 0
startDate: "2023-12-29T20:00:00Z"
endDate: "2023-12-29T21:00:00Z"
value: 0
startDate: "2023-12-29T21:00:00Z"
endDate: "2023-12-29T22:00:00Z"
value: 0
startDate: "2023-12-29T22:00:00Z"
endDate: "2023-12-29T23:00:00Z"
value: 0
startDate: "2023-12-29T23:00:00Z"
endDate: "2023-12-30T00:00:00Z"
value: 0

Another feature that would be pretty good:
If you could switch the fixed heating circuits (in my case index 1 and index 2) on and off. I would submit an explanation later this afternoon. I'm at work first.

Zwangseinschaltung: Das System stellt in jedem Fall eine Heizanforderung und erhitzt den Pufferspeicher auf den Wert der in dem virtuellen fixen Heizkreislauf eingestellt ist, plus den Offset Wert.
Der Vorteil, den virtuellen Festwertkreislauf ein und auszuschalten besteht darin, dass man bei diesem Kreislauf eine AT Temperatur einstellen kann.
————————————————-
The SG Ready / PV Ready interface on the heat pump control gives you the following options for heating requirements:
Switch-on recommendation: The buffer tank is heated to the setpoint required by the heating system to achieve the desired room temperature.
Forced switch-on: The system always makes a heating request and heats the buffer tank to the value set in the virtual fixed heating circuit, plus the offset value.
The advantage of switching the virtual fixed value circuit on and off is that you can set an AT temperature for this circuit. This means that the heating would not make a request if the outside temperature is above the set value. This would make it possible to increase the storage temperatures when there is a solar surplus or a cheap electricity price (I use Tibber) without creating unnecessary heating requirements.

It seems my assumption could be correct, that
    @property
    def raw_devices(self) -> Iterator[tuple[str, dict]]:
        for key, device in self.current_system.items(): # merged with self.current_system.secondary_heat_generator.items()
            if isinstance(device, dict) and "device_uuid" in device:
                dtc = self.diagnostic_trouble_codes_by_serial_number(
                    device["device_serial_number"]
                )
                device["diagnostic_trouble_codes"] = dtc
                yield key, device

is ignoring "secondary_heat_generator" as it isn't a dict and just need to also loop through it like it does with self.current_system directly.
@signalkraft Are you already fixing this and adding the new test data or shall i do it?
@andyx87 Seems interesting, but please create another issue for this as I hope to get this issue closed after merging.

@ml1nk I've added test data, but I haven't encountered this issue. Feel free to update it!

Thanks to @ml1nk multiple heatpumps should work in the latest pre-release. Let us know, if it works for you @andyx87!

Yes!
Its working. Thank you so much :-)

I tested the update on my system as well, everything seems fine so I'm closing this. Again, many thanks @ml1nk for your contribution!

@rmalbrecht Have you deleted the integration and added it again? What devices are duplicated?


Minor typo in output

In the output during a mysql audit:
[1/1] Module: mysql. 7 rule(s).  4 rule(s) succeded. [VULNERABLE]

"Succeeded" is missing an e.
I'd fix it myself, but I figure you are rooting around in there already so it will be easier for you to fix it while resolving some other issues. If you don't get to it before release I will deal with it.

Fixed.


setup script to cache artificial interim posterior ellipticities

We can start with NUTS with low MD and later iterate and optimize. The advatange is that we don't need to tune to get high efficiency and leaves us the option of using GPUs later if want to

Done in script get_toy_ellip_samples.py


ci: use google-cloud-cpp-common from vcpkg

Now that vcpkg has a good version of google-cloud-cpp-common, we do
not need to compile our own version.

This change is

This won't work. As the builds stand today we need the testing libraries and those are not installed by vcpkg (nor should they).


Update to use react-dom

For those on React 0.14 you get a deprecation warning about React.findDOMNode going away. This future proofs that.

+1

+1

+1

+1

+1


wait 10 sec before checking the log file.

additional fix for #8100

@amaltaro, Alan please review.

So, this condor logs are checked by the JobStatusLite component, which has a polling cycle of 60secs. AFAIK we don't want to raise that cycle because it's - sometimes - already too large.
Let me try to come up with another possible fix, this sleep is evil and it will eventually hurt us. Moreover, we are not sure this 10secs of grace time is going to be enough.

Yes, you are right. I could delay whole things although it only delays when FWJR are not found.
We could put the sleep outside the for loop (hoping) all the log files were transferred. Or we can only wait if we couldn't find the "Reason" string from the log, hoping there is not many jobs failed and actually didn't produce fwjr.

as Alan pointed out this could give more harm than benefit. Closing it. We could wait a little in the upper level outside job loop but we will always have a race condition


No Dev Yes wsa :)

I no dev channel but using wsa :)

You don't have to be in a Insider release to run this, when it talks about Developer Mode it's talking about the ability to promote and register appx files. Normally, without dev mode enabled you cannot register your own appx files and must get them from the store


Normalized event difference with ShadowDOM and Shady

Hello,
i've an element which contains a paper-fab, and a tap event as :
<dom-module id="my-toolbar">
  <template>
     <paper-fab on-tap="myHandler" icon="..."></paper-fab>
     <paper-fab on-tap="myHandler" icon="..."></paper-fab>
     <paper-fab on-tap="myHandler" icon="..."></paper-fab>
  </template>
</dom-module>

This element is instantiate into a tree of other elements, starting from a top element "my-app" instantiated in the html body.
I've one unique handler for 3 paper-fab. So in the handler, i want to identify the clicked paper-fab.
but i found in the normalized event object that :

with Shady

localTarget : is the app top element "my-app"
rootTarget : the "sub element" of paper-fab you click on (for example iron-icon if you click at the center)

with ShadowDOM

localTarget : is the app top element "my-app" too (is it correct ?)
rootTarget : is the paper-fab element you click on (seems to be the correct behavior to me)

Anthony.

The expected result is the following on both Shady & Shadow:

rootTarget: iron-icon (deepest element that was tapped)
localTarget: paper-fab (element that had the event listener)

The discrepancy appears to originate in the gesture system.  Using click results in the expected results (same between shady & shadow)
http://jsbin.com/lohepic/edit?html,console,output

It's strange, i've tested it with a click, and localTarget always seems to be my app top-level element, on both shadow and shady.

Wow, forget my last words. I made a mistake : i was logging Polymer.log(e), and then i was clicking on the localTarget property in chrome console to resolve the "getter" ... and it seems it's not correctly resolved (context problem ?)

@kevinpschaaf
I am not sure if this is related to this issue here. But what is the designed behavior when using an annotated event handler on the Local DOM which contains Light DOM elements. I modified your JSBIN and the only get the
 as event target: http://jsbin.com/qewahodoxa/edit?html,console,output.
I was expecting to get <x-test> at some point.

Right, the object returned from Polymer.dom(e) has getters that are evaluated lazily, and the event target changes once the event loop completes, so you may see different results if you expand the object on the console later.
You should never expect to see x-test as a target unless the event handler was placed on or above x-test.  Since the handler is on elements in the template stamped inside of x-test, localTarget gives you the first element inside the x-test scope (but at or below where the handler was added) as the event bubbles up, while rootTarget should give you the deepest-most element irrespective of scope boundaries (which may or may not be the same thing).
As I said, the bug appears to originate in the gesture system, as it is only the gesture-simulated events that have incorrect targets, not native ones like click.

@kevinpschaaf
I see but in the JSbin I posted,  I placed the on-tap and the on-click handler on the x-test element and still get
 as  localTarget:
<x-test on-click="myHandler">
Click
</x-test>

This is just a holdover from the initial design, where gesture events were non-bubbling. Should be pretty easy to change the initiator from nativeEvent.currentTarget to nativeEvent.target.

Ah, so the real problem is that when the event gets to the gesture recognizer, it has already passed through the shadowdom boundary and lost the original target. I'll have to use event.path in the recognizer to fire from the original source.
@kevinpschaaf we may to discuss the performance impact of this for ShadyDOM.

@kevinpschaaf @azakus This one could still be an issue. Or I might be doing this all wrong. I updated @kevinpschaaf jsbin to pass content as light DOM and I understand that Light DOM is ignored here and I therefore get content are localTarget. But, shouldn't it be paper-fab as localTarget? content doesn't seem to be on any use here.
http://jsbin.com/golome/edit?html,console,output

Looks like this is certainly a bug with ShadowDOM. If I enable ShadowDOM in above sample, it works. The problem is with ShadyDOM.
http://jsbin.com/golome/edit?html,console,output


Unsupport

import markdown

data = markdown.markdown(md, extensions=[
    'markdown.extensions.extra',
    'markdown.extensions.tables',
    'markdown.extensions.codehilite',
])

md:
>Sec-WebSocket-Accept = base64-value-non-empty
>base64-value-non-empty = (1*base64-data [ base64-padding ]) | base64-padding
>base64-data = 4base64-character
>base64-padding = (2base64-character "==") | (3base64-character "=")
>base64-character = ALPHA | DIGIT | "+" | "/"

data:

Sec-WebSocket-Accept = base64-value-non-empty,
base64-value-non-empty = (1*base64-data [ base64-padding ]) | base64-padding
base64-data = 4base64-character
base64-padding = (2base64-character "==") | (3base64-character "=")
base64-character = ALPHA | DIGIT | "+" | "/"

But I want to get
 in end of each line.

Maybe you want the nl2br extension?
Or just add two spaces to the end of lines where you want a
.

Waaaaaa! Thank you very much!


Do not require the SPDX version to be 1.2

@sschuberth @pombredanne
According to 2ecb3654647ed35cf34320ceafe44d8efdaa338d, we do not require the version to be 1.2, but we have a bug. (still require the version to be 1.2)
Steps to reproduce:

Modify SPDXSimpleTag.tag and replace this line -> SPDXVersion: SPDX-1.2 with SPDXVersion: SPDX-1.1.
Run python parse_tv.py '../data/SPDXSimpleTag.tag' from the examples directory.

Output that we receive:
examples git:(master) ✗ python parse_tv.py '../data/SPDXSimpleTag.tag'
Generating LALR tables
SPDXVersion must be SPDX-1.2 found SPDX-1.1.
Errors encountered while parsing

The cause of this behaviour is the given snippet inside tagvaluebuilders.py:
if vers == version.Version(major=1, minor=2):
    doc.version = vers
    return True

Hi @yash-nisar. I'm sorry, but I'm going to pull myself out of this project as I personally have to need for it anymore, and I'm not a real Python guy anyway. As a result, I will probably not respond to issues or pull requests.

No problem :). Thank you for your reviews.
On Mon, Mar 19, 2018 at 6:42 PM, Sebastian Schuberth <
<EMAIL_ADDRESS>wrote:

Hi @yash-nisar https://github.com/yash-nisar. I'm sorry, but I'm going
to pull myself out of this project as I personally have to need for it
anymore, and I'm not a real Python guy anyway. As a result, I will probably
not respond to issues or pull requests.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spdx/tools-python/issues/53#issuecomment-374206559,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOLKgEF13QpP2LjuFAyjnRL1Pq25ZIXjks5tf67GgaJpZM4SwAwS
.

--
http://www.somaiya.edu http://www.somaiya.edu/kjsmc
http://www.nareshwadi.org http://www.somaiya.edu
http://www.helpachild.in http://nareshwadi.org

@pombredanne Your views ?


feat: improve label ui

Signed-off-by: sachdeva-shrey<EMAIL_ADDRESS>Fixes https://github.com/bench-routes/bench-routes/issues/358
Changes

Improved the Add Label UI

Screenshots

@sachdeva-shrey This is great! The labels as of now look like a weird design, so I'm looking forward to the changes.

@aquibbaig Sure, I've added some styles and updated the screenshot in the description. Please let me know if it's better now.

@sachdeva-shrey Can you please rebase the PR with the master branch?

@muskankhedia Sure, would you like me to squash the commits? It doesn't have any conflicts with master.

@muskankhedia Sure, would you like me to squash the commits? It doesn't have any conflicts with master.

You can just merge the master branch, that is just as a safety measure to any conflicts. Also, if you can squash all the commits it will surely be a plus point.

@muskankhedia Sure, I've rebased and squashed the commits.


How can a new service ask zuul to update information (metadata) from eureka?

Hi!
I have eureka server that is running, zuul and several services that already have registered in eureka. But now I want to add a new service. As far as I understand, zuul periodically requires information about resgistered services from eureka. But I want in order a new service asks zuul to update information from eureka at once it registered in eureka. How can I realize it?

Why cant you wait for Zuul to just refresh the client cache?

It should do this automatically. It will take a few minutes to happen.

Yes, it takes a few minutes. But I need speed up this process. How can I do it??

The only way I know of speeding up the process is by adjusting eureka.instance.leaseRenewalIntervalInSeconds.

In documention I found the next:

If you are using @EnableZuulProxy with tha Spring Boot Actuator you will enable (by default) an additional endpoint, available via HTTP as /routes. A GET to this endpoint will return a list of the mapped routes. A POST will force a refresh of the existing routes (e.g. in case there have been changes in the service catalog).
NOTE
the routes should respond automatically to changes in the service catalog, but the POST to /routes is a way to force the change to happen immediately.

@YakovBurtsev so can we close this?

@spencergibb yes, of course


vtworker: Moved handling the "Reset" command from the gRPC server

implementation into the framework itself.
This way, we don't have to duplicate this code for every server
implementation.
@alainjobart @aaijazi

LGTM


Remove WskRestConsoleTest.

This test does not make sense to have as a REST based test, since it doesn't test any implementation other than the test's implementation, which is faulty.
The pollFor uses a different view than the underlying listActivations call, which results in intermittent failures.

Partially addresses https://github.com/apache/incubator-openwhisk/issues/2936


Consider dill instead of pickle?

Due to Pickle's limitation, it's currently impossible to use lambdas or curried/nested functions. Is it possible to consider using dill, which can handle yields, nested functions, and lambdas?
Thank you for your work on picking up this library.

Interesting library, never seen that before. I would be open to that change if it doesn't change the current pickled tasks, because that could cause conflicts in production (or a migration path would be fine too (preferably automated)).

The license for dill is BSD3, which is compatible from a licensing aspect. Which is good.


Fixe typo of example SamlIdpController

I have fixed typo for wiki page of Rails_Integration.
I replaced found_user to person, because the variable of found_user is not difeined in idp_make_saml_response method.

wiki: Rails_Integration page
https://github.com/saml-idp/saml_idp/wiki/Rails_Integration

diff
https://github.com/saml-idp/saml_idp/wiki/Rails_Integration/_compare/60f46e742a086fc2d1015e2cd6bef458f20eaa70...8d51ebaa10f9b9f71d2c0cfc94eb0eb5cf194c91

And, It can change the wiki page via not collaborator users.
Is it expected setting?
If you not allowed not collaborator users to edit page, check bellow settings.

Changing access permissions for wikis - GitHub Doc
https://docs.github.com/en/github/building-a-strong-community/changing-access-permissions-for-wikis

@jphenow you may want to check "permission" of WIKI page.

Thanks for the heads up folks! Disabled public edits, but also thanks for these edits :D


Update dependencies and fix typescript warnings

This PR updates all outdated dependencies to their latest versions, except for typescript, which is still on the highest 4.X.X version - upgrading typescript to 5.X.X results in errors from the hyperapp package.
The updated puppeteer no longer has the sandbox error on linux, so this PR makes #44 obsolete.

I realised that the latest version of puppeteer wouldn't work to run a prerender on netlify. I have fixed this by adding a @sparticuz/chromium dependency, and using puppeteer-core instead of puppeteer, configured to use chromium as the executable.

Woah this is great! Thanks for this, I will be checking it out sometime next week. Are you using hyperstatic in production? I was also planning a new major version bump and review a few things with newer vite features. This project has been needing some love for a while.

Hey @loteoo, yes I'm using it in production but just for an extremely small site.
I'm happy to beta test if you do make a new version.
And I won't be offended if you choose not to merge this 😂. The new dependency is heavy and might not be necessary for most use cases.


Add prefix to storage config

As discussed in #56, the storage.prefix config should be added to work with both storage types directory and s3, meaning that s3.prefix will be deprecated.
Concerns:

Will storage.directory.path config remain the same or should it be removed to make place to storage.prefix as well?

if so, which will the final storage path given the user configuration defines both?

{{ storage.prefix }} / {{ storage.directory.path }} or
{{ storage.directory.path }} / {{ storage.prefix }}

The easiest one is storage.prefix. We could provide a path prefix as a parameter in the constructor


New -> Text File fails

When looking at the Dash screen, you can click the "New" box on the top right, then select "File".
This creates an empty text file, with no data. However Spark Notebook tries to open it as a Notebook which in turn fails.
Possibly disable option to create empty text file ?

agreed let's remove this item for now until some work can be done on it.
In the meantime, I'll update the code to give a name to new created folders...


Unit test runner should not compile each individual .scss file

The regexp responsible for this is in test/unit/index.js

Not really an issue after working with tests for a while now.


Create data_scraper_CLI

scrape/collect users data and store in a texfile for usage

Codecov Report

Merging #489 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #489   +/-   ##
=======================================
  Coverage   41.08%   41.08%
=======================================
  Files          28       28
  Lines        2115     2115
=======================================
  Hits          869      869
  Misses       1246     1246

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 97465a7...621473c. Read the comment docs.


[Question] TypeError: Right-hand side of 'instanceof' is not callable when accessing an embedded formio form from Vue app

Hosting type

[x] Form.io
[x] Local deployment

Formio.js version: 4.14.5
Frontend framework: Vue.js (v3.2.31)
vue-formio version: 4.0.8
vue-router version: 4.0.14
Browser: Chrome
Browser version:  100.0.4896.75

I'm using formio with Vue.js (v3.2.31), trying to embed a formio resource (e.g., form) into the Vue app. When executing 'npm run dev', got below error:

When clicking into index.ts:3:1, it says ' Could not load content for http://localhost:3000/node_modules/vue-formio/src/index.ts (HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE)', see the screenshot below:
.
Steps to Reproduce

Created a sample Vue.js app by executing 'npm init vue@latest'
At https://next.form.io/, created a sample form.io project that contains a form resource.
Installed vue-formio by executing 'npm install --save vue-formio' command
Trying to embed the formio form into the Vue app by follow the instruction on form.io
Execute 'npm run dev'
Access 'http://localhost:3000/' from the browser, gets the above error.

Below is the vue component using the formio resource:
<template>
    <div id="app">
        <formio :src="formUrl" v-on:submit="onSubmitMethod" />
    </div>
</template>
<script>
    import { Formio } from 'vue-formio';

    export default {
        data: function() {
            // Load these from vuex or some other state store system.
            return {
                formUrl: "https://hcydsplculhonzk.form.io/crnacmpform"
            }
        },
        components: {
            formio: Formio
        },
        methods: {
            onSubmitMethod: function(submission) {
                console.log(submission);
            }
        }
    };
</script>

Am I missing any steps?

Figure out what causes the issue. Close the ticket.

@echo-liang how do you fixed the issue?


Option to serve Self-signed while letsencrypt rate limited

If e.g. a domain were to be temporarily rate limited (currently the letsencrypt limit is 20 certs/week) it would be quite neat if kube-lego could be configured to serve the service using a self-signed (staging) cert until it can get a valid cert. Perhaps this is already easily doable?

It seems to already do this to a certain extent, but the root seems to hit the default backend, not my app.

Hi @jmn & @bbhoss.
It seem's that the root is the nginx ingress controllers sending you to the default backend unless the tls secrets exists & is propagated for the domain (e.g ratelimiting when it was requested or other failure to request).
I use the following ingress to set a TLS secret on the default backend.
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: default-ingress
  namespace: default
  annotations:
    kubernetes.io/tls-acme: 'false'
    kubernetes.io/ingress.class: 'nginx'
spec:
  tls:
  - secretName: <your tls secret with default ssl cert>

  backend:
    serviceName: default-http-backend
    servicePort: 80


Make various layout properties enums.

Fixes #740.

:+1: on this.

(You could just merge it :)

I am leaving soon. I did not want to take upon myself to merge without testing.


Support multiple URI's for the Retrofit client factory

For certain scenarios, a Retrofit client might be required, rather than the recommended Feign client, such as when communicating with streaming endpoints. It'd be great to have a version of RetrofitClientFactory.createProxy that supports multiple URI's, similar to the FeignClientFactory.createProxy method.

@ellisjoe

closing this out, since this was addressed in #212


Drop reconciler.Base from eventing.

Dependent on https://github.com/knative/eventing/pull/2943
Proposed Changes

Remove last pointer to reconciler.Base from eventing.

Release Note

NONE

/hold
until the upstream PR lands

There is no dep on knative/eventing#2943
/unhold

@n3wscott mind doing a rebase?
happy to /lgtm afterwards!

/lgtm

/approve

/lgtm


[Documentation Bug]: readme file updation

Describe the bug ✍
updation of readme file
Provide step by step information reproduce the bug 📄
Description of the project
Select program in which you are contributing
GSSoC24
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

Hey @msifran !
Thank you for raising an issue 💗
You can self assign the issue by commenting /assign in comment 😀
Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
Don’t Forget to ⭐ our GameZone🎮
Make sure you join our Discord🕹

/assign

Hey @msifran ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇


ci: Fix github actions

What
Learned about the next exception to the norm in github actions due to the update-assets workflow
Unrecognized named-value: 'openfoodfacts'

When you use expressions in an if conditional, you may omit the expression syntax (${{ }}) because GitHub automatically evaluates the if conditional as an expression. For more information, see "Expressions."

So I have to explicitly have to mark it as string.

Codecov Report

Merging #3097 (3e0e8d0) into develop (de3b7bb) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           develop   #3097   +/-   ##
=======================================
  Coverage     6.46%   6.46%
=======================================
  Files          246     246
  Lines        12197   12197
=======================================
  Hits           789     789
  Misses       11408   11408

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Improve README.md

Add --silent option to ag command preventing error message (eg: permission denied error on some directories) flow to 'output' which should only be the file name selected.

Works for me. Thanks


[RFC][DNM] Experimental TCP

This implementation of the TCP protocol is a proof of concept of the following:
Goal

Verifiable with open source protocol test tools

Implementation

State machine
Separation of TCP control, data and retransmission mechanisms
Whitelisting approach

Status
Minimum viable prototype (MVP) of:

State machine
Retransmission
Data

Instructions on how to run the sanity check.
Sanity Check Suite
Along with it, a TTCN-3 based sanity check suite is being developed:
https://github.com/intel/net-test-suites/blob/master/src/tcp2_check.ttcnpp
Sanity check is able to verify:

Control flow (active, passive connection establishment, termination)
Retransmission
Data

Test Protocol
The test protocol is used to introspect the implementation
from the sanity check.

JSON based
Could be refactored into generic reusable modules on Zephyr’s and TTCN-3 sides
Could be used for other protocols testing

To run the sanity check:

Compile and start the net-test-tools:
https://github.com/intel/net-test-tools
Compile and run samples/net/echo_server for qemu_x86 with the prj.conf
Compile and run the sanity check:
https://github.com/intel/net-test-suites
# ttcn3_start test_suite tcp2_check_3_runs.cfg

NOTE: All modifications should pass the sanity check.

Codecov Report

Merging #11443 into master will decrease coverage by 4%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master   #11443      +/-   ##
==========================================
- Coverage   48.37%   44.37%   -4.01%
==========================================
  Files         265      253      -12
  Lines       42192    39699    -2493
  Branches    10137     9410     -727
==========================================
- Hits        20412    17615    -2797
- Misses      17703    18597     +894
+ Partials     4077     3487     -590

Impacted Files
Coverage Δ

include/net/buf.h
100% <ø> (ø)
:arrow_up:

include/net/net_pkt.h
78.71% <ø> (-7.01%)
:arrow_down:

subsys/net/ip/tcp2.c
0% <0%> (ø)

subsys/net/ip/tcp.c
9.03% <0%> (-48.35%)
:arrow_down:

subsys/net/l2/ethernet/ethernet_mgmt.c
9.24% <0%> (-45.38%)
:arrow_down:

subsys/net/lib/sockets/sockets.c
0% <0%> (-37.96%)
:arrow_down:

subsys/net/ip/connection.c
37.5% <0%> (-37.76%)
:arrow_down:

subsys/net/lib/dns/resolve.c
0.53% <0%> (-34.27%)
:arrow_down:

subsys/net/ip/net_private.h
38.09% <0%> (-32.5%)
:arrow_down:

subsys/net/ip/utils.c
45.92% <0%> (-32.04%)
:arrow_down:

... and 39 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 325954d...98b5885. Read the comment docs.

@ozhuraki, Thanks for posting this!

It's seen (and, indeed, was mentioned by @ozhuraki many times previously) that this is WIP and not something finished, and polish is to be applied at later stage. With that in mind, some suggestions still:

Would be nice to have comments around.
Would be nice to have more descriptive names than "q" or "e".
As far as can tell from quick look (there're no comments, yeah), there's some facility to inject test data, rooted (perhaps) in the function void tp_input(struct net_pkt *pkt), which uses json for that injection. I'd suggest to move all code related to this test injection to a separate file, to keep the TCP implementation itself clear from the test code.

Again, those are just quick notes. If all that's planned, but for a later stage, that's fine. Thanks again for sharing.

@pfalcon

I can add some comments. What in particular isn’t clear enough?
This can be done. But it looks that there’s a problem to have a doubly linked queue entry in net_buf.
Refactoring the test protocol into separate reusable module is planned and will be done to some extent. However, the test protocol requires an access to the TCP data structures. Might it be a good idea to keep the TCP specific parts if they will be relatively small?

Changes:

Added Kconfig entry, experimental TCP is disabled by default.
net_buf: Doubly-linked queue to singly-linked queue for TCP windows.

@ozhuraki:

I can add some comments. What in particular isn’t clear enough?

Well, your discussion with @jhedberg was a good example - there's a new member "q", double-linked list. But what it is for (to be able to assess how it's important to be double-linked). Note that it was among struct members already having comments each, so it should come as natural to add comment to any new member. I see that you did all that in a refactor to make it single-linked. So - :+1:. There's a similar case at https://github.com/zephyrproject-rtos/zephyr/pull/11443/files#diff-8983724739384d9f38274725cd16e2cfR57
Also, not clear the exact difference between tp_input() and tcp_input() (again, I had to follow the code to get an idea about that), etc.

Refactoring the test protocol into separate reusable module is planned and will be done to some extent. However, the test protocol requires an access to the TCP data structures. Might it be a good idea to keep the TCP specific parts if they will be relatively small?

I don't know in all fairness. I'd say the rule of thumb should be to keep it as separate as possible, and where it can't be, well, we'll consider it later.
All in all, these aren't request for changes (at this stage) - I remember your concern that if you post this, you'll be flooded with requests for cosmetic changes, while you're trying to get the functionality right. So, I'm NOT trying to do anything like that, at most to provide some feedback which hopefully will make polishing work easier later.
Otherwise, plugs for testing is absolutely needed. I have https://github.com/zephyrproject-rtos/zephyr/issues/3559 open for 1.5 years, with zero responses. I never went to do that myself, because I'm not familiar with that area, and don't know how to do that well. Per my understanding, you're effectively trying to implement such fault (and non-fault) injection framework, and it's direly needed. So, let the functionality come first, we can deal with the shape of code later.

@pfalcon

There's a similar case at […]

Refactored, please take a look.

Also, not clear the exact difference between tp_input() and tcp_input() (again, I had to follow the code to get an idea about that), etc.

To be done next.

Otherwise, plugs for testing is absolutely needed. I have #3559 open for 1.5 years, with zero responses. I never went to do that myself, because I'm not familiar with that area, and don't know how to do that well.

Per my understanding, you're effectively trying to implement such fault (and non-fault) injection framework, and it's direly needed. So, let the functionality come first, we can deal with the shape of code later.

It’s a bit broader thing (introspect, modify, command, inject, debug) and not all ideas are yet implemented.
Let me take a look into that issue and let’s think how to make it flexible.

@ozhuraki: Can you check/respond whether this PR is affected by https://github.com/zephyrproject-rtos/zephyr/pull/10547/ merged today? I guess that promise is that it shouldn't affect higher levels of stack, but who knows...
Otherwise I really hope to give this some try beginning of the next week, in time for monthly net forum meeting.
Thanks!

Implementation
Whitelisting approach

@ozhuraki: You may be surprised, but not everyone knows what "Whitelisting approach" means in that context. I heard that phrase from your presentations a few times, and (but?) what's left in my mind that the talk is about whitelisting state transitions in TCP state machine. But I won't be surprised if I didn't "get" it, or simply mix it up with something else by now.
Can you please expand in the description what is meant by that (and later in the commit message, if it's referenced there).

Added more networking people to the review. If it goes in, they'll be working with it, debugging issues, write tests, etc., so better they be prepared to it by reviewing it before that.

For example is there/can this be used with an echo server to which one can connect over telnet?

From the presence of samples/net/echo_server/overlay-tcp2.conf I conclude that the answer is.

@ozhuraki, Can you consider splitting  ext/fs/nffs/include/nffs/queue.h → include/sys/queue.h move commit as a separate PR? With the current state of review management in Zephyr, one can't be sure that it'll be merged, but it'll simplify management of this PR, and I ready to +1 it right away.

For example is there/can this be used with an echo server to which one can connect over telnet?
From the presence of samples/net/echo_server/overlay-tcp2.conf I conclude that the answer is.

However, on first run to try to use it like that, after just couple of pings:
uart:~$
[00:01:00.494,845] <err> net_tcp2.json_to_tp: Error: json_obj_parse()
[00:01:00.494,845] <err> net_tcp2.json_to_tp: Error: json_obj_parse()
***** Kernel OOPS! *****
Current thread ID = 0x00403d98
eax: 0x00000001, ebx: 0x0043c93a, ecx: 0x00404f01, edx: 0x0040a4b0
esi: 0xaaaaaaaa, edi: 0x00002640, ebp: 0x00491d64, esp: 0x00491d38
eflags: 0x00000206 cs: 0x0008
call trace:
eip: 0x0004df78
     0x0004e196 (0x404f00)
     0x0002d022 (0x4208c0)
     0x0001f07d (0x4208c0)
     0x0001f1a3 (0x4208c0)
     0x0001fc6b (0x4208c0)
     0x0001fc97 (0x4a6bc0)
     0x000563b4 (0x4208c4)
     0x00002657 (0x403d80)
Fatal fault in thread 0x00403d98! Aborting.

Actually, pings are not required - it crash just a few seconds after start. Anything can be involved, e.g. a network traffic which runs on any modern workstation.

Ok, so telnet to port 4242:
uart:~$ ***** Booting Zephyr OS zephyr-v1.13.0-2138-ga1fb55b876 *****

[00:00:11.646,978] <dbg> net_tcp2.tcp_in:
[00:00:11.646,978] <dbg> net_tcp2.tcp_in:
[00:00:11.648,088] <dbg> net_tcp2.tcp_in: NONE->LISTEN
[00:00:11.648,088] <dbg> net_tcp2.tcp_in: NONE->LISTEN
[00:00:11.650,822] <dbg> net_tcp2.tcp_in: SYN=0,ACK=1370579763
[00:00:11.650,822] <dbg> net_tcp2.tcp_in: SYN=0,ACK=1370579763
[00:00:11.651,040] <dbg> net_tcp2.tcp_in: LISTEN->SYN_RECEIVED
[00:00:11.651,040] <dbg> net_tcp2.tcp_in: LISTEN->SYN_RECEIVED
[00:00:11.658,172] <dbg> net_tcp2.tcp_timer_subscribe: SYN=0,ACK=1370579763
[00:00:11.658,172] <dbg> net_tcp2.tcp_timer_subscribe: SYN=0,ACK=1370579763
[00:00:11.661,589] <dbg> net_tcp2.tcp_out: SYN=0,ACK=1370579763
[00:00:11.661,589] <dbg> net_tcp2.tcp_out: SYN=0,ACK=1370579763
[00:00:11.666,497] <err> net_arp.net_arp_prepare: Gateway not set for iface 0x004a6bc0
[00:00:11.666,497] <err> net_arp.net_arp_prepare: Gateway not set for iface 0x004a6bc0
[00:00:11.667,225] <err> net_tcp2.tcp_out: Error: net_send_data()
[00:00:11.667,225] <err> net_tcp2.tcp_out: Error: net_send_data()
***** Kernel OOPS! *****
Current thread ID = 0x00403d98
eax: 0x00000008, ebx: 0x000630c6, ecx: 0x004a6bc0, edx: 0x0040a3d0
esi: 0xaaaaaaaa, edi: 0x00002640, ebp: 0x00491db4, esp: 0x00491d88
eflags: 0x00000202 cs: 0x0008
call trace:
eip: 0x0004ef7c
     0x0004d064 (0x405360)
     0x0004f022 (0x405360)
     0x0002d005 (0x4208c0)
     0x0001f07d (0x4208c0)
     0x0001f1a3 (0x4208c0)
     0x0001fc6b (0x4208c0)
     0x0001fc97 (0x4a6bc0)
     0x000563b4 (0x4208c4)
Fatal fault in thread 0x00403d98! Aborting.
[00:00:13.642,108] <dbg> net_tcp2.tcp_timer_cb:
[00:00:13.642,108] <dbg> net_tcp2.tcp_timer_cb:
[00:00:13.645,268] <err> net_arp.net_arp_prepare: Gateway not set for iface 0x004a6bc0
[00:00:13.645,268] <err> net_arp.net_arp_prepare: Gateway not set for iface 0x004a6bc0
[00:00:13.649,779] <err> net_tcp2.tcp_timer_cb: Error: net_send_data()
[00:00:13.649,779] <err> net_tcp2.tcp_timer_cb: Error: net_send_data()
***** Kernel OOPS! *****
Current thread ID = 0x004051a0
eax: 0x00000000, ebx: 0x00000000, ecx: 0x00499cdc, edx: 0x00000000
esi: 0xaaaaaaaa, edi: 0x004a1f68, ebp: 0x00499f30, esp: 0x00499f14
eflags: 0x00000206 cs: 0x0008
call trace:
eip: 0x0004e9e2
     0x00057196 (0x405398)
     0x00056c60 (0x405398)
     0x0001210c (0x1a)
     0x00012450 (0x0)
     0x00059353 (0x1)
     0x00059376 (0x1a)
     0x00002657 (0x0)
Fatal fault in ISR! Spinning...

Ok, seeing <IP_ADDRESS> is hardcoded in the source, switched to it after all (the above tests are with default <IP_ADDRESS> address as setup by loop-slip-tap.sh.
Now that the baseline is set - it's not yet there to deal with the trivial real-world cases - time to fire up a synthetic testsuite indeed.

Ok, so running the testsuite works:
$ ttcn3_start test_suite tcp2_check_3_runs.cfg
...
MTC@x230: Verdict statistics: 0 none (0.00 %), 24 pass (100.00 %), 0 inconc (0.00 %), 0 fail (0.00 %), 0 error (0.00 %).
MTC@x230: Test execution summary: 24 test cases were executed. Overall verdict: pass

Now just need to make it work with real connections ;-).

Changes since the previous version:

Changed to Zephyr's lists/queues
Split data to send and receive windows
Fixed net_pkt and net_buf leaks
Disabled retransmission (needs a slight rework)

@lpereira @pfalcon @dbkinder @jukkar @nashif @tbursztyka @agross-oss @rveerama1 @rlubos @aescolar @SebastianBoe @andrewboie @gnuless @nordic-krch @jhedberg
Re-review, please.

No documentation (rst files) changes, right?

Yes, no documentation changes.

@ozhuraki, Thanks for activity and pings on this PR!
But given that quite some time went with this PR not getting any textual updates, and only commits (hundreds of commits!), can you please summarize status of it? Has it reached a particular milestone? What should we expect from it? How are we supposed to test it?

There are many commits in this PR which makes reviewing challenging. I do not want all of these to be squashed into one as that would destroy the history of the PR, but there might be some that could be squashed.

+1. We definitely don't want to squash everything down to 1, that would be incomprehensible code mass-drop. But we want to squash everything which is just internal WIP. I'd say, number of commits should be decimated.
As an example, I don't think we'd want commit like this in upstream history:
@ozhuraki net: tcp2: Minor cleanup cbba023
@ozhuraki net: tcp2: Minor cleanup 99970c3

@jukkar @pfalcon

I do not want all of these to be squashed into one as that would destroy the history of the PR, but there might be some that could be squashed.

Done.

@pfalcon

How are we supposed to test it?

I have and would like to add this information. What would be the place for it?

agross-oss no longer works on Zephyr, removing him from reviewers.
Also, "area: Logging" labels looks confusing here, removing.

How are we supposed to test it?

I have and would like to add this information. What would be the place for it?

I would suggest to start with adding some information right here in the comments. Like, this PR didn't have any updates for several months, only stream of commits. What happened now that you call folks to re-review it?

I do not want all of these to be squashed into one as that would destroy the history of the PR, but there might be some that could be squashed.

Done.

Thanks. Old commit count: ~250, new commit: 127. It's great to see number of commits halved, but as I wrote, I would be nice to decimate it, i.e. make it 20-25 commits.
As it is now, this PR is very hard to work with in the Github UI, as due to length of the commit list (updated and reposted by Github on each push), even the latest comments are collapsed by default, and it takes a lot of scrolling and clicking around to get to them.
Again, would be nice to get a status update here, is it anyhow "ready", and in what sense. Then do rough cleanup, like reducing a number of these commits, then maybe close this and post a new PR to continue the review without accumulated UI issues.

An example re: squashing commits. Just around the top of the current commit list, there're commits:
net: tcp2: Check th_off in tcp_th()
@ozhuraki
ozhuraki committed on Jul 29
net: tcp2: Check th_off in tcp_data_len()
@ozhuraki
ozhuraki committed on Jul 29
net: tcp2: Guard on bogus th_off in tcp_in()
@ozhuraki
ozhuraki committed on Jul 29
net: tcp2: Reset fl in CLOSED
@ozhuraki
ozhuraki committed on Jul 29
All these are fixup commits, uninteresting on their own. Each of them should be squashed into corresponding "main" commit. For example, "Check th_off in tcp_th()". The idea is that tcp_th() should [have] check th_off from the beginning, so this fixup commit can/should be squashed to commit where tcp_th() was introduced.
I'm sure similar processing can be applied to dozens of other commits.

Ok, so I've tried to build and run this:
Branch: commit 290ce0ee4a916229b2d505383423ca3a5cb91ccb (ozhuraki/tcp2-dev)
Built samples/net/sockets/echo with: BOARD=qemu_x86 CONF_FILE="prj.conf overlay-e1000.conf overlay-tcp2.conf"
It runs, but first time I ran it, on client connection and initial data reception, QEMU crashed:
Booting from ROM..Optimal CONFIG_X86_MMU_PAGE_POOL_PAGES 8
***** Booting Zephyr OS build zephyr-v1.14.0-4904-g3e0be720da94 *****
Single-threaded TCP echo server waits for a connection on port 4242...
Connection #0 from <IP_ADDRESS>
recv: 6 byte(s)
zephyr/CMakeFiles/run.dir/build.make:57: recipe for target 'zephyr/CMakeFiles/run' failed
make[4]: *** [zephyr/CMakeFiles/run] Error 1
make[4]: Leaving directory '/home/pfalcon/projects-3rdparty/Embedded/Zephyr/zephyr/samples/net/sockets/echo/outdir/qemu_x86'
CMakeFiles/Makefile2:435: recipe for target 'zephyr/CMakeFiles/run.dir/all' failed
make[3]: *** [zephyr/CMakeFiles/run.dir/all] Error 2
make[3]: Leaving directory '/home/pfalcon/projects-3rdparty/Embedded/Zephyr/zephyr/samples/net/sockets/echo/outdir/qemu_x86'
CMakeFiles/Makefile2:442: recipe for target 'zephyr/CMakeFiles/run.dir/rule' failed
make[2]: *** [zephyr/CMakeFiles/run.dir/rule] Error 2
make[2]: Leaving directory '/home/pfalcon/projects-3rdparty/Embedded/Zephyr/zephyr/samples/net/sockets/echo/outdir/qemu_x86'
Makefile:209: recipe for target 'run' failed
make[1]: *** [run] Error 2
make[1]: Leaving directory '/home/pfalcon/projects-3rdparty/Embedded/Zephyr/zephyr/samples/net/sockets/echo/outdir/qemu_x86'
/home/pfalcon/bin/zephyr-make:30: recipe for target 'run' failed
make: *** [run] Error 2

On a few subsequent runs, it just loops with an error run from the start (without any client connection):
Connection #2755 from <IP_ADDRESS>
error: recv: 9
Connection from <IP_ADDRESS> closed
Connection #2756 from <IP_ADDRESS>
error: recv: 9
Connection from <IP_ADDRESS> closed
Connection #2757 from <IP_ADDRESS>
error: recv: 9
Connection from <IP_ADDRESS> closed
Connection #2758 from <IP_ADDRESS>
error: recv: 9

After a number of attempts, it subsides, this error looping no longer happens, and it goes to the initial state where client connection followed by data send crashes QEMU.
So, I'd like to extend query regarding status of this, what happened that re-review was requested, and how exactly to test it.

Then I tried BOARD=qemu_x86 CONF_FILE="prj.conf overlay-e1000.conf overlay-tcp2.conf overlay-log.conf". This crashed on startup, apparently stack overflow. Setting CONFIG_MAIN_STACK_SIZE=8192, I got it running. It even doesn't crash QEMU on incoming data. But neither it echoes anything back to client.

@pfalcon

Like, this PR didn't have any updates for several months, only stream of commits. What happened now that you call folks to re-review it?

Regarding the status, we were transparent, sharing the status and asking for the feedback at each Zephyr Networking Forum and keeping the daily workflow upstream.
Anyway, in case all this information was lost, the brief status it’s mostly feature complete (with a few exceptions), however needs more joint work in many areas.
We would like to merge it into 2.1, complete it and get to the production quality together thereafter.
In case there are bigger than style issues, it was agreed at the latest Networking Forum to file those as separate issues.
We have just completed the minimally sufficient integration with the socket API (however only smoke level testing is done, more extensive testing is required), which turned to be more challenging than anticipated.
Testing-wise we have started testing with Maxwel, I am working on unblocking @rmstoi and @rveerama1 on a few initial issues.
In addition, there has been developed a TTCN-3 based sanity check with completely open source workflow. The intent is to maintain it by side with the TCP2 implementation and to validate the functionality of each commit with it.

As it is now, this PR is very hard to work with in the Github UI

I can further compress the commits and create a new PR in case there are limitations of github preventing to look at it.
In case anything else might be helpful in the new PR, please summarize it, I will try to help.

@ozhuraki

Regarding the status, we were transparent, sharing the status and asking for the feedback at each Zephyr Networking Forum and keeping the daily workflow upstream.

I kinda guessed that. Sadly, since migrating Zephyr scheduling away from Google Calendar, I keep missing the Forum (because it's not possible to set custom notifications for all the monthly series, each meeting needs to be setup individually, which is easy to miss; oh, and official notifications, I somehow notice only when the meeting is over :-( ). So, I probably missed ~3 last forums due to vacations, travel, or above matters.
Anyway, this shouldn't be about me, because there can be other reviewers (or just interested parties) who weren't at the forum, so aren't in loop on the status.
And for my own and everyone else's reference, Network Forum archive folder is qt https://drive.google.com/drive/u/1/folders/1j6d0FLeOjiMil1Ellb59AsfHdzuWdAAc . I don't see an October report there, but September's one, should elaborate on info said by Oleg.

We would like to merge it into 2.1, complete it and get to the production quality together thereafter.

I'm happy to hear that! I don't see any other way forward, given that from my humble view, with the old stack, we reached a state when we apply workarounds, which lead to more subtle regressions, which are again worked around, etc., in circle. Let me remove my old -1 to not be obstacle to that plan. (But of course, I still the code in this PR needs more work, even if merely organizational (commit count, etc.)).

We have just completed the minimally sufficient integration with the socket API (however only smoke level testing is done, more extensive testing is required),

So, how to try things that you have tried?

which turned to be more challenging than anticipated.

May I suggest to open a separate ticket with a list of known issues/points to do? I'd be glad to help with these (disclaimer: for as long as I keep working i networking area). Btw, something I wanted to ask both yourself and @jukkar: will you guys be at ELCE (and associated Zephyr mini-summit) the week after next?

In addition, there has been developed a TTCN-3 based sanity check with completely open source workflow. The intent is to maintain it by side with the TCP2 implementation and to validate the functionality of each commit with it.

Ack, I personally remember that very well, kudos for doing that work, and thanks for including it into this status update for other reviewers' info.

I can further compress the commits and create a new PR in case there are limitations of github preventing to look at it.

That what I would suggest, but of course please wait for confirmation/further suggestions from @jukkar.

In case anything else might be helpful in the new PR, please summarize it, I will try to help.

With the above info (thanks!), the only missing part is how to see it in action with bare eyes/hands (i.e. without heavy machinery like TTCN3). As I wrote in https://github.com/zephyrproject-rtos/zephyr/pull/11443#issuecomment-543190295, I gave it a quick try, but couldn't get it to work.
Thanks!

@pfalcon

As it is now, this PR is very hard to work with in the Github UI

I have compressed the commits yet preserved @jukkar and @pfl contributions.
I am closing this PR and will open a new one as agreed.

May I suggest to open a separate ticket with a list of known issues/points to do?

I will open such a ticket shortly. I already addressed most of @jukkar input (but not everything). The rest of TODOs might go into this ticket. I will make a detailed list of the remaining ones there.

The only missing part is how to see it in action with bare eyes/hands (i.e. without heavy machinery like TTCN3).

I think, we will need both. Next, I will add the instructions and let’s proceed.


Disable Animations

In order to make Windows feel more snappy I always disable animations.
This app aiming to be a "one stop shop" for doing all good tweaks to Windows it would be nice to have this as one of the checkboxes under tweaks.

I believe "Set Display for Performance" does this

It didn't do it for me on a full clean USB install of the latest Windows 10.
I had to change it manually by hand.

Perhaps I am thinking of a different option, can you post a screen shot of what you do manually?

Please find a photo of this setting attached here:


stress test with malicious generators

(To be clear, I didn't review the correctness of the edge cases described in the document, mostly because it's written in an authoritative tone that suggest that you've examined this a lot closer than I have.)

yeah, ideally those test cases would be written in high level Chialisp. It's not good to have opaque tests like this. The problem, though, is that they use certain techniques not available from chialisp. For instance, chialisp always does constant folding/inlining eagerly. These programs rely on being small, but still executing large loops that could all be computed at compile-time, but doing so would make them have too high cost from space or be an ineffective attack since they would just boil down to a constant.

these were actually written by tweaking the output from the compiler, so it's not so easy to automate. Maybe if we have a more sophisticated compiler one day, we could improve them.


Error on start the extension

Extension: 0.2.3
VS Code: 1.9.1
MacOS 10.12.3
Node.js: 7.5.0
NPM: 4.2.0
Electron: 1.4.13

Steps to Reproduce:

On press opt-c p
Show the error (image)

[Extension Host] Start child process...
extensionHost.ts:282 [Extension Host] Check version of: electron-prebuilt
extensionHost.ts:282 [Extension Host] Check version of: comma-separated-values
extensionHost.ts:282 [Extension Host] Check version of: js-yaml
extensionHost.ts:282 [Extension Host] Check version of: process-bridge
extensionHost.ts:282 [Extension Host] Try to send IPC message: 1
extensionHost.ts:282 [Extension Host] [STDERR]: App threw an error when running [TypeError: Cannot read property 'on' of undefined]
extensionHost.ts:282 [Extension Host] Try to send IPC message: 1

Hi @Vitor83,
It seems that the Electron in your computer is broken.
Try this:
cd ~/.vscode/extensions/anseki.vscode-color-0.2.3
npm uninstall electron-prebuilt

And Pick Color command on VS Code.

Hi @anseki,
Thanks for the answer, but the error still persist, I've tried it before, I've uninstalled and reinstalled everything and still got the error

The electron and another electron might conflict.
Did you install Electron globally? If so, that should not be installed globally because various apps may not work by the confliction.
Try to remove that. (except there are special reasons)
npm uninstall -g electron
npm uninstall -g electron-prebuilt

Sure, when I use electron I only install with '--save-dev', it's not the case.
I just have two electron Apps installed on this Mac VSCode and Atom, I'm missing something maybe other package I really don't know.

I found it, you were right, I found two node_modules folder on my mac (I don't know why) and both had electron inside, I removed one and bingo!
Thanks a lot for your help, I love this extension

I'm glad to be of your help. 😄
So, the extension works fine?

Yes working fine, and I tested with electron installed globally and it works too, but I just uninstalled.

The "global" path in the process of VS Code might differ from that of your shell.
Your report may help me and someone. Thank you 😄


[Bug]: Professor Profile Picture not showing up

What did you expect to happen?
For professors with a profile picture, I expect it to show up in the Professor Overview component
What happened?
For Zygmunt Haas (ie) his profile picture did not show up.
Version
Beta 0.1.0
What browsers are you seeing the problem on?
No response
Relevant log output
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Testing shows that API is providing us with a dead link.
Richard Scotch shows up well, Zygmunt Haas does not.

Check back on this. Josh needs to run the scraper again.

The scraper was rerun and Zygmunt Haas is now showing up. Is this resolved?


feat(payment): STRIPE-82 Add code to show images on the list

What? STRIPE-82
Add images between list for payments providers
Why?
Display the payment method’s logo next the payment method radio selector label for the following payment methods
Testing / Proof

@bigcommerce/checkout @bigcommerce/apex-integrations

💚  testing done
https://bigcommercecloud.atlassian.net/jira/software/c/projects/STRIPE/boards/622?modal=detail&selectedIssue=STRIPE-82


Only one node status is healthy and all other nodes status  are pending

one node status is Healthy and other nodes status is pending and all have same node id

Figured it out it was a duplicate ID issue because all the machines were cloned from a VM that already have docker installed,so all have same node id. To resolve this delete this file  /etc/docker/key.json  and restart the docker service.


Fix flannel integration broken by #6921

#6921 seems to have broken flannel integration. Figure out why.

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
4eb97b91843981b999f39e4acebf59681571f91f
566330e874424561dc4089865e299e5520c88d37
d65adf83f8be1d6d07f63a9b1e8a8197bfe7ae0e

@aanm This is probably caused due to the way I removed the bpf redirects in the TPROXY PR. I guess that with flannel (and possibly with other integrations) we don't use the cilium_host/net veth pair for all traffic, which the new TPROXY iptables rules now somehow depend on. I'll fix this next week by reintroducing the bpf_redirects for the proxy traffic.

I believe that the integration was fixed it v1.6, however the CI testing continues to be broken.

This has been open for a long time and would usually be marked as stale, but it's assigned to someone so that mechanism did not work. I don't see a reason to keep it open.


Round cpuPerc values before casting to short

To keep the behaviour we had before, see:
https://github.com/crate/elasticsearch/commit/029f6c67d8ddf6ef2700014938f3a0b40b92e9d5

ack


For Review

@maratkomarov Policy to validate iam_instance_profile name from approved list.
Validates any resource with argument "iam_instance_profile" and caters for aws_launch_template where the argument is iam_instance_profile.name

@pgale61 tests are failing for this policy  https://github.com/Scalr/sample-tf-opa-policies/pull/16/checks?check_run_id=1059262424
You can debug test locally with opa test aws/enforce_iam_instance_profiles.rego aws/enforce_iam_instance_profiles.test.rego aws/enforce_iam_instance_profiles.mock.json -v

@pgale61 tests are failing for this policy https://github.com/Scalr/sample-tf-opa-policies/pull/16/checks?check_run_id=1059262424
You can debug test locally with opa test aws/enforce_iam_instance_profiles.rego aws/enforce_iam_instance_profiles.test.rego aws/enforce_iam_instance_profiles.mock.json -v

@maratkomarov I think I misunderstood the logic of the tests. One test is supposed to fail but I see now that the test itself is supposed to pass even though the policy check fails (Count == 0 vs Count > 0). Test amended.


Run biocro_regional with calibrated configuration file

Hi David, I guess I only need to replace config.xml under run/ directory and execute job.sh to run biocro regional on ROGER. But these files and folders are generated by RStudio from the VM, if we would like to submit the job through RStudio interface, should we put the calibrated configuration file somewhere in VM?

PEcAn doesn't currently have the capacity to do these analyses (using one
set of parameters and propagating met uncertainty). Indeed, I used a very
straight-forward MLE method rather than the Bayesian calibration in PEcAn
for this because we aren't propagating the parameter uncertainty that has
been a central focus of PEcAn development to date. it would be great to
build this in, but for now, lets work backward from the analysis that we
implemented on Roger.
So in short, lets not worry about the VM for now, until we want to
calibrate the model again.
Does that sound sensible?
On Mon, Oct 12, 2015 at 2:48 PM, Hao Hu<EMAIL_ADDRESS>wrote:

Hi David, I guess I only need to replace config.xml under run/ directory
and execute job.sh to run biocro regional on ROGER. But these files and
folders are generated by RStudio from the VM, if we would like to submit
the job through RStudio interface, should we put the calibrated
configuration file somewhere in VM?
—
Reply to this email directly or view it on GitHub
https://github.com/ebimodeling/biocro_regional/issues/16.

Still a little bit confused for the part running biocro_regional without VM. What we did before is to configure the illinois_mxg_settings.xml on VM and then do the job submission there. Then we are able to see the output on ROGER as well as config.xml and job.sh under run folder for each job. I know we can directly execute the job.sh to run biocro_regional on ROGER. For the calibrated configuration file, I guess I should choose one of the job folder (e.g.<PHONE_NUMBER>2), replace config.xml with calibrated one, and then run job.sh. Is that the correct way to directly run biocro_regional on ROGER?

The key bit is to run the script
/gpfs_scratch/haohu3/pecan/models/biocro/inst/regionalbiocro_met_uncertainty.Rscript
using the calibrated config.xml
I forget how we submitted this the last time - the key bit is that we need
to parallelize the for loop so that it doesn't take forever and doesn't
reserve a whole node and only use a single core.
On Mon, Oct 12, 2015 at 3:29 PM, Hao Hu<EMAIL_ADDRESS>wrote:

Still a little bit confused for the part running biocro_regional without
VM. What we did before is to configure the illinois_mxg_settings.xml on VM
and then do the job submission there. Then we are able to see the output on
ROGER as well as config.xml and job.sh under run folder for each job. I
know we can directly execute the job.sh to run biocro_regional on ROGER.
For the calibrated configuration file, I guess I should choose one of the
job folder (e.g.<PHONE_NUMBER>2), replace config.xml with calibrated one, and
then run job.sh. Is that the correct way to directly run biocro_regional on
ROGER?
—
Reply to this email directly or view it on GitHub
https://github.com/ebimodeling/biocro_regional/issues/16#issuecomment-147511506
.

That rings a bell. Last time, we added the ability to sample from met data and select random seed in the R code. I remembered we discussed a little bit about paralleling the code with parallel R loop (e.g., foreach) to improve the performance last time, but did not implement it yet. I think I can take it from here. Thanks!

David, have the calibrated results adjusted by the decreasing factor for each year along time? If not, can you remind me where should I find the those parameters?

I have not added this to the script. Eventually, I plan to add it to the run.biocro function. But I have not done so yet. At this point, it would be easier (less prone to error and having to redo the runs) to do the revise the output after the runs are finished.
But you can see the code used to fix the data in vignettes/regional_miscanthus_calibration.Rmd,
the key bit is:
ricker <- function(t, phi1 = 1, phi2 = 1, phi3 = 1) {
  y <- phi1 * (t^phi2) * exp(-phi3 * t)
  return(y)
}

correct.yield <- function(biocro_result){
  start <- min(biocro_result$year)

  # lesur correction
  corrected_lessur <- ricker(0:20, 4.12, 1.12, 0.13)
  age_correction <- corrected_lessur/max(corrected_lessur)

  # from Miguez et al 2013
  # http://onlinelibrary.wiley.com/doi/10.1111/j.1757-1707.2011.01150.x/epdf;
  # multiplied by 0.67 to take account of losses in senescence, postsenescence,
  # and during harvest (Clifton-Brown et al., 2004; Heaton et al., 2008).
  harvest_correction <- 0.67
  correction <- age_correction * harvest_correction

  correction <- data.table(year = start + 0:(length(correction) - 1), adjust = correction)

  a <- merge(biocro_result, correction, by = 'year')
  a[, `:=`(corrected_yield = Yield * adjust)]
  return(a)
}

## In the vignette, I use lapply on a list of results, but this is how the function would work:
correct_yield(biocro_result)

Looking through the code, I notice that if you are using the version of pecan in your directory on scratch, we need to add met_uncertainty = TRUE to the call to run.biocro in the file pecan/models/biocro/inst/regionalbiocro_met_uncertainty.Rscript

Thanks, David! I've already have the yield correction code applied. One thing is that I noticed the age_correction does not reach the peak value until year 10. Tao suggested that the peak harvest year for Miscanthus should be around year 4. Does it look right?
> age_correction
 [1] 0.0000000 0.2414986 0.4609035 0.6373440 0.7724084 0.8708191 0.9378916 0.9787567 0.9980848 1.0000000 0.9880754 0.9653652 0.9344510
[14] 0.8974945 0.8562890 0.8123083 0.7667518 0.7205845 0.6745732 0.6293172 0.5852762

Hmm we are both correct. I got that function from a review by Lesur et al; here is the figure:

I used that function to calibrate the model because I didn't want to use the same data for the adjustment and calibration. I need to think about this and look back at Becky Arundale's data and will get back to you.

In short, it wouldn't be wrong to use this correction, though we can certainly do a better job. Its mostly a matter of fitting the ricker model to the Illinois data with a random effect of site. If I give you the Arundale data, would you be able to do this?

If I am not mistaken, I recall the age correction rate you shared with me before is different.
Can we use our existing Illinois data to generate an age correction rate? The curve by Lesuer et al seems a very slow increase in the first 5 years and slow decrease after year 10; while the other curve I recalled has a fast increase at first 3 years but decrease fast after year 5.
I agree with you that both methods are correct in theory. Shall we find a time to meet to finalize this before moving to climate data set generation and associated group job submissions?

On Oct 14, 2015, at 4:38 PM, David LeBauer<EMAIL_ADDRESS>wrote:
In short, it wouldn't be wrong to use this correction, though we can certainly do a better job. Its mostly a matter of fitting the ricker model to the Illinois data with a random effect of site. If I give you the Arundale data, would you be able to do this?
—
Reply to this email directly or view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ebimodeling_biocro-5Fregional_issues_16-23issuecomment-2D148209878&d=BQMCaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=WMJKW94VqGLYjSbI6YrTJ9xywEYF-yldNx6D6Bi7k7w&m=y9D3mlOHwHbZpBUtMwoJiO1XDeKeuOXhQQBigrC9VS4&s=4DvDBO7tKrxX2EQ5p-1KdASotR6VqlDAuY19yIx90ZM&e=.

I think it is clear that we should use the Arundale data. It would be better to add the ricker model parameters to the calibration, but in lieu of that, we can fit it given the Arundale data. It is mostly a challenge to fit appropriately, and to make sure that we don't 'double dip' with the calibration data.
there are two reasons the Lesur may have that trend. first, it could be influenced by a few sites with long time series. Second, it is likely that in Europe the crops and pathogens grow more slowly and do not deplete resources as quickly.

I can try to do the fitting. I guess the Arundale data will look like the points in the plot and what I need to do is to find the parameters of a fitting curve (similar as in ricker model), is that correct?

Another thread for model simulation:
Would BioCro consider the impact of inter-annual weather variations? Saying: if we have two sets of 4-year weather data: (1999,2000,2001,2002) versus (1997,1999,2001,2002), would the yield in 2002 be the same in two sets?
Hao told me that the results look the same. If that is the case, how shall we incorporate the inter-annual weather variations in the analysis?
On Oct 15, 2015, at 12:51 AM, Hao Hu <notifications<EMAIL_ADDRESS>wrote:
I can try to do the fitting. I guess the Arundale data will look like the points in the plot and what I need to do is to find the parameters of a fitting curve (similar as in ricker model), is that correct?
—
Reply to this email directly or view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ebimodeling_biocro-5Fregional_issues_16-23issuecomment-2D148289336&d=BQMCaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=WMJKW94VqGLYjSbI6YrTJ9xywEYF-yldNx6D6Bi7k7w&m=9UDoJZ5ERWcpfS_cmzldPoP1sIKtSdGwCJuTM1fGMWg&s=rCjeb1MQrhASLE14OGb4J05Ee-mgPeuKW9zjIHb1EKw&e=.

@haohuuiuc
OK. instead of using corrected_lesur, try using this parameterization for the correction (other than replacing corrected_lesur for corrected_arundale, the above code should work now):
corrected_arundale <- ricker(0:20, 5.2 2.6 0.46)

This is how I got the parameters (directly from the figure below)
d <- data.frame(age = 1:10, arundale = c(5, 10, 21.5, 31.2, 34.5, 33.3, 29.5, 25.1, 23, 23.2))
nls(arundale ~ a * (age) * exp(-c * age), data = d, start=list(a = 4, c = 0.5))

Note that this parameterization reaches a max at year 5, (approximately b / c = 5.6 like the figure below:
t <- seq(0,10, by = 0.1)
y <- ricker(t, 5.2, 2.6, 0.46)
t[which.max(y)]

2.6 / 0.46

Figure from Arundale 2013 (Miscanthus on right):

@taolin1 @haohuuiuc the weather history should determine the yield of a given year. It is possible that the randomization code I put into run.biocro is not working, but the best place to start would be to open a new issue with the code used to determine otherwise.

David, I was able to do the yield correction based on the corrected_arundale value provided in your code. Here is how the result looks like,

In this case, I am doing a 15-year biocro model run, from 1996 to 2010 without random sampling. Year 1996 is the plant year, and the peak yield occurs at year 2002. The maximum corrected yield reaches at 18.382 (the unit is Mg/ha i guess?), do you think this result looks correct?

It looks too low by about 1/3 Can you point me to the raw model output and the code that you used to generate the figures?

The annual.result.RData is on the following path on ROGER,
/gpfs_scratch/haohu3/pecan/models/biocro/inst/IL/out/abbyak,resykt,RData
The R code I have is as follows,
load("annual.result.RData")

# Correct yield
ricker <- function(t, phi1 = 1, phi2 = 1, phi3 = 1) {
  y <- phi1 * (t^phi2) * exp(-phi3 * t)
  return(y)
}

correct.yield <- function(biocro_result){
  start <- min(biocro_result$year)

  # lesur correction
  #corrected_lessur <- ricker(0:20, 4.12, 1.12, 0.13)
  corrected_arundale <- ricker(0:20, 5.2, 2.6, 0.46)
  age_correction <- corrected_arundale/max(corrected_arundale)

  # from Miguez et al 2013
  # http://onlinelibrary.wiley.com/doi/10.1111/j.1757-1707.2011.01150.x/epdf;
  # multiplied by 0.67 to take account of losses in senescence, postsenescence,
  # and during harvest (Clifton-Brown et al., 2004; Heaton et al., 2008).
  harvest_correction <- 0.67
  correction <- age_correction * harvest_correction

  correction <- data.table(year = start + 0:(length(correction) - 1), adjust = correction)

  a <- merge(biocro_result, correction, by = 'year')
  a[, `:=`(corrected_yield = Stem * adjust)]
  return(a)
}
correct_yield<-correct.yield(annual.result)

# funtion to create a raster
createRaster <- function(df, brks, cols, name) {
  r <- raster(ncols=16, nrows=23, xmn=-91.50, xmx=-87.50, ymn=37.00, ymx=42.75)
  y <- df$lat
  x <- df$lon
  xy <- cbind(x,y)
  r0 <- rasterize(xy, r, df$corrected_yield, fun=mean)
  plot(r0, breaks=brks, col=cols, main=name)
  plot(counties.il, add=T)
}

# Add raster layer of yield
yield.max <- max(correct_yield$corrected_yield)
yield.min <- min(correct_yield$corrected_yield)
step <- (yield.max - yield.min)/8
brks <- seq(yield.min, yield.max, by=step)
brks <- round(brks,1)
nb <- length(brks)-1
cols <- rev(terrain.colors(nb))

par( mfrow = c(3, 5))
for (i in 1996:2010){
  createRaster(subset(correct_yield, year == i), brks, cols, i)
}

I am getting

Error in plot(counties.il, add = T) :
error in evaluating the argument 'x' in selecting a method for function 'plot': Error: object 'counties.il' not found

I've also edited your comment to include the required libraries.

Sorry that I forgot to comment out this line,
plot(counties.il, add=T)

It is just for adding the county boundaries to the map. To make this line work, you will need to load a shapefile first,
library("rgdal")
counties.il <- readOGR("./shp", "il") # il is the shapefile in the working directory

In general, this isn't doing what I thought it was doing, and I will have to check.

My fault - I forgot to yield should be (Stem + Leaf) * adjust. I can not recall if I did this in the calibration step

But in the current code, using stem, I would expect that in year 6, corrected_yield would equal Stem. So the following plot should have all points on the 1:1 line, but doesnt
correct_yield[year == 1986,plot(Stem, corrected_yield)]; abline(1:15,1:15)

So I will have to take a look to figure out where the errors are.

Thanks for exploring this. If you think it is necessary, we can have a meeting to work around this.

to parallelize around the lat/lon points, make sure to regester enough cores in your qsub statement then do this:
qsub -l walltime=03:00:00,nodes=1:ppn=20

library(foreach)
library(doMC)
registerDoMC(20)
foreach(point=1:nrow(latlon)) %dopar% {

}

David, can you help me check the following R code on ROGER when you have time? /gpfs_scratch/haohu3/pecan/models/biocro/inst/regionalbiocro_met_uncertainty.Rscript,
I don't think run.biocro function takes the met.uncertainty parameter correctly. The corresponding run.biocro.R are at /gpfs_scratch/haohu3/pecan/models/biocro/R.

what error occurs? Please see http://adv-r.had.co.nz/Reproducibility.html

Error in BioGro(WetDat = WetDat, day1 = day1, dayn = dayn, soilControl = soil.parms,  :
day1 and dayn should be between 0 and 365
Calls: run.biocro -> BioGro
Please see the following error,
In addition: Warning messages:
1: In max(doy) : no non-missing arguments to max; returning -Inf
2: In min(doy) : no non-missing arguments to min; returning Inf
Called from: stop("day1 and dayn should be between 0 and 365")
Here is the R code for regionalbiocro_met_uncertainty.Rscript
#!/usr/bin/env Rscript

args   <- commandArgs(trailingOnly = TRUE)
rundir <- args[1]
outdir <- args[2]

#library(PEcAn.all)
require(PEcAn.data.land)
require(PEcAn.BIOCRO)
require(BioCro)
require(PEcAn.data.atmosphere)
require(PEcAn.utils)
require(lubridate)

options(error = browser)

if(interactive()) {

  library(PEcAn.settings)
  settings <- read.settings("/gpfs_scratch/haohu3/pecan/models/biocro/inst/IL/illinois_mxg_settings.xml")
  runid <- tail(readLines(file.path(settings$rundir, "runs.txt")), n = 1)
  rundir <- file.path(settings$rundir, runid)
  outdir <- file.path(settings$outdir, "out", runid)
  point <- 1
}

config <- read.biocro.config(file.path(rundir, "calibrated_config.xml"))

met.nc  <- nc_open(config$run$met.file)
soil.nc <- nc_open(config$run$soil.file)
# atmco2.nc <- nc_open(file.path(inputdir, "co2/CO2_Global_HD_v1.nc"))

lat <- ncvar_get(met.nc, "latitude")
lon <- ncvar_get(met.nc, "longitude")
latlon <- expand.grid(lat = lat, lon = lon)

annual.result <- NULL
biocro_result <- NULL

for(i in 1:1){
 for(point in 1:1) {
  set.seed(i)
  lat <- latlon$lat[point]
  lon <- latlon$lon[point]
  out <- run.biocro(lat, lon, met.nc = met.nc, soil.nc = soil.nc, config = config, met.uncertainty=TRUE)
  #out <- run.biocro(lat, lon, met.nc = met.nc, soil.nc = soil.nc, config = config)
  annual.result <- rbind(annual.result, out$annually)
  save(annual.result, file = file.path(outdir, "annual.result.RData"))

  point.outdir <- file.path(outdir, i, paste0(lat, 'x', lon))
  dir.create(point.outdir, showWarnings = FALSE, recursive = TRUE)

  hourly <- out$hourly
  daily <- out$daily
  #save(hourly, file = file.path(point.outdir, 'hourly.result.RData'))
  #save(daily, file = file.path(point.outdir, 'daily.result.RData'))

  biocro_result <- rbind(biocro_result, data.table(lat = lat, lon = lon, daily))

  model2netcdf.BIOCRO(result = hourly, genus = config$pft$type$genus, outdir = point.outdir,
                      lat = lat, lon = lon)
  rm(hourly)
  rm(daily)
 }
}
save(biocro_result, file = file.path(outdir, "biocro_output.RData"))

By setting the met.uncertainty parameter to TRUE,
out <- run.biocro(lat, lon, met.nc = met.nc, soil.nc = soil.nc, config = config, met.uncertainty=TRUE)

We call run.biocro.R with an additional step to sample years from met data, here is the extra code in run.biocro.R to handle when met.uncertainty is TRUE,
 if(met.uncertainty == TRUE){
    met <- load.cfmet(met.nc, lat = lat, lon = lon, start.date = ceiling_date(as.POSIXct("1979-01-01"), "day"), end.date = floor_date(as.POSIXct("2010-12-31"),"day"))
    #met <- load.cfmet(met.nc, lat = lat, lon = lon, start.date = start.date, end.date = end.date)

    #year.sample <- met[,list(year = sample(unique(year), size = 15))]
    #met <- met[year %in% year.sample$year]
    year.sample <- list(year = sample(unique(met$year), size = 15))
    met <- met[met$year %in% year.sample$year,]
  } else {
    met <- load.cfmet(met.nc, lat = lat, lon = lon, start.date = start.date, end.date = end.date)
  }

thanks. I had to fix a bunch of things in the code to get this working. They are in commit c508b5bf79808bd1e82fc7c50dfa089448c56423.
It should work now after you pull changes from github (or copy regional_met_uncertainty.Rscript and run.biocro.R from /home/dlebauer/dev/pecan/biocro) and rebuild the PEcAn.BIOCRO module
e.g.
cd /gpfs_scratch/haohu3/pecan/
git reset --hard HEAD
git pull origin biocro-module2
R CMD INSTALL models/biocro

OR
rsync -r /home/dlebauer/dev/pecan/biocro /gpfs_scratch/haohu3/pecan/models/
R CMD INSTALL /gpfs_scratch/haohu3/pecan/models/biocro

THEN you can run the script
This is how I ran it for testing
/home/dlebauer/pecan/models/biocro/inst/regional_met_uncertainty.Rscript /home/dlebauer/pecan_remote/mxg_calibration/Brownstown_Research/run/28523/ /tmp/

and the results in /tmp/annual.result.RData look okay!
but you will probably need to change the name of the calibrated_config.xml file to config.xml and place it in some run directory.

Thanks David for the quick fix. I got the code working. One thing I am curious about is that the uncertainty run generated 13 records for each scenario instead of 15 (# of samples you set in the sample function) in my test run. The output can be found here, /gpfs_scratch/haohu3/pecan/models/biocro/inst/IL/out/annual.result.RData.

the easy fix is to change replace = TRUE to replace = FALSE

I see, it's due to the duplicated year value during the sampling. Thanks!

And I've push (but not extensively tested) a fix that should allow sampling with replacement: PecanProject/pecan@b2ca2c120997ab503e4ae665cf2ac7ea76800c9d
Then cleaned it up a little bit for clarity here: PecanProject/pecan@13889d013d0e89016f8e00b1fdb6b2d674ba116f

David, I have generated the results of Champaign for 500 runs. To calculate the yield, should I use, (Stem + Leaf) * adjust ?
The data is in the following path on ROGER,
/gpfs_scratch/haohu3/pecan/models/biocro/inst/IL/out

Please let me know if you find anything wrong this result.

Yes, stem+leaf * adjust
On Fri, Nov 6, 2015 at 10:34 AM, Hao Hu<EMAIL_ADDRESS>wrote:

David, I have generated the results of Champaign for 500 runs. To
calculate the yield, should I use, (Stem + Leaf) * adjust ?
The data is in the following path on ROGER,
/gpfs_scratch/haohu3/pecan/models/biocro/inst/IL/out
Please let me know if you find anything wrong this result.
—
Reply to this email directly or view it on GitHub
https://github.com/ebimodeling/biocro_regional/issues/16#issuecomment-154461693
.

Thanks, David. I updated the result for Champaign her /gpfs_scratch/haohu3/pecan/models/biocro/inst/IL/out_champaign on ROGER. And I applied the age and harvest correction with the following R code,
# Correct yield
ricker <- function(t, phi1 = 1, phi2 = 1, phi3 = 1) {
  y <- phi1 * (t^phi2) * exp(-phi3 * t)
  return(y)
}

annual.result$yeari<-round(annual.result$yearindex/10000)
correct.yield <- function(biocro_result){
  start <- min(biocro_result$yeari)

  # lesur correction
  corrected_arundale <- ricker(0:20, 5.2, 2.6, 0.46)
  age_correction <- corrected_arundale/max(corrected_arundale)

  # from Miguez et al 2013
  # http://onlinelibrary.wiley.com/doi/10.1111/j.1757-1707.2011.01150.x/epdf;
  # multiplied by 0.67 to take account of losses in senescence, postsenescence,
  # and during harvest (Clifton-Brown et al., 2004; Heaton et al., 2008).
  harvest_correction <- 0.67
  correction <- age_correction * harvest_correction

  correction <- data.table(yeari = start + 0:(length(correction) - 1), adjust = correction)

  a <- merge(biocro_result, correction, by = 'yeari')
  a[, `:=`(corrected_yield = (Stem+Leaf) * adjust)]
  return(a)
}
correct_yield<-correct.yield(annual.result)

The yield now ranges from 0 to 19.08. Do you think this value is still too low? If you think it looks okay, I will start the run for entire Illinois.

Yes it seems too low. I will review my calibration over the weekend

@haohuuiuc OK. I have updated the calibration step to calibrate against the mean yields for each site, rather than each year independently, and have selected the parameter set that minimizes the RMSE over the training / calibration data:  https://gist.github.com/dlebauer/1cdcda5bdc7ce2818dc8
Give this one a shot.

Thanks, David. After applying the new config.xml, the maximum corrected yield increases to 30.92 for 500 runs in Champaign. Do you think it now looks reasonable? The raw yield data is generated on ROGER /gpfs_scratch/haohu3/pecan/models/biocro/inst/IL/out

Yes that is reasonable. Go for it! After IL we can do the US.

On Nov 10, 2015, at 4:39 PM, Hao Hu<EMAIL_ADDRESS>wrote:
Thanks, David. After applying the new config.xml, the maximum corrected yield increases to 30.92 for 500 runs in Champaign. Do you think it now looks reasonable? The raw yield data is generated on ROGER /gpfs_scratch/haohu3/pecan/models/biocro/inst/IL/out
—
Reply to this email directly or view it on GitHub.


Fix code scanning alert - This API (java/io/File.(Ljava/lang/String;)V) reads a file whose location might be specifie...

Tracking issue for:

[x] https://github.com/NicheToolkit/rest-toolkit/security/code-scanning/74

The handling of such inspection vulnerabilities in the tool group is temporarily ignored