bevy_reflect: Add Type type

Objective
Closes #7622.
I was working on adding support for reflecting generic functions and found that I wanted to use an argument's TypeId for hashing and comparison, but its TypePath for debugging and error messaging.
While I could just keep them separate, place them in a tuple or a local struct or something, I think I see an opportunity to make a dedicate type for this.
Additionally, we can use this type to clean up some duplication amongst the type info structs in a manner similar to #7622.
Solution
Added the Type type. This should be seen as the most basic representation of a type apart from TypeId. It stores both the TypeId of the type as well as its TypePathTable.
The Hash and PartialEq implementations rely on the TypeId, while the Debug implementation relies on the TypePath.
This makes it especially useful as a key in a HashMap since we get the speed of the TypeId hashing/comparisons with the readability of TypePath.
With this type, we're able to reduce the duplication across the type info structs by removing individual fields for TypeId and TypePathTable, replacing them with a single Type field. Similarly, we can remove many duplicate methods and replace it with a macro that delegates to the stored Type.
Caveats
It should be noted that this type is currently 3x larger than TypeId. On my machine, it's 48 bytes compared to TypeId's 16. While this doesn't matter for TypeInfo since it would contain that data regardless, it is something to keep in mind when using elsewhere.
Testing
All tests should pass as normal:
cargo test --package bevy_reflect

Showcase
bevy_reflect now exports a Type struct. This type contains both the TypeId and the TypePathTable of the given type, allowing it to be used like TypeId but have the debuggability of TypePath.
// We can create this for any type implementing `TypePath`:
let ty = Type::of::<String>();

// It has `Hash` and `Eq` impls powered by `TypeId`, making it useful for maps:
let mut map = HashMap::<Type, i32>::new();
map.insert(ty, 25);

// And it has a human-readable `Debug` representation:
let debug = format!("{:?}", map);
assert_eq!(debug, "{alloc::string::String: 25}");

Migration Guide
Certain type info structs now only return their item types as Type instead of exposing direct methods on them.
The following methods have been removed:

ArrayInfo::item_type_path_table
ArrayInfo::item_type_id
ArrayInfo::item_is
ListInfo::item_type_path_table
ListInfo::item_type_id
ListInfo::item_is
SetInfo::value_type_path_table
SetInfo::value_type_id
SetInfo::value_is
MapInfo::key_type_path_table
MapInfo::key_type_id
MapInfo::key_is
MapInfo::value_type_path_table
MapInfo::value_type_id
MapInfo::value_is

Instead, access the Type directly using one of the new methods:

ArrayInfo::item_ty
ListInfo::item_ty
SetInfo::value_ty
MapInfo::key_ty
MapInfo::value_ty

For example:
// BEFORE
let type_id = array_info.item_type_id();

// AFTER
let type_id = array_info.item_ty().type_id();

Merging on Monday; feel free to clean it up in the meantime :)

Updated the code to use Type for FunctionInfo and ArgInfo.
Also renamed Type::type_id to Type::id (like how we have Type::path), as suggested by @killercup.


docs: db_groups parameter takes a list, not str

Description
connect parameter db_groups is expected to be a list, not a str.
Change README.rst and connect doc string to reflect this.
Testing
Tests already test db_groups with a list.
License

[X] By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Hey @hoxu , thank you for opening this PR correcting our docs. I should be able to review by end of week. From a quick glance, all looks good :)

verified build & readme generation
➜  amazon-redshift-python-driver git:(pr/99) ✗ python3.9 -m twine check dist/*
Checking dist/redshift_connector-2.0.906-py3-none-any.whl: PASSED
Checking dist/redshift_connector-2.0.906.tar.gz: PASSED


Warming Up an IP Address on 100% Autopilot

As we know, warming up the IP is a must when we need to target inboxes with a scale - even when we use a so popular SMTP service like Amazon SES. Now, planning the "gradually increasing" limit per hour or per day for proper warming up is an easy task. But, applying this plan actually on a server is still a boring manual work with a lot of hassle if we need to reach something like 500k emails per day or 1M emails per day. Because it would take up to 3 months!
It would be a life saving to make a built-in OR addon feature for this useful and important task. And I think something like Postal should has it. Period.
This free tool generates the right plan we should use based on our goal to reach in a safe period of time:
https://h-educate.com/email-marketing-toolkit/
I hope you can make something similar that we can actually implement it on the server with no hassle. So it got our back and warm up the IP on 100% autopilot.

#833 is supposedly getting some effort in the near future so might be the answer?


brains are upside down

In the preprocessed image previewers, brains are upside down.
Anterior should be up.
(Thanks @aylward !)
The OASIS data we've been working with is in an outdated format (ANALYZE) that doesn't have trustworthy image orientation info. So we might have to look at our process of converting to nifti and make sure the headers contain the correct orientation info.
Then we just need to make sure that the preprocessed image preview components actually check the nifti header for the orientation info and use that to orient the display.

I checked the nifti headers on the OASIS images we have been using.
For example
import nibabel as nib
nib.load('OAS1_0019_MR1.nii.gz')
print(img.affine)
# [[   1.    0.   -0. -176.]
#  [   0.    1.   -0. -208.]
#  [   0.    0.    1.    0.]
#  [   0.    0.    0.    1.]]

This suggests they are already in RAS axis order, and I somewhat confirmed this by inspection.
"Somewhat" because I've been worried that the conversion to nifti from the old ANALYZE format may have caused a left-right flip, and that's less straightforward to determine from inspection. Humans and their left-right symmetry :roll_eyes: . Will worry about that later.
For now, to close this issue:

We can adjust the camera in all our viewers (the four preprocessed image viewers and the analysis results viewer) to achieve the desired rotation for the OASIS images we've been working with. This will make things work at least when the orientation matrix in the nifti header is the identity matrix  as we see it is above.

Ideally, eventually:

We should read the affine from the nifti header and use it to determine how to display things (which array index to refer to as "axial" slice, and how to orient the view so that the brain is "anterior up")


Not all notifications got response

Hi,
Do anybody experience the problem, that each hour pushy does not get response to some sent notifications from Apple. That happens at the time when  auth token expires (it happens for about one hour and ten minutes when it is issued). Count of unanswered notifications depends on how many notifications are sent through connection. Seems that 50 notifications per second are enough.
It could be reproduced with following conditions. Say for example you have 8 threads where each tries to send 10 notifications at the same time. And for example you do it once a minute. When auth token expires, and you send next time, pushy receives response about expired token for some first notifications, but for all other remaining there is no response at all. It looks like that Apple when it receives new auth token in some way fails to respond on notifications with old token that it receives shortly after that. These numbers of threads and notifications I found empirically and with them this situation could be reproduced consistently. Also have tried to comment out auth token change and continue to send with expired token. on first attempt have not got response on all sent notifications, but on next iterations all notifications are responded. So it looks like a problem at Apple side, but would be nice if somebody can clarify. My pushy version is 0.10 and netty is 4.1.11.Final.
Thanks,
Z.

P.S> when tested, I have sent all notifications to the same device.

Hi @jchambers.
Could it be possible to add optional configurable time to live to TokenAuthenticationApnsClientHandler::authenticationToken. So when token is created, handler would schedule token recreation after some time. It could live together with current token expiration handling. Hope that then lost response problem would go away if token TTL is set less than one hour.
Seems that there is another problem with lost responses. It happens in our production environment about once per week. Monitoring system shows that couple thousand notifications do not receive response in short time and we manually cancel them (Future<PushNotificationResponse>) after some large timeout (it would be nice to have such a feature in pushy itself so caller simply gets some timeout or cancellation exception). I have limited access to production servers so can’t tell much details but it looks like some connection issues. Have tried to reproduce it locally, and seems that have found one case. If run tcpkill net <IP_ADDRESS>/16 to simulate problems with network, in log I see how connections are dropped and then reconnection process starts. When then cancel tcpkill, everything restores and normal delivery process continues, but responses to some connections are missed - GenericFutureListener::operationComplete. is not called. From debug it seems that ApnsClientHandler::close is not called and therefore responsePromises are lost on channel close. May be responsePromises cleanup could be moved to some other place, seems that ApnsClientHandler::channelInactive would be fine.
Thanks,
Z.

If you'd be so kind, could you share (probably as a gist?) logs at the TRACE level around a token expiration? It'd be helpful if we could see the traffic from the server for notifications before the expiration, notifications sent with the expired token, and a notifications sent with the new token.

Could it be possible to add optional configurable time to live to TokenAuthenticationApnsClientHandler::authenticationToken.

We'll consider it if this turns out to be an unavoidable server-side problem, but I'm not yet convinced that's the case (i.e. I'm still willing to believe it's on our end).

From debug it seems that ApnsClientHandler::close is not called and therefore responsePromises are lost on channel close. May be responsePromises cleanup could be moved to some other place, seems that ApnsClientHandler::channelInactive would be fine.

Could you please open a separate issue for this problem?
Thanks!

Hi, Jon.
There is gist: https://gist.github.com/janzar/3698e595d1fb8fa9309c177315a2434f. It contains both log file and application used to generate it. I have changed pushy a little bit:
TokenAuthenticationApnsClientHandler:
long keyShift = 67L * 60L * 1000L;
Date issuedAt = new Date(System.currentTimeMillis() - keyShift);
this.authenticationToken = new AuthenticationToken(signingKey, issuedAt);
and removed some fields from log file. Test application works following: once a minute it tries to send a batch of 50 pushes. As I create token with time set to 67 minutes ago, after little more than 3 minutes of creation it expires. So all pushes in first three batches get response, but only fraction in 4th batch and all again in 5th batch.

Could you please open a separate issue for this problem?

Created #480 for it.

Thanks for the logs; this is a big help! Will dig in and see what I can figure out as soon as possible.

Thank you!

I've done some more investigation and discovered some surprising behavior. As soon as we send a push notification with an expired token, we get a new SETTINGS frame from the APNs server:
[id: 0x9f0b0f7c, L:/<IP_ADDRESS>:49491 - R:api.push.apple.com/<IP_ADDRESS>:443] SETTINGS: ack=false, settings={HEADER_TABLE_SIZE=4096, MAX_CONCURRENT_STREAMS=1, INITIAL_WINDOW_SIZE=65535, MAX_FRAME_SIZE=16384, MAX_HEADER_LIST_SIZE=8000}

The notable thing here is MAX_CONCURRENT_STREAMS=1, which then leads to a a few frames being reset as follows:
[id: 0x9f0b0f7c, L:/<IP_ADDRESS>:49491 - R:api.push.apple.com/<IP_ADDRESS>:443] RST_STREAM: streamId=25, errorCode=7
[id: 0x9f0b0f7c, L:/<IP_ADDRESS>:49491 - R:api.push.apple.com/<IP_ADDRESS>:443] RST_STREAM: streamId=25, errorCode=5

Error codes 7 and 5 are "refused stream" and "stream closed," respectively (we get two messages because we had already sent a HEADERS and DATA frame for the given stream before getting the SETTINGS or RST_STREAM frames from the server).
There are two conclusions to draw here:

Within Pushy, we are not handling RST_STREAM frames correctly, and should be able to handle this problem without losing notifications. This will be our first priority.
The behavior of the APNs server, while technically legal, is a little surprising. I'll make some noise to make sure that's intentional and not a bug.

In any case, I think we have enough evidence to understand what's causing this problem and to understand that it's within our power to fix.

Hi, Jon.
Looks like it is the root cause of the problem. Great that you find it! Would like to know is there any plan when v0.10.1 could be released?
Thanks,
Z.


it just stops after a few days

I keep encountering a problem with the player freezing after a few days (usually around 4 days) of running.  It happens no matter which version of the code I use (master or 0.2, or missionfloyd’s python3 version.)  It happens when running on a Pi zero and on a Pi 4b.   No error is displayed on the attached monitor.  The frame count stops and it stops displaying new frames.  After reboot it works well again for a while.  Anyone else having that problem?  Any ideas what is going wrong?

There was an issue like that discussed here - https://github.com/TomWhitwell/SlowMovie/issues/18 - but shouldn't be in all those versions

If it's the issue mentioned above, it's fixed in the latest waveshare driver (https://github.com/waveshare/e-Paper). Try installing that.

I installed the updates driver.  Will give it a try.  thanks

Sorry for my question, but how can I change the driver? Is it enough to overwrite the "/home/pi/SlowMovie/e-paper/RaspberryPi&JetsonNano/python/lib/waveshare_epd/epd7in5_V2.EPD" file with the new version?
Thanks for your help!

The waveshare driver (https://github.com/waveshare/e-Paper) was updated a few days ago. Try using that.

Sorry, but I think the driver is an old version - just uploaded new. The Version is 4.0 from 2019-06-20.

Sorry for my question, but how can I change the driver? Is it enough to overwrite the "/home/pi/SlowMovie/e-paper/RaspberryPi&JetsonNano/python/lib/waveshare_epd/epd7in5_V2.EPD" file with the new version?
Thanks for your help!

To install: in the e-paper/RaspberryPi_JetsonNano/python folder there's a file called setup.py. Run sudo python setup.py install
To uninstall: sudo pip uninstall waveshare_epd
Or, you can just put the waveshare_epd folder in /home/pi/SlowMovie

Still stopping after several days.  The pi keeps working, the program just stops showing new frames.  No error shown.  Works after reboot for a bunch more days then same thing happens.  I have tried several versions of SlowMovie or three different versions of pi (zero, 3 and 4b) using two different 7in5_V2 e-paper displays.  Used the driver from v0.2 here and the most recent one from Waveshare github and same on all from fresh reinstalled sd cards.  The longest it has worked has been 6 days before no new refreshes.  No one else encountering this problem?  Any ideas how to track down where the driver may be entering an unending loop or other potential explanations?  Is there a way to have the pi reboot if the program doesn't update the log file with a new frame number after a specified amount of time if the problem cannot be identified or corrected?  thanks for your help.

Very strange - definitely not having that problem here - mine runs for months quite happily

Still stopping after several days. The pi keeps working, the program just stops showing new frames. ..

Hi!
I have the exact same problem. But just on one of my frames - the second one is at my mothers home - and hers is working perfect. I'll try to put her SD-card in my frame in a few days, so I can check if its a software or hardware problem.

@victorwisc do you have the logs from the new code? tail -f /home/pi/SlowMovie/slowmovie.log

I do but there is not much in there.
[2021-04-13 21:28:42,014] Update interval: 120
[2021-04-13 21:28:42,014] Frame increment: 4
[2021-04-13 21:28:43,572] Resuming at frame 38192
[2021-04-13 21:28:43,573] Playing 'Lifeboat.mp4'
[2021-04-13 21:28:43,574] Video info: 140166 frames, 23.976fps, duration: 5846.08s
[2021-04-19 22:17:32,763] Exiting Program
I did do another git pull before restarting this time and I notice my log is now different, with INFO on the line so I did not have the most recent but should have been at least when it was restarted on 4/13.

Yeah, should be recent enough. It looks like something killed the program and the code on line 33 was run. Were you running it as a service? @missionfloyd might know more also

Yea, I am guessing the Exiting Program was when I restarted the Pi. It is running as a service. It is strange that it only happens every 4 or 5 days so far. I was considering simply restarting the Pi every 3 days in the middle of the night and I would probably never know it. :)

Thanks for the new version and perfect readme and step-by-step manual! My frame is now working since a view days without any problems. My frame is not running as a service.

Mine has been running for 15 days on a Pi4 without stopping.  That is by far the longest so fingers are crossed the problem is resolved.  I followed the directions posted at that time exactly as written and used the repo that was available at noon that day.  The only other thing I did differently than in my previous attempts was I used it as a hat instead of the SPI cables to connect to the pi.  Unfortunately there doesn't appear to be a log file for this one so if it crashes I wont be able to look back to see what may have happened.
I have a second e-paper connected to a Pi Zero I started yesterday.  In the past the Zero crashes within 24 hours but it is still running.  I used the installer for that one.  I have this logging at the debug level.
Both are running as a service.
Will post if they stop.
Victorwise: I tried an auto-reboot at 3 AM every night a few weeks ago.  The reboots went well but the Pi4 images still stopped after 4-5 days.  Can't figure out why that would happen.  Hopefully another scheduled reboot attempt won't be necessary if things keep running well.

Seems like something about the newer updates fixed whatever issue this was! I'm going to close the issue, but please feel free to update or reopen if the issue pops up again!


chore: 🤖 upgrade to angular v18

(new PR without npm install --force)

npm run update
npm run migrate
fix: Invalid self-closing element in index HTML file: '<app-root />'.
npm i -D @jscutlery/semver@latest --save-exact
npm i -D postcss@latest --save-exact
bump @ngneat/spectator to v19
npm audit fix (14 vulnerabilities (10 moderate, 4 high))

PR Checklist
Please check if your PR fulfills the following requirements:

[x] The commit message follows our guidelines: https://github.com/ngneat/transloco/blob/master/CONTRIBUTING.md#commit
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[x] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[x] Other... Please describe: Angular version update

What is the current behavior?
Angular 17
Issue Number: N/A
What is the new behavior?
Angular 18
Does this PR introduce a breaking change?

[ ] Yes
[ ] No

Other information

Why is this marked as a breaking change? Nothing is actually affected

You're right.
Removed the "breaking change" from commit body and PR description

I ran yesterday the upgrade as well, but I did nx 17 -> 18 -> 19 instead of the latest directly, and I didn't get all these changes in the project.json files which makes me wonder which version is the correct one and why I didn't get those changes.
I ran the migrations between each step of course.
Any idea?

Hmmm... Indeed, I don't see project.json changes with intermediate upgrades.
Found Nx's recommendation on upgrades:

One major version at a time, small steps
https://nx.dev/recipes/tips-n-tricks/advanced-update#recommendations

I'll revert the commits and rebase this branch with intermediate upgrades

Yep, that's what I did 😄

Much better now without project.json changes 😄


Added fix for issue #72 and solved reference issues

This pull request adds the enhancement I detailed in Issue #72 (it uses the system accent color for the title bar by default). It also fixes an issue I've found, whereby a missing dependency on the Visual C++ 2015 Runtime for UWP wreaks project-wide havoc. I fixed it by (gasp) adding the missing dependency. Don't know how that got missed in the first place, but it's fixed now!

I've added a few changes since my original pull request. This pull request also uses the accent by default for the backgrounds of PageHeader and Hamburger menu. Feel free to test the heck out of this!


Fix the Sum method.

There is currently a method called Sum that lives in /lib/sum.js.
It is incomplete and needs to be fixed!
Acceptance Criteria:

Running yarn test sum results in tests passing.

Please include the skeleton of a new method + an accompanying test for someone else to work on at the time of creating a pull request!

I'm working on it

Great- Thanks @dudmacedo !

Haha so I PR'd all the open tickets, and in that five minutes it looks like they've all been claimed :)

Anyhow this is out there..
https://github.com/BlakeGuilloud/ganon/pull/6

Solved, PR #17


Add tests for --privileged with other flags

With Podman v2.0, we broke (or thought we were going to break) using --privileged with --group-add and --security-opt (specifically using --security-opt for SELinux config).

@rhatdan PTAL at the SELinux bits

LGTM, but the SELinux test could be slightly improved

LGTM

@baude @QiWang19 @ashley-cui @vrothberg @TomSweeneyRedHat @jwhonce PTAL

LGTM
assuming happy tests

/lgtm


Create hidooor

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

welcome

It seems that the wrong repo was used for the pull-request.
I'll close this PR and if needed, a new PR can be introduced that provides more information and content about the desired changes to Envoy.


Feat: Added ktgeducation.edu.my

@Xavier-IV I am sorry, this domain seems to not be eligible for this repository. I have to close the request without merging. If you are a student of this school, please apply with official document ("OFFICIAL DOCUMENT" tab at https://www.jetbrains.com/shop/eform/students).


syclcc export correct include paths in 'syclcc -v -E -x c /dev/null'

vscode cpptool uses 'syclcc -v -E -x c /dev/null' to fetch include directories. The change fixes the issue in wrapper.

Thank you!
I think the root cause of this is that syclcc does not treat the invocation as a compilation. The logic to determine whether a syclcc invocation is a compilation at the moment is pretty basic: It only checks if there is a source file argument that ends in .cpp and other common C++ file endings.
So I wonder, wouldn't it be a much cleaner solution to just change this so that if -x c (and -x c++) arguments are present it also causes it to treat the invocation like a compiler invocation?
This would also cause it to correctly emit the include paths of backend dependencies (e.g. CUDA or ROCm) which I think your change does not do.
I think this would only require changing is_pure_linking_stage() here:
https://github.com/illuhad/hipSYCL/blob/77db25b6d077d3fb1fb2079fd085e629689f46e0/bin/syclcc-clang#L542
PS: If you want vscode integration, it might be a good idea to rely on the hipSYCL cmake integration which should automatically export this kind of stuff.

So I wonder, wouldn't it be a much cleaner solution to just change this so that if -x c (and -x c++) arguments are present it also causes it to treat the invocation like a compiler invocation?
nice suggestion - however "-std=c++17" in comm_cxx_flags is incompatible with "-x c".
In general, we should treat invocation with 4 possible modes - preprocess (-E, -S etc), compilation (-c) or impure/pure linkage.
'syclcc -v -E -x c /dev/null' is preprocess mode - common include should be passed but no cuda/rocm specific flags.


pull request for e2e test at 12/16/2016 09:56:18

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


3.2.13 Adjustments

Adjustments for 3.2.13 release.

Superseded by https://github.com/scalatest/scalatest/pull/2152 , we cannot update scalatest+testng to 3.2.12 yet because scalatest+testng <IP_ADDRESS> contains a bug that will be fixed by https://github.com/scalatest/scalatestplus-testng/pull/22 , since it is only used by our tests, let's only update that after the fix is released.


[BUG]: Self-hosted agent on M2 hangs when building MAUI on NET6 / NET7 / NET8

What happened?
Previously following https://github.com/xamarin/xamarin-macios/issues/17825 but can't update it since I'm not a collaborator. Also related https://github.com/microsoft/azure-pipelines-agent/issues/4205
This has been an issue on NET 6 & NET 7. I just upgraded to NET 8 since it released and the above bug has been fixed, however I still have the issue. Corresponding macios bug: https://github.com/xamarin/xamarin-macios/issues/19547
Kicking off a dotnet publish for a NET 8 Maui ios app, hangs forever. Opening a terminal on the mac, pasting the exact line, and executing it in the agent's folder, runs fine.
Command
dotnet publish 'Maui.csproj' -f net8.0-ios --self-contained -r ios-arm64 -c Release -p:BuildIpa=True -p:ApplicationId=com.app.dev /p:ApplicationVersion=61217 /p:ApplicationDisplayVersion=0.1.31 /p:ApplicationTitle='AppDevTemplate Dev'
Hardware
Model Name:	Mac Studio
Model Identifier:	Mac14,13
Model Number:	Z17Z000JXLL/A
Chip:	Apple M2 Max
Total Number of Cores:	12 (8 performance and 4 efficiency)
Memory:	64 GB
Agent Capabilities

Name
Value

clang
/usr/bin/clang

curl
/usr/bin/curl

docker
/usr/local/bin/docker

dotnet
/usr/local/share/dotnet/dotnet

git
/usr/bin/git

HOME
/Users/envocadmin

InteractiveSession
False

java
/usr/bin/java

JAVA_HOME
$(/usr/libexec/java_home)

JAVA_HOME_11_X64
/Library/Java/JavaVirtualMachines/microsoft-11.jdk/Contents/Home

JDK
/usr/bin/javac

LANG
en_US.UTF-8

LOGNAME
envocadmin

make
/usr/bin/make

MSBuild
/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild

PATH
/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:/Users/envocadmin/.dotnet/tools:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/homebrew/bin:/opt/homebrew/bin

PWD
/Users/Shared/Agents/Agent2

python3
/usr/bin/python3

rake
/usr/bin/rake

ruby
/usr/bin/ruby

SDKMANAGER
$ANDROID_HOME/cmdline-tools/7.0/bin/sdkmanager

sh
/bin/sh

SHELL
/bin/zsh

SSH_AUTH_SOCK
/private/tmp/com.apple.launchd.fZyAVo93Zs/Listeners

TMPDIR
/var/folders/xk/7mssnqbd0v5f_z1fwblvmw1m0000gn/T/

USER
envocadmin

VSTS_AGENT_SVC
1

Xamarin.Android
/Library/Frameworks/Xamarin.Android.framework/Versions/Current/Version

Xamarin.iOS
/Applications/Visual Studio.app/Contents/MacOS/vstool

xcode
/Applications/Xcode.app/Contents/Developer

XPC_FLAGS
0x0

XPC_SERVICE_NAME
0

Let me know if I can email you diagnostic logs or the link to the build definition. I'd prefer not to post the whole logs here.
Versions
Agent version: 3.230.0
macOS: 13.6
.NET: 8.0.100
Microsoft.Maui.Sdk: 8.0.3
Microsoft.iOS.Sdk: 17.0.8478
Environment type (Please select at least one enviroment where you face this issue)

[X] Self-Hosted
[ ] Microsoft Hosted
[ ] VMSS Pool
[ ] Container

Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
No response
Operation system
macOS Ventura 13.6.1
Version controll system
Git on Azure Devops
Relevant log output
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/bitcode_strip
  Optimizing assemblies for size may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
  Optimizing assemblies for size. This process might take a while.
/Users/envocadmin/.nuget/packages/xamarin.firebase.ios.core/<IP_ADDRESS>/lib/net6.0-ios15.4/Firebase.Core.resources/GoogleUtilitiesComponents.xcframework/ios-arm64/GoogleUtilitiesComponents.framework/GoogleUtilitiesComponents : warning MT7091: The framework /Users/envocadmin/.nuget/packages/xamarin.firebase.ios.core/<IP_ADDRESS>/lib/net6.0-ios15.4/Firebase.Core.resources/GoogleUtilitiesComponents.xcframework/ios-arm64/GoogleUtilitiesComponents.framework is a framework of static libraries, and will not be copied to the app. [/Users/Shared/Agents/Agent2/_work/9/s/Mobile/src/Maui/EnvocTemplate.AppDevTemplate.Maui.csproj::TargetFramework=net8.0-ios]
/Users/envocadmin/Library/Caches/XamarinBuildDownload/FAnlytcs-8.9.1/FirebaseAnalytics-8.9.1/Frameworks/FirebaseAnalytics.xcframework/ios-arm64_armv7/FirebaseAnalytics.framework/FirebaseAnalytics : warning MT7091: The framework /Users/envocadmin/Library/Caches/XamarinBuildDownload/FAnlytcs-8.9.1/FirebaseAnalytics-8.9.1/Frameworks/FirebaseAnalytics.xcframework/ios-arm64_armv7/FirebaseAnalytics.framework is a framework of static libraries, and will not be copied to the app. [/Users/Shared/Agents/Agent2/_work/9/s/Mobile/src/Maui/EnvocTemplate.AppDevTemplate.Maui.csproj::TargetFramework=net8.0-ios]
/Users/envocadmin/Library/Caches/XamarinBuildDownload/GAppM-8.9.1/GoogleAppMeasurement-8.9.1/Frameworks/GoogleAppMeasurement.xcframework/ios-arm64_armv7/GoogleAppMeasurement.framework/GoogleAppMeasurement : warning MT7091: The framework /Users/envocadmin/Library/Caches/XamarinBuildDownload/GAppM-8.9.1/GoogleAppMeasurement-8.9.1/Frameworks/GoogleAppMeasurement.xcframework/ios-arm64_armv7/GoogleAppMeasurement.framework is a framework of static libraries, and will not be copied to the app. [/Users/Shared/Agents/Agent2/_work/9/s/Mobile/src/Maui/EnvocTemplate.AppDevTemplate.Maui.csproj::TargetFramework=net8.0-ios]
##[debug]Re-evaluate condition on job cancellation for step: 'Publish iOS'.
##[debug]Started cancellation of executing script
##[debug]Exit code null received from tool '/bin/bash'
##[error]The operation was canceled.
##[debug]System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.VisualStudio.Services.Agent.Util.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, Boolean continueAfterCancelProcessTreeKillAttempt, CancellationToken cancellationToken) in /Users/runner/work/1/s/src/Agent.Sdk/ProcessInvoker.cs:line 376
   at Microsoft.VisualStudio.Services.Agent.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, InputQueue`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, Boolean continueAfterCancelProcessTreeKillAttempt, CancellationToken cancellationToken) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/ProcessInvoker.cs:line 308
   at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.DefaultStepHost.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, Boolean continueAfterCancelProcessTreeKillAttempt, CancellationToken cancellationToken) in /Users/runner/work/1/s/src/Agent.Worker/Handlers/StepHost.cs:line 76
   at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.NodeHandler.RunAsync() in /Users/runner/work/1/s/src/Agent.Worker/Handlers/NodeHandler.cs:line 212
   at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.RunAsync() in /Users/runner/work/1/s/src/Agent.Worker/TaskRunner.cs:line 407
   at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken) in /Users/runner/work/1/s/src/Agent.Worker/StepsRunner.cs:line 238
Finishing: Publish iOS

Hi @svaldetero thank you for reporting this issue. We're working on higher-prioritized issues at the moment, but we will get back to this one as soon as we can.

The underlying issue appeared to be https://github.com/microsoft/azure-pipelines-tasks/issues/19174. This issue can probably be closed.


Use an overridable method for all logging

Hi,
I currently use middleware to wrap the service logger with context info and place it in the ctx - this allows me to add tracing/session information to the log entries automagically that binds them together which is very useful in a busy multi-user system.
All my logging statements then use ctx.logger instead of this.logger and in my api services I have overridden logRequest and logResponse so I can use req.$ctx.logger instead of this.logger.
This is working well, but some of the logging statements in moleculer-web don't "work" because they are directly written using this.logger.  Example in callAction:
if (route.logging) {
	if (this.settings.logRequest && this.settings.logRequest in this.logger)
		this.logger[this.settings.logRequest](`   Call '${actionName}' action`);
	if (this.settings.logRequestParams && this.settings.logRequestParams in this.logger)
		this.logger[this.settings.logRequestParams]("   Params:", params);
}

Would it be possible to send all "generic" logging to a method, e.g. "log" with parameters level, req (if available) and message?
That way I could override it and if req exists and has $ctx, I can use that.
The default implementation would be something like:
log(req, level,message) {
    if (level in this.logger) this.logger[level](message);
}

and the above code would change to:
if (route.logging) {
	if (this.settings.logRequest)
		this.log(req, this.settings.logRequest,`   Call '${actionName}' action`);
	if (this.settings.logRequestParams)
		this.log(req, this.settings.logRequestParams, "   Params:", params);
}

Then, I can override "log" and use req.$ctx.logger.
There may even be a neater way you can think of?
Many thanks,
S.

Very interesting, but doesn't that still rely on the method/action using the store info?  I can't repoint this.logger to anything as that would affect concurrent requests.  I don't think I can get the logging statements inside moleculer-web to use my logger unless it's through overridden methods can I?

  I was playing around with the AsyncLocalStorage and it solves all my problems.  Using it for the localAction and localEvent hooks, I can then retrieve it in the newLogEntry hook and add in all the session/trace data that makes it into Google Cloud Logging.
Thanks a million!

Awesome. Maybe would be good if you can share some code of your solution that can help others 👍

You bet, here's a summary:
In my case, the problem was that we run a highly multi-user system with several different moleculer applications (each with dozens of services) that talk to each other.  This is all running in the cloud and we wanted to make use of the cloud logging a tracing tools to link up the spans and logs for a single user "session" (initiated by the first API call in).
With the standard moleculer logging you don't get access to the ctx so you have to get the developers to put ctx info in their messages which is a pain and error prone.  Plus this still doesn't provide the necessary tracing info for the cloud logging/trace tools.
Our solution is specific to Google Cloud and uses the Winston logger with the LoggingWinston (@google-cloud/logging-winston) transport.
Thanks to advice from @icebob , all we need to do to automatically get the session and tracing information into the LogEntry objects that are sent to GCP is to implement a couple of middleware hooks:
const { LoggingWinston } = require("@google-cloud/logging-winston");
const _ = require("lodash");
const { AsyncLocalStorage } = require("async_hooks");
const asyncLocalStorage = new AsyncLocalStorage();
const util = require("util");

module.exports = {
    name: "SessionLoggingMiddleware",

    localAction(next, action) {
        return (ctx) => asyncLocalStorage.run(ctx, () => next(ctx));
    },
    localEvent(next, action) {
        return (ctx) => asyncLocalStorage.run(ctx, () => next(ctx));
    },
    newLogEntry(type, args, bindings) {
        // Get the context
        const ctx = asyncLocalStorage.getStore();
        if (process.env.LOGGERTYPE === "cloud") {
            if (ctx) {
                // Add in the tracing info
                let opts = {};
                opts.timestamp = new Date();
                if (ctx.requestID && _.has(ctx, "broker.options.cloudinfo.resource.labels.project_id")) {
                    const traceValue = `projects/${ctx.broker.options.cloudinfo.resource.labels.project_id}/traces/${ctx.requestID}`;
                    opts[LoggingWinston.LOGGING_TRACE_KEY] = traceValue;
                }
                if (ctx.id) {
                    const spanId = ctx.id.replace(/-/g, "").substring(0, 16);
                    opts["logging.googleapis.com/spanId"] = spanId;
                }

                args.push(opts);
            }

            // Use the bindings to add namespace and module data to log entry as
            // the LoggingWinston transport doesn't respect the format options at the moment
            // We end up with 2 args: the message and the options
            // The options will be the last arg and will be an object
            let opts = args.pop();
            if (!_.isObject(opts) || !opts.timestamp) {
                // Not a logging-winston object so put back in the args array
                args.push(opts);
                opts = null;
            }
            // Now format the message
            const printArgs = (args) => {
                return args.map((p) => {
                    if (_.isObject(p) || Array.isArray(p)) {
                        let objPrinter = module.exports.defaultObjectPrinter;
                        if (ctx) objPrinter = _.get(ctx, "broker.options.logger.options.objectPrinter", objPrinter);
                        return objPrinter(p);
                    }
                    return p;
                });
            };
            args[0] = `${bindings.ns}/${bindings.mod.toUpperCase()}: ${printArgs(args).join(" ")}`;
            args[1] = opts;
            args.splice(2);
        } else {
            // Console logger
            if (ctx && ctx.requestID) args[0] = `[RequestID=${ctx.requestID}] ${args[0]}`;
        }
    },

    defaultObjectPrinter: (o) => util.inspect(o, { showHidden: false, depth: 2, colors: false, breakLength: Number.POSITIVE_INFINITY }),
};

Then in moleculer-config.js:
    /// Register custom middlewares
    middlewares: ["SessionLoggingMiddleware"],

Using the AsyncLocalStorage to store the ctx means we can retrieve it in the newLogEntry and perform the steps needed to format the log entry and add the tracing info to the options.  We also have a simpler entry when using the default console logger (still outputs the "session" id).
This in combination with the rootCallOptions feature added in #225 means we can link all of our applications together and they all use the same session id.  This results in the whole chain of calls across actions and events over 5 or 6 applications is presented in a single place - a godsend for the operations team.
Thanks again to the contributors to the moleculer eco-system, we've found it to be great code with a great community.

Awesome, thanks for sharing!

@synsteve @icebob
Hello,
I have a similar issue to the one described above and ran into an edge case that I hope one of you can help me with.
The solution provided works great whenever I make requests via the gateway service, however when I request with broker.call instead from another service (using AMQP transporter), the "localAction" hook in the middleware doesn't fire prior to the "before: *" hooks in the service's mixins. So whenever a "newLogEntry" event is triggered via a log in one of these mixin hooks, the context in the asyncLocalStorage is undefined. Once the mixin hooks finish and "localAction" kicks in, it works perfectly for all logs after that.
Any ideas/tips that could help me with this issue?


README.md per issue 11

Closes #11

I used the top half (overview) of the old readme since the full readme is in the Android folder.


net/haproxy: improve GUI usability

UPDATE:
Thanks to the great feedback and suggestions in the comments I was able to implement the following changes in #330:

[X] diverge from HAProxy's wording, better align with other LB solutions
[X] restructure and group GUI elements (should be easier to grasp)
[X] hide unused options in many places with some simple JS code
[X] add a "Quick Start Guide" and multiple smaller help pages
[X] refactor the model to allow validations for all fields
[X] while here, add a few missing HAProxy options
[X] fix bugs in rarely used HAProxy functions (and introduce new bugs, of course)

The new tab/menu structure is as follows:
|-Introduction
|
|-Real Servers
||-Introduction
||-Servers
|
|-Virtual Services
||-Introduction
||-Backend Pools
||-Public Services
|
|-Rules & Checks
||-Introduction
||-Health Monitors
||-Conditions
||-Rules
|
|-Settings
||-Service Settings
||-Global Parameters
||-Default Parameters
||-Logging Parameters
||-Statistics Configuration
|
|-Advanced
||-Introduction
||-Error Messages
||-Lua Scripts

You'll notice that many entries have been rephrased to better align with other Loadbalancer GUIs – this also means that we've diverged more from HAProxy's own wording:

ACL => Condition
Action => Rule
Backend => Backend Pool
Error Files => Error Messages
Frontend => Public Service
Health Check => Health Monitor

Although I was unable to reach all goals, I think it's still an improvement and a first step towards a more user-friendly HAProxy GUI.

ORIGINAL DESCRIPTION
Currently the GUI for HAProxy ACLs and Actions is rather confusing for new users, even if they have some experience with HAProxy. I think there are two major issues:

Input fields only have generic names like "Value", "Name", and so on. While this makes the code/model less complicated, it's hart to grasp for a user what to enter in those input fields. Furthermore the naming is odd in some cases and does not follow HAProxy's official wording.
All input fields are always visible. There's no indication for a user which field is required or may be left empty.

My proposal is a major rework of the ACL/Action model to implement specific input fields for every ACL/Action type. And I'm going to add the JS code from our acme plugin to hide input fields that are not required. Besides that this allows us to add input validations for every field. Futhermore I plan to follow HAProxy's official wording more closely to make it easy for veterans to relate to our GUI.
Below is an example for a redesigned ACL page. The Actions page will benefit even more, because it has twice as many input fields.
OLD GUI

NEW GUI

Note that in the new GUI all inapplicable input fields are hidden and the input field has a precise name. It now works very similar to Acme's Validation page.
A LOT code will be changed, so this should become version 2.0 of our HAProxy plugin.

@fichtner @AdSchellevis Would love to get some more input before working on this, especially if you found other quirks that should be addressed or if suggestions have been made by other users. The ACL/Action stuff is essential in HAProxy, but I think it's not easy to do it the right way (easy and flexible).

@fraenki it's indeed a bit complicated, quite flexible, but I can imaging people are searching a bit from time to time. Hiding the non usable items in the gui would certainly help, maybe some general help on the first screen could also help understanding the steps you should take. Let me think about this some more.
By the way, I did help a customer lately installing haproxy and the setup went quite smooth from my end :)

@EugenMayer This is loosely inspired by a conversation we had in June. Feel free to suggest specific enhancements here. I plan to follow the official HAProxy wording more closely this time.

@fraenki I've browsed through haproxy and came up with the following:
If I'm understanding the basics correctly, normally a setup should look something like this

My first idea would be to add an overview such as this to the "general" tab, so a new user can grasp the context of what the solution can do.
Next I would rearrange the tabs in the following order:
[General], [Servers], [Backends], [Frontends], [Actions], [Errors],  [Advanced]
The rational  for this order is that you need to have servers to setup a backend, then you can connect your default backend to you frontend.
In the actions tab I would combine ACL/Actions (both grids on the same page with some explanation on the page) , because the end result seems to be an action which we can "salt" the backend and frontend with.
Errors would contain the error file serving options, I would add a field for "default" there (initially checked).
Then we have the Lua scripts, this sounds like an "Advanced" option, maybe we should put it on it's own there.
Last and final are the dialog options that should hide when not selectable, that would really help to create simple setups really fast.

@fraenki i would love to participate here, but first, i would like to set the expectations we have / understand them.
For me, i see 2 different user-types and as equal use cases.
normal user:
Tends to stay in use-case "simple", more or less wires up domains to servers, no fallbacks, no special treatment. All the user is going to do is, set a domain(s)[important) to be forwarded on a specific server ( one server, no balancing ). He decides the port, if the dest server has SSL on or of, never verify. Of course, adding SSL certificates for SNI based offloading is important.
advanced user:
this user tends to use the same use-case above mostly, 99% of his setup has this kind of idea. This user might specifically care about verifying endpoint SSL connection ( server SSL ) in the normal mode.
But this user will also need the "advanced' mode from time to time. Special rules like redirect, special ACLs to match against and / or special balancing settings, several servers and all this. In this mode, this user is mostly no HAproxy beginner or he is at least willing to read documentation. He is mostly even aware of the documentation even and wants to use it ( haproxy docs ).

That set straight, i would actually like to address both users or more, both use-cases. I learned a lot from the use-cases with HAproxy on opnsense as my central ssl offloader, i have special rules, but 99% normal rules.
but also i learned a lot from ranchers HAproxy configurator and one things is clear: they have a great interface which allows you to quickly setup domain->server configuration, with or without SSL offloading while still offer the complete flexibility if needed - but it does not cross your way or makes the setup bloated.
The latter point is one of the biggest issues with opn HAproxy. It is powerful, it is flexible and it does offer a good structure - but it is heavily overloaded when covering the 90% normal usage.
First i started creating the 20 ACLs, 20 Actions, 20 Backends and 20 Servers and then wiring them all using the same rule: frontend 443 if domain X -> server X.
This whole click process, combined with all the glitches the UI has, is such a huge effort, that its not only failing in being structured and therefor offering oversight, it also fails in actually being handy at all.
In the end, in opn haproxy, i do no longer have any ACLs, Actions - all i use is the advanced mode ( text mode ) and backends /servers in the GUI. this drastically reduced the effort adding or adjusting things, it reduced ordering issues ( redirects ), it reduced complexity and gave me even better oversight.
Compared to rancher, where i happily use the UI for exactly the same reason, exactly the same use-case and amount of rules.

Before i start to outline my idea, i wanted to see what your expectations are when redesigning this - i would like to offer a fast, reliable way to do "when port X with domain X got to ip Y port Z" + adding SSL

i would like to ensure that the expert mode is still available and gives the user the right flexibility when its needed - but does not slow the usual process down

If i am dragging this into the wrong direction, just tell me and i do not try to pull this discussion into something you do not actually plan to, safe your and my time and more or less happily continue to use the text-mode

Having used HAProxy for the first time and only through OPNsense's GUI, I'd like to add my 2 cents and describe what confused me the most:

Terminology. ACLs should be more aptly named conditions (more HAProxy's "fault" than OPNsense's)
Tab order (as already mentioned). For setting up the first time I need servers and backend first and then frontends. Afterwards I pretty much only changed frontend and ACLs because the servers don't change that often. Maybe have a "setup wizard" that asks things in the right order for the first setup?
Overhead. Pretty much all my actions have a 1:1 relationship to their ACLs. Maybe combine those somehow? As in "if THIS condition (= ACL setup) then do THAT (= action setup)"?
Not being able to use host and port aliases. Everything was going so well and then I had to look up and enter those IPs again ;) The server tab at least lets me use FQDNs, but in the frontend I have to enter raw IPs.
No validation on addresses and ports. I accidently entered a wrong IP address in the frontend (typo) and all that happend was the HAProxy service stopping/not starting. Took me a while to find out what happened and why.
No automatic firewall rule for frontend ports. I guess the NAT auto rule creation spoiled me there, but it still would be nice ;)
No (visible) automatic config validation - I always have to remember to click "test syntax" before I click "apply".
Useless validation warnings like

[WARNING] 239/123609 (29929) : parsing [/usr/local/etc/haproxy.conf:54] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.
Configuration file is valid

when I don't have any control over the rule order (or do I?).
All those generically named and/or extra inputs not needed for a specific ACL or action didn't confuse me that much, to be honest. I just tried to enter the bare minimum that I recognized and it pretty much worked.
All in all I would like to see HAProxy more integrated into core OPNsense (use of aliases, automatic FW rules), and that fatal configuration errors would create a notification and not just a red icon somewhere.

Hi all,
thank you for the great feedback and suggestions! Really appreciated! I've found some time to work on this and updated the description with the current state. I think a new "development" release will follow shortly for you to test.
First of all: Please continue the feedback loop! This plugin will continue to evolve, but this takes time and requires continued feedback. :)
With that said I need to clarify that I was unable to address all suggestions. Many GUI improvements would require an "UI artist" to join and send patches. Other improvements would require extensive code contributions to the plugin framework, which is out-of-scope – at least for now.
I want to address some of the suggestions, because I feel sorry that I wasn't able to fully implement them and I think they should be implemented in a future release:

Overhead in ACLs and Actions

There's overhead, that's true. I still think it was a good decision to separate these concerns in the backend code. That being said, from a user-perspective I agree that it would be desirable to have both in the same GUI. But this would require extensive JS code that I'm unable to provide... patches are welcome!

"Not being able to use host and port aliases"

Truely desirable, but...

requires a model/framework change to support multiple IP/port pairs for the same frontend
a workaround in JS might be possible

I'll try if this can be implemented for servers, though.

"No validation on addresses and ports."

The "Test Syntax" button should be able to catch these errors. The upcoming release contains work towards more validations: ACLs and Actions fields are greatly improved.

"No automatic firewall rule for frontend ports"

Unfortunately not on my list for a long time. Patches welcome!

"Useless validation warnings [...] when I don't have any control over the rule order (or do I?)."

Rules are processed in the same order that is visible in the GUI. Currently the order cannot be changed; the rules need to be removed from the list and added in the required order.
Would require some GUI/framework additions to make the entries sortable by dragging and dropping items. Help wanted!

"it is heavily overloaded when covering the 90% normal usage"

I've tried to mitigate this issue by hiding many unused options.

"i would like to ensure that the expert mode"

I don't think we should introduce two separate configuration modes (expert mode/easy mode). But a "Setup Wizard" for some easy tasks would certainly be useful. Needs contributions, I can't provide the required JS code (and there's currently no MVC example for this in OPNsense).

Thank you for working on this! Period. :)
I will comment briefly later on

+1 Thanks ;)

"No validation on addresses and ports."
The "Test Syntax" button should be able to catch these errors.
Well, it didn't. To clarify: It was a syntactically correct address, I just swapped two digits and HAProxy consequently could not bind to it.

Re: aliases:

requires a model/framework change to support multiple IP/port pairs for the same frontend
Not sure what you mean? I can already configure multiple IPs and ports for the same frontend.
Maybe as a first idea the entered aliases could resolve themselves to IPs and ports on saving the frontend record? Then you wouldn't have to change the backend code itself and I wouldn't have to remember IPs ;)

Dumb question: How can I test the new version?

@StrikerTwo a very valid question...
This will delete your previous /usr/plugins directory, make sure you have backed up your changes if you have any (or custom plugins):
# opnsense-code -f -a fraenki plugins
# cd /usr/plugins
# git checkout haproxy_model200
# cd net/haproxy
# make upgrade

Cheers,
Franco

@StrikerTwo  Oh and please note that the model changes will not be able to revert to the previous config layout so make a copy of your config.xml to restore if needed

I've tried to mitigate this issue by hiding many unused options.
Thanks for trying this, but i guess, you got me wrong on this. I did not complain about the complexity of the UI, i rather did complain about the effort to do anything. After the 5th time you can safely navigate through the forms having 20 fields you do no need - no hassle with the "complexity of forms".

But the amount of clicks, entities and effort (flow) you need to create a single "if that domain then this backend" is far too much.
So the expert mode was not about hiding fields ( like the advanced mode ), its more a flow where you can set and chose all the options explicitly, requiring you to actually go through those 5 forms . The default mode should put that all on one form and guess / default on all the others.
I think it is like a wizard, but please do not get this like a starter-wizard or a "once done never again" wizard, which is most of the cases those are used. lets call it a proxy form, where e.g. you enter the name, server ip/port and the domain you match against, if its HTTPS and the required certificate if needed. This creates rules, actions, backends servers and adds entries in the frontend. Sounds easy at first, will become hard if you have to find out which frontend to actually modify - you will need machine names for the auto-configured frontends, but then, you will run into the issue that people need to mix modes... and so on. Usually that gets into a dead end technical wise or usability wise.

Would require some GUI/framework additions to make the entries sortable by dragging and dropping items. Help wanted!

Pick jquery drag&drop and sortable, should be fairly easy to implement. You will create a new field in the model "weight" holding the sort value:
https://jqueryui.com/sortable/
https://jqueryui.com/resources/demos/draggable/sortable.html
https://stackoverflow.com/questions/18399543/jquery-ui-combine-sortable-and-draggable

I fully understand that you ask for contributions here and that very valid. Even though i would be able to actually provide this, i am just to involved in other things including the opnsense development stack is horrible overcomplicated, especially for quick contributions.
This said, i would rather sponsor your time working on this then engaging myself - but i guess you are not interested since you never answered on this.

Even though i would be able to actually provide this, i am just to involved in other things including the opnsense development stack is horrible overcomplicated, especially for quick contributions.

Do you mean MVC or legacy code? Sorry, but for me as a non-programmer it's quite easy to read/write/understand this (MVC).

@mimugmail no its not the actually MVC framework, it rather running, deploying and testing the opnsense stack during development. Including setting up the freeBSD VM, code sharing but also splitting between "oh thats legacy" and oh thats "not legacy" code. When i was writing https://github.com/EugenMayer/tinc-opnsense i had a lot of fun with https://github.com/EugenMayer/tinc-opnsense/blob/master/security/tinccustom/src/etc/rc.d/routes.addon (nice example about legacy ... ) .. https://github.com/EugenMayer/tinc-opnsense/blob/master/security/tinccustom/src/etc/rc.syshook.d/tinc0_inteface.early and a lot more quirks like https://github.com/EugenMayer/tinc-opnsense/blob/master/security/tinccustom/src/etc/inc/plugins.inc.d/tincdcustom.inc#L20 being all undocumented, guess your way through it and try hard.
I am not at all eager jumping on this again at all. You need huge insight into opnsense, its historical creation and its current status in the components to know how and what you can develop. Beside that, reading what Frank is writing, there seem to be 0 layer between Model and View - what Frank is fighting with ( model representation being 1:1 in the UI ) is the reason MVC died and being replaced by MVVM or MWV - esp. MVVM would help a lot layering the backend Models against the View Models so one can have a total different representation in the UI compared to the backend / technical representation. Being forced to bundle those together leads to bad UI / user experience or to bad technical design or to massive overhead layering this "the manual way".
But that is far off topic here, sorry.

Quick feedback: I like the new UI. Especially the cleaned condition/rule dialogs.
The tabs and their sub-menus look odd, though:

I found myself clicking the wrong sub-menus because the dashed lines seem to separate the tabs themselves and not the sub-menu from the tab label.
Maybe you could add an option to disable all the introduction pages, or at least move them to the end of the sub-menus? They are very helpful for beginners, but just add clicks and UI clutter for experienced users.
IPs are still not validated (in the sense that they actually need to be assigned to an interface) - the service just doesn't start in that case (not even an error log entry).

Thanks again for your feedback!

"No validation on addresses and ports."
IPs are still not validated (in the sense that they actually need to be assigned to an interface) - the service just doesn't start in that case (not even an error log entry).

This field supports all sorts of different values: FQDNs, IPv4 and IPv6 addresses along with a port number. TBH, I have no idea how to protect this field against all the possible misconfigurations.

I think it is like a wizard, but please do not get this like a starter-wizard or a "once done never again" wizard, which is most of the cases those are used. lets call it a proxy form, where e.g. you enter the name, server ip/port and the domain you match against, if its HTTPS and the required certificate if needed. This creates rules, actions, backends servers and adds entries in the frontend.

Got it. And I like the idea, but unfortunately it's not going to happen anytime soon. :(

The tabs and their sub-menus look odd, though:

Hmm. The tabs are defined in core, so it's not a bug, it's a feature... :) but I will look into it. Maybe it can be improved.

Maybe you could add an option to disable all the introduction pages, or at least move them to the end of the sub-menus? They are very helpful for beginners, but just add clicks and UI clutter for experienced users.

I like the idea. I will open a new ticket to investigate how this could be implemented.

The next version of the HAProxy plugin will include the new GUI.

This field supports all sorts of different values: FQDNs, IPv4 and IPv6 addresses along with a port number. TBH, I have no idea how to protect this field against all the possible misconfigurations.

Okay. Maybe at least better error handling for this case then? Push a notification to the OPNsense GUI and/or write an entry to the logfile if the service cannot be started?

Sorry for touching this old issue.
But can anyone give directions how to make an own plugin version that will only support text box to configure haproxy myself? I hate so much fields (items) to create instead of writing 10 lines of config. IIRC, pfSense had an option of manual haproxy configuration, but opnsense doesn't.
All I want is to pass a single text config to save and restart daemon. Could anyone recommend a simple plugin as an example? In present state haproxy is unusable for me, too much hassle to find out how to make things I had in a text config...
Even better if the original interface had a big switch like "Advanced configuration", turning that on will remove all options but single text box. I think it's easy to do, but I just started palying with opnsense, and this seems to be like a stopper for me. It might be user friendly for newbies, but it is hard to configure if you know haproxy well, but don't like overcomplicated UI...

@osnwt: Maybe ask over in the forums. That being said, nothing prevents your from using plain HAProxy – you don't need to install the plugin.

Yes, you are right. But in that case I can use GUI-less plain OS with few packages.
It's possible to improve the plugin. It's possible to ask on forums. I switched to original pfSense, where custom passthru options for every haproxy section are available out of the box:
If it is about usability, here is it:

@osnwt Please stop hijacking this issue. Feel free to open a new issue or post on the forums instead.

@osnwt beside that the opnsense plug in has the same backend textfield to define both, backends and frontends in text Form. just activate the advanced mode above.
so there is hardly any difference between opnsense and pfsense in this regard

I agree with said above that mixing of both approaches creates some mess. If anyone uses only basic features, he may be happy with gui way. If one wants to create an advanced config like using the same port for (ssh, https, openvpn), or even few single server backends, then adding a lots of conditions, backends and servers via GUI requires a lot of objects and time instead of simple text construct like
frontend http-in
        bind <IP_ADDRESS>:80
        use_backend http-www.example.com if { hdr(host) -i www.example.com }
        use_backend http-mail.example.com if { hdr(host) -i mail.example.com }

backend http-www.example.com
        server www.example.com <IP_ADDRESS>:80

backend http-mail.example.com
        server mail.example.com <IP_ADDRESS>:80

haproxy is not an integral part of *sense, it is a plugin. So I would like to have a single text box version of such plugin, where I can edit, test and save single config using GUI and without ssh connection. That is the reason for simple "skeleton plugin" to add new features without spending time to fancy GUI. I would definitely want to adopt such, but have neighter time nor motivation to build a full blown GUI.
In any case, I did not find pass thru options for haproxy in all sections when I looked for (maybe just missed). In addition, I run it as VM under ESXi server, and opnsense for some reason sometimes failed to boot (stopped at db> prompt until restarted). So I tried the pfSense (that did not do that), and now playing with it.


[mesheryctl] Improve user experience for mesheryctl system lifecycle commands

See mesheryctl Command Tracker for current status of commands. For additional context, see https://github.com/meshery/meshery/pull/5661.
Current Behavior
Responses from mesheryctl commands don't always have the best user experience in mind.
Desired Behavior
Improve the error messages in mesheryctl commands from the user's perspective.

Ensure that there are proper error messages when users:

Give more arguments to a command than required
Misspell a command
Do not give the required input via flags or args
Use the command without any arguments or flags.

The error messages being displayed to users are meaningful from user's perspective. And if applicable, a potential fix for the problem should be shown too.  For example

"failed to unmarshal response body" could be modified to "couldn't process JSON response from server"
"manifest not found" could be modified to "manifest not found. Please provide the path to the app manifest."

Implementation
The commands in scope of this issue are:

[ ] mesheryctl system
[ ] mesheryctl system stop
[ ] mesheryctl system update
[ ] mesheryctl system reset
[ ] mesheryctl system status
[ ] mesheryctl system restart

Acceptance Tests

Contributor adhered to the Meshery CLI Style Guide.

Contributor Resources

⌨ mesheryctl Contributing Guide
Beginner's guide to contributing to Meshery and mesheryctl
mesheryctl Command Tracker
Meshery CLI Commands and Documentation
🙋🏾🙋🏼 Questions: Layer5 Discussion Forum and Layer5 Community Slack
📚 For the Reference head over to this pr.

Hi Is this open to assignment @leecalcote ?

Yes, although, I thought that you were already collaborating with someone on a similar issue. No?
Do you have output from having locally built and run mesheryctl?

yes we are doing . I felt this is similar so asked upon. I will finish that first 👍
yes this is the output

hey @leecalcote
could this be assigned to me?

Hello, this is my first time contributing to meshery, can I take this issue up?
Here's the output after building it locally ->

Thank you for confirming that you're able to build and run the project @hamees-sayed.

@leecalcote this issue seems to be a little outside my scope for the time being so I'll unassign it, take a step back, familiarize myself more with the project and hopefully return to it later. @legendarykamal can take this issue

@hamees-sayed, no sweat. Thank you for resetting expectations. Let's be sure to find another issue that fits. 👍

@leecalcote I would like to work on it
My last PR #6430 is ready for review

Oh, very good, @devkapilbansal. ✅


Tests broken against official install dependencies

Things affecting tests:

postgis installation
triggers extension installation

See http://cartodb.readthedocs.org/en/latest/install.html#postgresql vs https://github.com/CartoDB/cartodb/blob/ead4e65b83a1a562cb6b3e5718acbc60a0fca685/README.md#install-postgresql
Tested against postgis-2.1.7 and worked, but installing the official current version from ppa's did not. To be taken into account along with CI builds.
# regression.diffs
*** /root/src/cartodb-postgresql/expected/test/CDB_QueryTablesTest.out  2015-09-08 17:32:09.772957923 +0000
--- /root/src/cartodb-postgresql/results/test/CDB_QueryTablesTest.out   2015-09-08 17:33:37.692954005 +0000
***************
*** 1,5 ****
  \set ECHO none
! SELECT * FROM geometry_columns|{pg_catalog.pg_attribute,pg_catalog.pg_class,pg_catalog.pg_namespace,pg_catalog.pg_type}
  SELECT a.attname FROM pg_class c JOIN pg_attribute a on (a.attrelid = c.oid)|{pg_catalog.pg_attribute,pg_catalog.pg_class}
  CREATE table "my'tab;le" as select 1|{}
  SELECT a.oid, b.oid FROM pg_class a, pg_class b|{pg_catalog.pg_class}
--- 1,5 ----
  \set ECHO none
! SELECT * FROM geometry_columns|{pg_catalog.pg_attribute,pg_catalog.pg_class,pg_catalog.pg_constraint,pg_catalog.pg_namespace,pg_catalog.pg_type}
  SELECT a.attname FROM pg_class c JOIN pg_attribute a on (a.attrelid = c.oid)|{pg_catalog.pg_attribute,pg_catalog.pg_class}
  CREATE table "my'tab;le" as select 1|{}
  SELECT a.oid, b.oid FROM pg_class a, pg_class b|{pg_catalog.pg_class}
***************
*** 21,24 ****
  DROP TABLE
  DROP SCHEMA
  SELECT
! * FROM geometry_columns|{pg_catalog.pg_attribute,pg_catalog.pg_class,pg_catalog.pg_namespace,pg_catalog.pg_type}
--- 21,24 ----
  DROP TABLE
  DROP SCHEMA
  SELECT
! * FROM geometry_columns|{pg_catalog.pg_attribute,pg_catalog.pg_class,pg_catalog.pg_constraint,pg_catalog.pg_namespace,pg_catalog.pg_type}

======================================================================

Should be ok now


email node "Error: Greeting never received"

Hi!
I am a newbie on node-red, and I am trying to send an email, when a GPIO button is pressed.
Latest version of node-red (installed a week ago) on Raspbian Jessie distro RPi ver.3
I tried with 3 different mail clients that I use (gmail, outlook, yahoo) but I still get the same error on the debug tab: "Error: Greeting never received".
The email starts sending proccess and after 2-3 secs it says "send failed" and the error on the debug tab appears.
Is there any solution to this?
Thank you in advance.

Please use the Google Group or Slack to ask usage questions, both are linked to from the Node-RED home page.

Ok, thank you!

Actually this may be a bug... or at least could be made more easily worked around locally.
(by exposing security option).

Closed by 367237d94647ea1b4b4ccc8c7f546d1481755669


Grab bag of things to try and speed up C++ tests

sort by cpu cost to enable better bin packing
refine qps test cost estimates
hand tweak a few test cpu cost estimations
revert gtest splitting: allows better estimation of costs

Jenkins: test this please

Jenkins: test this please

No significant performance differences

? what happened?


Massive increase of DutyCycle in a connected CCU/RaspberryMatic when used in combination with custom_homematic

custom_component/hahomematic version (if applicable):
0.36.3
Home Assistant version (if applicable):
core-2022.3.3
CCU version:

Model (CCU2/3, Raspi3/4, NUC, ProxmoxVE, ...): ProxmoxVE
OS (Original, RaspberryMatic, debmatic/...): HomeAssistantOS 7.4
Version:

Problem-relevant configuration:
Do you use tls?
No
Do you use callback?
No
Do you use username and password?
Yes
Which interfaces do you use?
BidCos-RF, homematicIP, homematicIP-Wired
Describe the bug
In case of a restart of HomeAssistant core or the whole HomeAssistantOS, or even in case of an update of the custom_homematicversion the queried CCU/RaspberryMatic instance receives quite a tremendous amount of device / data queries in a way that causes the CCU / RaspberryMatic device to generate real RF traffic so that the DutyCycle of the connected CCU / RaspberryMatic device is raised quite tremendously. Dependent on the actual number of RF devices connected to the CCU / RaspberryMatic device, this could easily result in a DutyCycle of 100% (e.g. here in case of two consecutive restarts of custom_homematic) so that no further RF communication is possible for that CCU / RaspberryMatic device
To Reproduce
Steps to reproduce the behavior:

Go to HomeAssistant -> Settings -> Devices -> HACS
Click on "..." and select "Reload"
Monitor the DutyCycle value on your connected CCU / RaspberryMatic device
Notice the substantial increase of DutyCycle while custom_homematic is querying all necessary device information.

Relevant HA log entries
n/a

Expected behavior
custom_homematic should use device query pathes / APIs that won't result in actual RF communication to be forced on the connected CCU / RaspberryMatic device so that the DutyCycle won't be increased at all.
Screenshots
n/a
Additional context
If wanted, I could personally have a look at the actual mechanisms which are currently used to query device information from within hahomematic and/or custom_homematic and then try to work out a solution which won't increase the DutyCycle anymore. However, for this to happen it would be great if someone (@SukramJ ?) could actually point his finger at the actual code passages in hahomematic or custom_homematic which are doing the device queries, so that alternative APIs or access pathes could be considered by me.

Ok Jens, let's see where this journey ends.
From my personal experience i cannot confirm this dramatic increase of the RMs duty cycle. My base DC is around 25% and a restart raises this value by 10-20%. I have around 120 devices connected which are in majority HMIP devices.
We use the getValue method to fetch device data. I also tested getParamset to reduce the number of api calls, but that was a major issue with bidcos and virtual devices.
On an initial startup we load all paramset descriptions with getParamsetDescription and cache them in a file, to avoid future api calls on restarts. Also the device descriptions are cached in a file.
Rooms and names are loaded by a call to the json api.
CONFIG_PENDING,  ERROR, STICKY_UN_REACH/UN_REACH, UPDATE_PENDING are loaded, if in the paramset_description  for every device once (values are cached). Maybe CONFIG_PENDING, ERROR and UPDATE_PENDING could be avoided on initial load.
All other values are loaded, if the HA entity is not disabled.
If a more complex entity like climate uses the parameter temperature, and there is an additional sensor for temperature then this value is also loaded only once.
Please come back, if you need more information.
Thanks for your support.

I created a version without CONFIG_PENDING, ERROR and UPDATE_PENDING.
you could test it by replacing
"requirements": ["hahomematic==0.36.3"],
by
"requirements": ["hahomematic==0.37.0b1"],
in the manifest in "HA config folder/custom_components/hahm"

From my personal experience i cannot confirm this dramatic increase of the RMs duty cycle. My base DC is around 25% and a restart raises this value by 10-20%. I have around 120 devices connected which are in majority HMIP devices.

Well, in my case I have about 200 devices with a baseline DC of about 30% and a restart of HACS/custom_integration raises the DutyCycle by about 30-35% which IMHO is quite substantial given the fact that other external applications like ioBroker or even Smartphone apps like pocketControl cause a 0% increase of DC due to how they query the current parameters.

We use the getValue method to fetch device data. I also tested getParamset to reduce the number of api calls, but that was a major issue with bidcos and virtual devices.

Well, the question here is really if hahomematic/custom_homematic should really exclusivly rely on xmlrpc communication with the CCU. The point is, that by using getValue() via xmlrpc this could actually be the main reason why there is any rf communication being forced when querying current device parameters. Without having digged into the code of hahomematic or having it compared to the methode ioBroker, etc. uses my guess is that it actually could be better to use e.g. the jsonrpc methods to query for the current values because these methods will then actually use the cached values on the CCU / RaspberryMatic rather than freshly querying them from the RF devices simply because you are asking for them using xmlrpc. But as said: haven't digged through the code yet, thus this is just my feeling here and comparing the negative effects seen in using hahomematic compared to ioBroker or other external apps querying for device parameters/values on a CCU / RaspberryMatic.

On an initial startup we load all paramset descriptions with getParamsetDescription and cache them in a file, to avoid future api calls on restarts. Also the device descriptions are cached in a file.
Rooms and names are loaded by a call to the json api.

Why don't you also try to load the current values and the paramset descriptions using the json api? As said, this should then allow to use the internally cached values rather than freshly querying them from the rf devices.

CONFIG_PENDING, ERROR, STICKY_UN_REACH/UN_REACH, UPDATE_PENDING are loaded, if in the paramset_description for every device once (values are cached). Maybe CONFIG_PENDING, ERROR and UPDATE_PENDING could be avoided on initial load. All other values are loaded, if the HA entity is not disabled. If a more complex entity like climate uses the parameter temperature, and there is an additional sensor for temperature then this value is also loaded only once.
Please come back, if you need more information. Thanks for your support.

Well, thanks for these first information. Perhaps you can point me more to the actualy code passages that are doing the xmlrpc queries and all the other different data queries during startup of hahomematic/custom_homematic. So perhaps explain the query procedure more in a step-by-step guide here with reference to the actual code so that I can review and debug them in my own test environment so that I can identify the actual query code that causes most of the DC increase. Or perhaps you already debug this and find out which queries are responsible for the DC increase. But as it stands, IMHO the current way of querying / initialising the device parameters is definitly not perfect yet, because - as said - other external applications like ioBroker, etc. are also not causing any noteable DC increase.

Ok, before writing lots of step-by-step guides, i will try using the json methods.
I wasn't aware, that these calls use cached values.
@danielperna84 Any arguments against json methods?
@jens-maus I'll come back in short.

@SukramJ
Well, it will break compatibility with Homegear, as there we only have XML-RPC. If it's reasonable to implement both mechanisms and use one or the other depending on the detected backend (like we're doing with the metadate already), then we could use JSON-RPC if it's not Homegear. I suspect Homegear installations not to be big as CCU users have them.
@jens-maus
As mentioned above, getValue via XML-RPC is required because we aim for compatibility with Homegear. And I'm 99% sure the problem is caused by calling that method. As SukramJ said, Paramsets are only fetched once initially and then stored in a cache. So I would rule out getParamsetDescription, which is the only other read-access done via XML-RPC if I'm correct.
So essentially this is what hahomematic is doing during startup:
from xmlrpc.client import ServerProxy
for host in ["http://<IP_ADDRESS>:2001", "http://<IP_ADDRESS>:2010"]:
    client = ServerProxy(host)
    devices = client.listDevices
    for device in devices:
        for channel in device:
            for param in channel:
                if not param.blacklisted:
                    # This is the problematic read-access to the XML-RPC API
                    value = client.getValue(device + channel, param)

Note: this is not code that actually works in this form. It's just to illustrate what is happening during startup.

I created a beta release 0.37.0b1 that should address the following things for CCU/RM users:

Use getParamset from JSON-RPC instead of getValue from XML-RPC for initial data load. This results in zero (?) raise of duty cycle when the system starts.
use getParamsetDescription from JSON-RPC instead of getParamsetDescription from XML-RPC. This works, but i need an extra converter for the output. Here i see no benefit as also the XML-RPC version uses cached information from CCU.

@jens-maus I have some questions:

Any reasons why using JSON-RPC should be prefered over XML-RPC for getParamsetDescription?
Is the CCU Cache persistet between restarts? I guess not.

If the cache is empty after a CCU restart, and HA is restarted directly afterwards, then the values are empty for a while.

I created a beta release 0.37.0b1 that should address the following things for CCU/RM users:
[...]

This sounds great. I will try to test it ASAP and report back if it really improves the DC handling. Do you have any easy description on how to force an update to 0.37.0b1 in an already installed HACS/cutsom_homematic installation?

@jens-maus I have some questions:

Any reasons why using JSON-RPC should be prefered over XML-RPC for getParamsetDescription?

Well, I am not 100% sure and would have to check it. As the getParamsetDescription function in the JSON-RPC api performs essentially the same like the getParamset, my guess would be that there the same caching would apply and thus should improve the overall handling rather than directly querying each interfaces via direct XML-RPC calls.

Is the CCU Cache persistet between restarts? I guess not.

Not it is not.

If the cache is empty after a CCU restart, and HA is restarted directly afterwards, then the values are empty for a while.

This should be essentially the very same like having the CCU restarted and looking into the WebUI. There, also in some cases (e.g. for homematicIP devices) values have default values until a battery driven homematic devices sends an update after some cyclic time. Thus, I would not overestimate the importance of that "limitation". More important is, that the DutyCycle of a CCU/RM will not be increased just because a restart of an external engine results in massive direct XMLRPC requests that end up in massive RF communication and thus an overload. As said, a smart home central is something that you would not restart daily or even regularly and "stablizes" only in mid/long term uptime.

Do you have any easy description on how to force an update to 0.37.0b1 in an already installed HACS/cutsom_homematic installation?

If you use hacs you could click on the integration and press "erneut herunterladen". Then you could also select the beta.

Simply buy a UPS to prevent power outages, I would suggest as a valid solution ;) Really: it is totally valid and normal that battery driven RF devices should and often cannot be actively queried. Thus, you have to make sure that your automations can deal with the default values as well until sich battery RF devices send new values with their next cycle!

I'm playing around with xmlrpc.getValue vs jsonrpc.getParamset a bit. From the point of view of data actuality after a HA restart, I don't see any differences. Also the xmlrpc.getValue method does not initially return any data. As already mentioned, I have almost exclusively HmIP devices.

Do you have any easy description on how to force an update to 0.37.0b1 in an already installed HACS/cutsom_homematic installation?

If you use hacs you could click on the integration and press "erneut herunterladen". Then you could also select the beta.

So, Now I could test the latest 0.37.0b1 beta version you referenced with my production environment at home here. While I could immediately spot an improvement in terms of raised dutycycle due to a restart of HomeAssistant or HACS or even HA core, I still spot an increase of about 15% of DutyCycle here.
So can you please perhaps elaborate on the exact API methods you are using one after another to query all the different device information on a restart of hahomematic/custom_homematic so that we can perhaps try to further optimize the data queries up to a level that would definitly not increase the DC like other external applications like ioBroker or pocketControl (smartPhone app)?

Start XmlRPC Server and register clients a CCU with XmlRPC (init)
First start only (No cached data) fetch all paramset descriptions with JsonRPC (Interface.getParamsetDescription) (NEW with 0.37.x)
On every HA startup:

fetch all data required for active HA entities with jsonRPC (Interface.getParamset). Results are cached for reuse. One call per used channel. (NEW with 0.37.x)
fetch device names with JsonRPC (Device.listAllDetail). one call
fetch rooms with JsonRPC (Room.getAll). one call

During HA runtime:

every minute alive check with  XmlRPC (ping)
every minute load system variables with JsonRPC (SysVar.getAll)

Updates are received by XmlRPC callbacks.

I created a new beta (0.37.0b2) that should further reduce the no of api calls.

I created a new beta (0.37.0b2) that should further reduce the no of api calls.

Thx. Any of these kind of changes are appreciated. However, I still notice a DC increase of about 18-20% upon restart of the integration. So no real change compared to 0.37.0b1 but definitly better than 0.36.0.
And thans for the details on the API calls used. I will try to invest some time in future to look more detailed into that in trying to identify the concrete API calls that are responsible for the rest of the DC increase. My current guess is, that we might try to think about replacing some of the JSONRPC or XMLRPC calls by queries to the ReGa-Skripting API on Port 8181 as the ReGaHss is more or less the centrale db for all homematic devices in a CCU and it uses massive caching to reduce RF communication to a minimum.

I created a new beta (0.37.0b3)  as an alternative to 0.37.0b2 with jsonrpc.getValue instead of jsonrpc.getParamset.
More api calls, but as already menioned, i had some issues with xmlrpc.getParamset.
With these json implementations i cannot see a HA restart on the DC diagram anymore.
Would is your mixture of HmIP/BidCos/Virtual devices?
A good test might be to disable interfaces to see if it is more related to HmIP, BidCos or VirtualDevices.
You could do this by clicking on "Konfigurieren" on the integration, and disable interfaces on the second page.

0.37.0b3 was just a quick test to sort out if the problem might be related to getParamset, which it is obviously not as much as getValue.

Installed 0.37.0 (which comes with hahomematic 0.37.1), but noticed the same slight DC increase like 0.37.0b2. Tried your suggestion to identify the interface which actually causes the DC increase and found out that as soon as I disable the BidCos-RF interface the 18-20% DC increase vanishes and that with just HmIP and VirtualDevices enabled I also cannot see any DC increase at all. So I will need to debug this further in seeing why exactly only with the BidCos-RF interface (rfd) a DC increase is happening and why this is only happening for hahomematic and not e.g. with ioBroker.
Still my current guess is, that this could also be some slight limitation of the rfd XMLRPC interface to some extend and that we would probably need to perform some data queries with ReGa scripting instead (ioBroker is actually doing this) - which in fact we could also do with the JSONRPC interface function ReGa.runScript instead of having to perform this using the explicit rega scripting port 8181. What do you think?
And thanks for the debugging possibilities. This really helps and I was also thinking about requesting improved debugging capabilities for hahomematic :). Only thing that you might probably improve/change in this respect is to make sure that the password for Session.login will not be logged verbatim in the debug log but the password being replaced/removed for security reasons.

So I will need to debug this further in seeing why exactly only with the BidCos-RF interface (rfd) a DC increase is happening and why this is only happening for hahomematic and not e.g. with ioBroker.

From my past experience i expected the problem with the BidCos devices. I'm curious what you find
We could give ReGa scripting a try, if this is the only solution, alltought i have no clue how to do it. So it is up to you to deliver the required code pieces.
ReGa.runScript seems to be a better option than using the scripting port 8181.
Is iobroker really loading all data at startup?

So I will need to debug this further in seeing why exactly only with the BidCos-RF interface (rfd) a DC increase is happening and why this is only happening for hahomematic and not e.g. with ioBroker.

From my past experience i expected the problem with the BidCos devices. I'm curious what you find

Haven't invested a lot of time yet (more to come) I just walked through the Homematic XMLRPC documentation and stumbled on the getParamset and getValue documentation (see chapter 4.2.4). I noticed that there is an extra mode parameter for getParamset and getValue which can be used during query of the BidCos-RF interface. And now I wonder if using this parameter might make any difference or at least reveal additional information (which can be added to the debug output) which devices were actively queried and for which devices cached values were used/retrieved. Perhaps you can have a further look at that and implement/test it to some extend?
And still I don't understand yet, why only the BidCos-RF interface seems to be affected by the DutyCycle increase if getParamset or getValue is used...

We could give ReGa scripting a try, if this is the only solution, alltought i have no clue how to do it. So it is up to you to deliver the required code pieces. ReGa.runScript seems to be a better option than using the scripting port 8181.

Definitly. If we really would need rega scripting here to further reduce the DC I think I can definitly assist here by writing/providing the necessary rega scripts for the data queries.

Is iobroker really loading all data at startup?

That's actually a very good question, which I would have to find out first (this is nothing I regularly have on my table). But perhaps @foxriver76 can assist on that question or provide the necessary links to iobroker code passages of the xmlrpc or rega scripting based data queries in ioBroker.hm-rpc or ioBroker.hm-rega?
At least in a quick search I could not spot any getParamset or getValue uses within main.ts of the ioBroker.hm-rpc adapter. So a question might be, why do you need to use getParamset or getValue altogether in hahomematic when, e.g., ioBroker is not using any of those but is also able to have all current device values upon startup. Or I am misinterpreting here something because I am not completly into this topic right now.

I think i already found something interesting https://github.com/ioBroker/ioBroker.hm-rega/blob/master/regascripts/datapoints.fn

I think i already found something interesting https://github.com/ioBroker/ioBroker.hm-rega/blob/master/regascripts/datapoints.fn

Funny. This is exactly what I would have suggested next, but first wanted to share my thoughts in my last comment. :) So yes, this kind of rega code snippet is able to retrieve all current datapoint values of all devices in the rega db.

@jens-maus
you can test it, if you change the requirement in manifest.json
"requirements": ["hahomematic==0.37.1b1"],

you can test it, if you change the requirement in manifest.json

Perfect. I just did it and now there is no DutyCycle increase at all upon restart of HomeAssistant. I can also spot the ReGa.runScript executions in the debug logfile. Didn't check if it correctly set/updated the device values upon restart (how can I debug this?).
In addition, I can see that you execute that ReGa script for every interface. But AFAICS this script has to be executed only once for all interfaces and not for every interface startup again. So perhaps you can change your internal logic in some way that it will only be executed once before all interface xmlrpc registrations.
Furthermore, using this ReGa.runScript together with some dedicated script can have more potential in future. For example, you could in principle now automatically identify which interfaces are available and then register them accordingly. So there might be no need anymore to switch BidCos-RF/HmIP/VirtualDevices on/off simply because you can fetch that information via ReGa scripting using the JSONRPC API.
But seeing that using ReGa scripting seems to work and does not raise the DutyCycle anymore is really great. Thanks for that and I hope you/we can make it even more smooth until this is available to the public.
BTW: Where can I see your 0.37.1b1 code modifications in hahomematic because I can't see these changes in the hahomematic respository yet?!?!

Perfect. I just did it and now there is no DutyCycle increase at all upon restart of HomeAssistant. I can also spot the ReGa.runScript executions in the debug logfile. Didn't check if it correctly set/updated the device values upon restart (how can I debug this?).

I don't want to add the script result to the debug output, but if a device has initial data then it is from the script.

In addition, I can see that you execute that ReGa script for every interface. But AFAICS this script has to be executed only once for all interfaces and not for every interface startup again. So perhaps you can change your internal logic in some way that it will only be executed once before all interface xmlrpc registrations.

It's currently a dirty hack. It's not executed per interface. It should be executed once initially and when newDevices is called.

Furthermore, using this ReGa.runScript together with some dedicated script can have more potential in future. For example, you could in principle now automatically identify which interfaces are available and then register them accordingly. So there might be no need anymore to switch BidCos-RF/HmIP/VirtualDevices on/off simply because you can fetch that information via ReGa scripting using the JSONRPC API.

Good idea, but there might be an issue if people use different ports.

But seeing that using ReGa scripting seems to work and does not raise the DutyCycle anymore is really great. Thanks for that and I hope you/we can make it even more smooth until this is available to the public.

Thanks for pointing me in that direction.

BTW: Where can I see your 0.37.1b1 code modifications in hahomematic because I can't see these changes in the hahomematic respository yet?!?!

The code is currently a fast and dirty hack. Now i will clean it up and push it to the repo.

Code can now be seen here

Can this issue be closed?
The DutyCycle issue is solved in latest release.
For further improvement ideas we should switch to new  discussions / feature requests

@SukramJ

Can this issue be closed? The DutyCycle issue is solved in latest release. For further improvement ideas we should switch to new discussions / feature requests

Regarding this issue, I can indeed report that the increase of DutyCycle due to the use of xmlrpc.getParamset/jsonrpc.getParamset or xmlrpc.getValue/jsonrpc.getValue has now vanished. I even tested the latest 0.37.3 version of custom_homematic and there is no DutyCycle increase anymore. So yes, this issue can probably be closed - thanks again.
However, perhaps you can indeed have a look at what @Baxxy13 stated here regarding warning and error messages which are appearing now in the /var/log/messages file within RaspberryMatic when reloading the custom_homematic integration.

I noticed that on restart of Home Assistant the Duty Cycle still goes up for me on 1.0.4.
Also I can see my HM Doorbell (HM-OU-CFM-TW) blinking red once on reboot of HA. Since it is not used in any automation or any other means in Home Assistant (spare device...)

I noticed that on restart of Home Assistant the Duty Cycle still goes up for me on 1.0.4.

Which CCU device are you using? External or using the RaspberryMatic Addon within HA?

Original CCU3 (=Raspi 3) flashed with RaspberryMatic (latest).


ICEndpoint is used as AccountInfo-Dataobject

Some methods expect account data and get an ICEndpoint instead. ICEndpoint does carry the needed information but it would be smarter to use the data directly for the call. Especially because the collection of the ICEndpoint data could be in a seperate method

Deprecated after ICAccountInfo was introduced