HDDS-11483. Make s3g object get and put operation buffer configuable

What changes were proposed in this pull request?

make s3g object read/write buffer configurable
change ozone.s3g.client.buffer.size default value from 4KB to 1MB

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11483
How was this patch tested?
wrap get benchmark test

before the patch, ozone.client.bytes.per.checksum = 16KB, s3 get buffer size 8KB

[root@vc0616 systest]# warp get --warp-client=vc0616.halxg.cloudera.com --host=localhost:9879 --tls --autoterm --insecure --access-key=$AWS_ACCESS_KEY_ID --secret-key=$AWS_SECRET_ACCESS_KEY --obj.size=20mb --concurrent=20 --bucket=bucket-20-24 --objects 10000 --duration=5m
warp: Running benchmark on all clients...
warp: Benchmark data written to "warp-remote-2024-09-20[053918]-cTQe.csv.zst"

----------------------------------------
Operation: PUT. Concurrency: 20
* Average: 358.85 MiB/s, 18.81 obj/s

Throughput, split into 104 x 5s:
 * Fastest: 452.0MiB/s, 23.70 obj/s
 * 50% Median: 360.3MiB/s, 18.89 obj/s
 * Slowest: 267.6MiB/s, 14.03 obj/s

----------------------------------------
Operation: GET. Concurrency: 20
* Average: 224.64 MiB/s, 11.78 obj/s

Throughput, split into 294 x 1s:
 * Fastest: 262.4MiB/s, 13.76 obj/s
 * 50% Median: 230.3MiB/s, 12.08 obj/s
 * Slowest: 49.0MiB/s, 2.57 obj/s
warp: Cleanup done.

with the patch,  ozone.client.bytes.per.checksum = 16KB, s3 get buffer size 1MB

[root@vc0616 systest]# warp get --warp-client=vc0616.halxg.cloudera.com --host=localhost:9879 --tls --autoterm --insecure --access-key=$AWS_ACCESS_KEY_ID --secret-key=$AWS_SECRET_ACCESS_KEY --obj.size=20mb --concurrent=20 --bucket=bucket-24-10-56 --objects 10000 --duration=5m
warp: Running benchmark on all clients...

warp: Benchmark data written to "warp-remote-2024-09-23[201117]-qHUD.csv.zst"

----------------------------------------
Operation: PUT. Concurrency: 20
* Average: 349.16 MiB/s, 18.31 obj/s

Throughput, split into 107 x 5s:
 * Fastest: 441.5MiB/s, 23.15 obj/s
 * 50% Median: 351.3MiB/s, 18.42 obj/s
 * Slowest: 259.9MiB/s, 13.63 obj/s

----------------------------------------
Operation: GET. Concurrency: 20
* Average: 1002.26 MiB/s, 52.55 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1064.7MiB/s, 55.82 obj/s
 * 50% Median: 1041.5MiB/s, 54.60 obj/s
 * Slowest: 59.2MiB/s, 3.10 obj/s
warp: Cleanup done.

3.  ozone.client.bytes.per.checksum = 1MB, s3 get buffer size 8KB
[root@vc0616 systest]# warp get --warp-client=vc0616.halxg.cloudera.com --host=localhost:9879 --tls --autoterm --insecure --access-key=$AWS_ACCESS_KEY_ID --secret-key=$AWS_SECRET_ACCESS_KEY --obj.size=20mb --concurrent=20 --bucket=bucket-19-23 --objects 10000 --duration=5m
warp: Client <IP_ADDRESS>:7761: Requested stage prepare start...
warp: Benchmark data written to "warp-remote-2024-09-20[044056]-999v.csv.zst"

----------------------------------------
Operation: PUT. Concurrency: 20
* Average: 371.36 MiB/s, 19.47 obj/s

Throughput, split into 100 x 5s:
 * Fastest: 433.5MiB/s, 22.73 obj/s
 * 50% Median: 378.1MiB/s, 19.82 obj/s
 * Slowest: 255.2MiB/s, 13.38 obj/s

----------------------------------------
Operation: GET. Concurrency: 20
* Average: 980.89 MiB/s, 51.43 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1093.4MiB/s, 57.33 obj/s
 * 50% Median: 1040.7MiB/s, 54.56 obj/s
 * Slowest: 31.8MiB/s, 1.67 obj/s
warp: Cleanup done.

heap space consideration

according to this link, the max threads of jetty is 200.  So there is max 200 get/put requests handled concurrently, each request by default 1MB, totally 200MB, which I think is acceptable.

@ChenSammi  Hi, based on your testing conclusions, we understand that changing the parameter ozone.s3g.client.buffer.size to 1M significantly improves performance. However, does this size have a relationship with the file sizes I need to read? Should I set this parameter to 1M, 2M, or even larger? How can I find the optimal value? Is there any guideline for setting this parameter that I can refer to?

@ChenSammi Hi, based on your testing conclusions, we understand that changing the parameter ozone.s3g.client.buffer.size to 1M significantly improves performance. However, does this size have a relationship with the file sizes I need to read? Should I set this parameter to 1M, 2M, or even larger? How can I find the optimal value? Is there any guideline for setting this parameter that I can refer to?

@weimingdiit , there is a previous patch #6331 which changes the ozone.client.bytes.per.checksum default from 1MB to 16KB. After that, it's found that the s3g read performance is degraded by 4X.  This patch tries to fix the problem by increase the buffer size in s3g.  If your cluster is still using 1MB ozone.client.bytes.per.checksum configuration, then you don't need to bother to apply this patch.  Either 1MB ozone.client.bytes.per.checksum or 1MB s3g read buffer size will have the same read performance.
For s3g, theoretically, bigger buffer size will have better performance. But the margin performance gain decreases with the buffer size increased.  Here is some data of same "warp get" test,

1MB s3g buffer, the result is "Average: 1002.26 MiB/s, 52.55 obj/s"
128KB s3g buffer, the result is "Average: 757.84 MiB/s, 39.73 obj/s"
64KB s3g  buffer, the result is "Average: 550.13 MiB/s, 28.84 obj/s"
16KB, the result is "Average: 224.64 MiB/s, 11.78 obj/s"

If you want to find the optimal value, you can try test the different s3g buffer size values, for example, 2MB, 4MB etc. But bigger buffer size will cost more heap. Make sure there is enough heap configured for s3g.

Thanks for bringing this up! Our implementation for S3G read needs improvement. ozone.s3g.client.buffer.size should always be minimum of chunk size if the object is larger than a chunk. This will need configuring the web app used to implement S3 to also use larger buffer than 1MB. The additional heap memory used is trivial in comparison to 4 round trips that the client will need to make when reading a single chunk. This will need configuring the web app used to implement S3 to also use larger buffer than 1MB. Alternatively we can implement  https://issues.apache.org/jira/browse/HDDS-11487

Related PR: https://github.com/apache/ozone/pull/6613

@ChenSammi Hi, based on your testing conclusions, we understand that changing the parameter ozone.s3g.client.buffer.size to 1M significantly improves performance. However, does this size have a relationship with the file sizes I need to read? Should I set this parameter to 1M, 2M, or even larger? How can I find the optimal value? Is there any guideline for setting this parameter that I can refer to?

@weimingdiit , there is a previous patch #6331 which changes the ozone.client.bytes.per.checksum default from 1MB to 16KB. After that, it's found that the s3g read performance is degraded by 4X. This patch tries to fix the problem by increase the buffer size in s3g. If your cluster is still using 1MB ozone.client.bytes.per.checksum configuration, then you don't need to bother to apply this patch. Either 1MB ozone.client.bytes.per.checksum or 1MB s3g read buffer size will have the same read performance.
For s3g, theoretically, bigger buffer size will have better performance. But the margin performance gain decreases with the buffer size increased. Here is some data of same "warp get" test,

1MB s3g buffer, the result is "Average: 1002.26 MiB/s, 52.55 obj/s"
128KB s3g buffer, the result is "Average: 757.84 MiB/s, 39.73 obj/s"
64KB s3g  buffer, the result is "Average: 550.13 MiB/s, 28.84 obj/s"
16KB, the result is "Average: 224.64 MiB/s, 11.78 obj/s"

If you want to find the optimal value, you can try test the different s3g buffer size values, for example, 2MB, 4MB etc. But bigger buffer size will cost more heap. Make sure there is enough heap configured for s3g.

@ChenSammi  Thank you for your explanation; it is very detailed and clear.

Currently we are writing 1 MB chunks from S3. This change LGTM

@ChenSammi Hi, based on your testing conclusions, we understand that changing the parameter ozone.s3g.client.buffer.size to 1M significantly improves performance. However, does this size have a relationship with the file sizes I need to read? Should I set this parameter to 1M, 2M, or even larger? How can I find the optimal value? Is there any guideline for setting this parameter that I can refer to?

@weimingdiit , there is a previous patch #6331 which changes the ozone.client.bytes.per.checksum default from 1MB to 16KB. After that, it's found that the s3g read performance is degraded by 4X. This patch tries to fix the problem by increase the buffer size in s3g. If your cluster is still using 1MB ozone.client.bytes.per.checksum configuration, then you don't need to bother to apply this patch. Either 1MB ozone.client.bytes.per.checksum or 1MB s3g read buffer size will have the same read performance.
For s3g, theoretically, bigger buffer size will have better performance. But the margin performance gain decreases with the buffer size increased. Here is some data of same "warp get" test,

1MB s3g buffer, the result is "Average: 1002.26 MiB/s, 52.55 obj/s"
128KB s3g buffer, the result is "Average: 757.84 MiB/s, 39.73 obj/s"
64KB s3g  buffer, the result is "Average: 550.13 MiB/s, 28.84 obj/s"
16KB, the result is "Average: 224.64 MiB/s, 11.78 obj/s"

If you want to find the optimal value, you can try test the different s3g buffer size values, for example, 2MB, 4MB etc. But bigger buffer size will cost more heap. Make sure there is enough heap configured for s3g.

Good！
We still used ozone.client.bytes.per.checksum=1M, the effect is the same as increasing ozone.s3g.client.buffer.size.
BTW, I verified that the effect of this PR was as expected, see the test logs: link

Currently we are writing 1 MB chunks from S3. This change LGTM

This is true only for docker dev environment, we are creating 4 MB chunks.

HDDS-11483 s3g ozone.s3g.client.buffer.size test.pdf
All data are collected within same physical cluster.

HDDS-11483 s3g ozone.s3g.client.buffer.size test.pdf
All data are collected within same physical cluster.

Do you have the GC numbers? Also, this is a single S3G talking to a cluster, when we have multiple S3Gs talking to a cluster, buffer size being less than chunk size would imply that the datanodes are getting 4x the read chunk rpc rate. I did a test locally and the main reason 4 MB is slower is due to GC load. Bumping the heap the difference is minimal and the write performance goes up. Also, if we want to go with 1MB we should bump up heap and run the a cluster wide test and record Datanode disk load.
1 MB buffer and checksum size 1 MB
----------------------------------------
Operation: PUT. Concurrency: 20
* Average: 331.11 MiB/s, 86.80 obj/s

Throughput, split into 20 x 1s:
 * Fastest: 400.7MiB/s, 105.04 obj/s
 * 50% Median: 337.7MiB/s, 88.53 obj/s
 * Slowest: 166.4MiB/s, 43.61 obj/s

----------------------------------------
Operation: GET. Concurrency: 20
* Average: 1079.56 MiB/s, 283.00 obj/s

Throughput, split into 299 x 1s:
 * Fastest: 1099.5MiB/s, 288.24 obj/s
 * 50% Median: 1083.0MiB/s, 283.91 obj/s
 * Slowest: 758.6MiB/s, 198.87 obj/s

4 MB and checksum size 1 MB
----------------------------------------
Operation: PUT. Concurrency: 20
* Average: 388.16 MiB/s, 101.75 obj/s

Throughput, split into 23 x 1s:
 * Fastest: 485.1MiB/s, 127.16 obj/s
 * 50% Median: 430.4MiB/s, 112.83 obj/s
 * Slowest: 185.2MiB/s, 48.56 obj/s

----------------------------------------
Operation: GET. Concurrency: 20
* Average: 992.53 MiB/s, 260.19 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1058.5MiB/s, 277.47 obj/s
 * 50% Median: 997.8MiB/s, 261.57 obj/s
 * Slowest: 662.1MiB/s, 173.57 obj/s

@kerneltime , thanks for the new data. As 4MB can improve the s3g write performance, it's make sense to make it default.
Could you help to review the patch again?

Thanks @ChenSammi for the patch, @kerneltime, @weimingdiit, @whbing for the review.

Thank you @adoroszlai @kerneltime @weimingdiit @whbing for the review.


ionic 3 after selecting photo or take photo, actionsheet or fab buttons push down 20px

I'm submitting a ...  (check one with "x")
[x] bug report
Current behavior:
Once click on a fab camera button, either selecting existing photo from the gallery or take a new photo, fab button keeps pushing down like 20px.
Expected behavior:
It should not push fab button or actionsheet controller down.

Ionic info: (run ionic info from a terminal/cmd prompt and paste output below):
cli packages: (/private/var/www/experiment/MotionscloudWebrtc/node_modules)

    @ionic/cli-utils  : 1.19.2
    ionic (Ionic CLI) : 3.20.0

global packages:

    cordova (Cordova CLI) : 6.5.0

local packages:

    @ionic/app-scripts : 1.3.7
    Cordova Platforms  : android 6.1.2 ios 4.3.1
    Ionic Framework    : ionic-angular 3.3.0

System:

    Android SDK Tools : 26.1.1
    ios-deploy        : 1.9.2
    ios-sim           : 5.0.4
    Node              : v9.10.1
    npm               : 5.6.0
    OS                : macOS High Sierra
    Xcode             : Xcode 9.4.1 Build version 9F2000

no update on this?


feat: allow to choose symmetric or asymmetric withdraw from thorchain LP

Description
We want to allow the users to choose which asset they will receive when withdrawing from LPs.
There is a limitation on thorchain side where asymmetricaly deposited pools can't withdraw in a symmetrical way, so I disabled the symmetrical deposit with a tooltip on such opportunities.
The idea behind this PR is to customize the opportunityType depending on the radio button clicked, but keep the opportunityId as it is, and then populate the memo depending on which side the users chosen.
If you don't push any :ASSET part to the memo, it will be be a symmetric withdraw.

Pull Request Type

[ ] :bug: Bug fix (Non-breaking Change: Fixes an issue)
[ ] :hammer_and_wrench: Chore (Non-breaking Change: Doc updates, pkg upgrades, typos, etc..)
[x] :nail_care: New Feature (Breaking/Non-breaking Change)

Issue (if applicable)

closes #6680
Risk
High risk as it could cause issues on LP deposits and withdraws

High Risk PRs Require 2 approvals

What protocols, transaction types or contract interactions might be affected by this PR?

Testing

Try to deposit into a pool (symmetric)
Try to deposit into a pool (asymmetric)
Try to withdraw from a pool (symmetric)
Try to withdraw from a pool (asymmetric)

Verify that the asym button is disabled on sym pool and enabled on asym pool!

Engineering
n/a

Operations
n/a

Screenshots (if applicable)

Rune withdraw: https://viewblock.io/thorchain/tx/B490B382978C1AC6779C17879450BF58A1E4C47D39B37051DFED515BFD048D42
Sym withdraw ETH: https://viewblock.io/thorchain/tx/4D14ECB93F9B052CB4567878092E499A640E82E4ACA4A9ADB7E9709ABDA0F0C0
Avax withdraw: https://viewblock.io/thorchain/tx/4f2b4a5f6ebbb00d85c82e33f75588eb12080abd114ba4821a5c32205c4ff203

This is pretty hard to understand, it seems to work as expected on other pools: https://viewblock.io/thorchain/tx/7F2B6DBFD9A72D3D47BF9B83885A5DC1E8A7C55E6711423651E3367F36D8CA65
https://viewblock.io/thorchain/tx/01eb5efc083fc795c24daaca955fec3b66189a84b7a603a15097a7ea5b0223d2
I feel like you don't have enough BTC in that pool to withdraw this amount, I have the same problem with a pool in ATOM, definitely something that can happen even sym I guess

This is pretty hard to understand, it seems to work as expected on other pools: https://viewblock.io/thorchain/tx/7F2B6DBFD9A72D3D47BF9B83885A5DC1E8A7C55E6711423651E3367F36D8CA65 https://viewblock.io/thorchain/tx/01eb5efc083fc795c24daaca955fec3b66189a84b7a603a15097a7ea5b0223d2
I feel like you don't have enough BTC in that pool to withdraw this amount, I have the same problem with a pool in ATOM, definitely something that can happen even sym I guess

Looks like this is a regression, comparing to prod it seems the insufficient funds for gas protection is no longer working.
Prod:

This PR:

This is pretty hard to understand, it seems to work as expected on other pools: https://viewblock.io/thorchain/tx/7F2B6DBFD9A72D3D47BF9B83885A5DC1E8A7C55E6711423651E3367F36D8CA65 https://viewblock.io/thorchain/tx/01eb5efc083fc795c24daaca955fec3b66189a84b7a603a15097a7ea5b0223d2
I feel like you don't have enough BTC in that pool to withdraw this amount, I have the same problem with a pool in ATOM, definitely something that can happen even sym I guess

Looks like this is a regression, comparing to prod it seems the insufficient funds for gas protection is no longer working. Prod:
This PR:

Mmh, why don't we see the withdraw type selector in the borked screenshot?

I think your screen doesn't reflect this branch :(

Something that can improve clarity imo would be to distinguish opportunityType from intentType. It is a bit confusing now to know if the opportunityType is referencing the position (opportunity) type, or the intent type in the case of withdrawals


ValueError: Could not extract json string from output

from llama_index.core.node_parser import MarkdownElementNodeParser
node_parser = MarkdownElementNodeParser(llm=Settings.llm, num_workers=8)
nodes = node_parser.get_nodes_from_documents(documents)
llm = mistral
ValueError: Could not extract json string from output: {
"summary": "As of December 31, 2021 and March 31, 2022, the company's assets include cash and cash equivalents, restricted cash and cash equivalents, accounts receivable, prepaid expenses, investments, equity method investments, property and equipment, operating lease right-of-use assets, and intangible assets. Liabilities include accounts payable, short-term insurance reserves, operating lease liabilities (current and non-current), long-term insurance reserves, long-term debt, and other long-term liabilities. Non-controlling interests are also reported. As of the end of the periods, total assets were $38,774 and $32,812, and total equity was $15,145 and $9,613, respectively.",
"table_title": "Balance Sheet",
"table_id": "id_16",
"columns": [
{
"col_name": "Assets",
"col_type": "string",
"summary": "As of December 31, 202

update to the latest version, this resolved the error for me. I'm guessing an API change.

didn't work for me with llama-parse 0.4. Which library do we need to update?

Same here, I'm on version 0.4.1 using a quantized Llama2-7B. Was following the tutorial for llama-parse from https://medium.com/@yu-joshua/using-llamaparse-for-knowledge-graph-creation-from-documents-3bd1e1849754 loading a quarterly report pdf from META.

Facing the same issue

It seems to be fixed now. I just ran this code successfully:
from llama_index.llms.mistralai import MistralAI
from llama_index.core.node_parser import MarkdownElementNodeParser
from llama_parse import LlamaParse

llm = MistralAI()
node_parser = MarkdownElementNodeParser(llm=llm, num_workers=8)
pdf_file_name = './insurance.pdf'
documents = LlamaParse(result_type="markdown").load_data(pdf_file_name)
nodes = node_parser.get_nodes_from_documents(documents)
print(nodes)


Add a mechanism to define plugins

This PR builds upon the bugfix of https://github.com/mapproxy/mapproxy/pull/577 (first commit)
The scope of this PR is best explained by the new documentation page: https://github.com/rouault/mapproxy/blob/plugins/doc/plugins.rst
It can also be quickly demo'ed with:
python3 -m venv myvenv
source myvenv/bin/activate
pip install git+https://github.com/rouault/mapproxy@plugins
pip install git+https://github.com/rouault/mapproxy_hips

git clone https://github.com/rouault/mapproxy_hips
mapproxy-util serve-develop mapproxy_hips/hips_examples/ogc_as_hips/mapproxy.yaml &
curl http://localhost:8080/hips/mars_tiled_geodetic/properties

A heads-up: I am not a committer nor reviewer, but a bit dormant PSC-member, sorry. From scanning the code in this PR, I think this is a major step forward for the project! (future Vector Tile plugin?). I hope someone with inside architectural knowledge can review. Or we can make Evan committer, as he being a developer of stature...

Same heads up here, I am sure we'll be giving this a proper review.

Code looks good to me! i also added just minor comments on the documentation part.
My tests did work like expected.
Thanks for your contributions so far!


Global menu

Would it be possible to come up with a better ui for the global menu functionality?
The current multilevel menu is prone to mishovering/misclicking and thus dangerous in the sense that it is too easy to end up modifying the wrong cluster.

Useful
Info

Versions
Rancher v2.3.3 UI: v2.3.25

Route
global-admin.accounts.index

Also, the menu is not possible to navigate with keyboard-nav extensions like Saka keys. If you make changes to it, please test it for accessibility at the same time.

There is a feature request that making the current cluster be the first item in the drop down. Not sure if it will help.  @sdahlbac

@loganhz it might be a slight improvement, but IMO it needs a bigger re-thinking.

Not that it matters that much, but current layout just does not work on mobile (iOS 13). Default/System menus are overlapped by some other dropdown div and user is locked out of... well, everything.

Thanks @toxuin Filed a new issue for it https://github.com/rancher/rancher/issues/27183


FishEye in Sigma 2?

Hi, I haven't found any implementation of FishEye in the new Sigma. Or have I missed something? Thanks!

Hi, I'd also really like to see this in the new Sigma. I'd be prepared to write it myself for the canvas renderer if I knew how to start ,,, can anyone give me some directions? Thanks!

anyone knows if this has been implemented?


a bug with security manager

I am using the newest version 1.2 of superset,i download it via git clone.
recently i have been having a bug with security manager, it could happen anytime,but basically whenever i am doing something related to role control, i would be sent to an empty page says
{"errors": [{"message": "'SupersetSecurityManager' object has no attribute 'get_role_permissions'", "error_type": "GENERIC_BACKEND_ERROR", "level": "error", "extra": {"issue_codes": [{"code": 1011, "message": "Issue 1011 - Superset encountered an unexpected error."}]}}]}
this could happen anytime, either when i am logged in and try to click any dataset or when i successfully get the dataset for a while and then click it again the error pops up
i think this might be sth related to role control but I am not sure, i tried to search for the definition of security_manager in the source code but i have not found it yet.
while i am logged in as an administrator, i am not able to delete any datasets as it says 'There was an issue deleting: Unicode Cloud'
Expected results
get datasets or delete datasets
Actual results
sent to an empty page says the bug
Screenshots
superset_app             | 2021-05-07 12:42:58,406:ERROR:superset.views.base:'SupersetSecurityManager' object has no attribute 'get_role_permissions'
superset_app             | Traceback (most recent call last):
superset_app             |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
superset_app             |     rv = self.dispatch_request()
superset_app             |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
superset_app             |     return self.view_functionsrule.endpoint
superset_app             |   File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps
superset_app             |     return f(self, *args, **kwargs)
superset_app             |   File "/app/superset/utils/log.py", line 228, in wrapper
superset_app             |     value = f(*args, **kwargs)
superset_app             |   File "/app/superset/views/core.py", line 803, in explore
superset_app             |     "user": bootstrap_user_data(g.user, include_perms=True),
superset_app             |   File "/app/superset/views/utils.py", line 77, in bootstrap_user_data
superset_app             |     roles, permissions = get_permissions(user)
superset_app             |   File "/app/superset/views/utils.py", line 94, in get_permissions
superset_app             |     permissions_ = security_manager.get_role_permissions(role)
superset_app             |   File "/usr/local/lib/python3.7/site-packages/werkzeug/local.py", line 347, in getattr
superset_app             |     return getattr(self._get_current_object(), name)
superset_app             | AttributeError: 'SupersetSecurityManager' object has no attribute 'get_role_permissions'
How to reproduce the bug
I actually do not know how to produce this, but even if i do 'rm -rf superset' and 'rm -rf .git' then re-clone the repo it would happen again
Environment
(please complete the following information):

superset version: v1.2
python version:
node.js version:

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[ chekced ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[ chekced  ] I have reproduced the issue with at least the latest released version of superset.
[ chekced  ] I have checked the issue tracker for the same issue and I haven't found one similar.

docker-compose down and up again does not work?

sadly it does not,
one weird thing is that, this thing happens much more often when i am using docker-compose up rather than
docker-compose -f docker-compose-non-dev.yml up

Please consider the following :
1- cd to the superset directory
2- docker exec -it <app_container_name> bash
3- pip install Flask-AppBuilder --upgrade
This should work .

thanks for the reply, would try it out!

{"errors": [{"message": "'SupersetSecurityManager' object has no attribute 'get_role_permissions'", "error_type": "GENERIC_BACKEND_ERROR", "level": "error", "extra": {"issue_codes": [{"code": 1011, "message": "Issue 1011 - Superset encountered an unexpected error."}]}}]}

docker-compose down and up again does not work?

sadly it does not,
one weird thing is that, this thing happens much more often when i am using docker-compose up rather than
docker-compose -f docker-compose-non-dev.yml up

Please consider the following :
1- cd to the superset directory
2- docker exec -it <app_container_name> bash
3- pip install Flask-AppBuilder --upgrade
This should work .

sadly it comes back with the same error:
{"errors": [{"message": "'SupersetSecurityManager' object has no attribute 'get_role_permissions'", "error_type": "GENERIC_BACKEND_ERROR", "level": "error", "extra": {"issue_codes": [{"code": 1011, "message": "Issue 1011 - Superset encountered an unexpected error."}]}}]}
i think it may be related to something like cache? im not sure, but i have installed and uninstalled the superset  directory a couple times just to get it works, but when it is finally functional it  stucks with this error again.

yet another thing i feel SUPER-WEIRD about is that , even if i copy paste the whole docker-compose-nondev.ymp thing to docker-compose.yml, and then start the docker containers using : docker-compose up, i would still get to the same error!


Increment package version after release of azure_core_tracing_opentelemetry

Increment package version after release of azure_core_tracing_opentelemetry

Closing since it got updated to 1.0.0b2 manually for this release


Dictionary view in Jadite should have way to expose transient symbol list

If you actually do put symbol dictionaries into the transient symbol list, they cannot be browser using the System Browser ... it should be possible to choose to view the transient symbol dictionary. See Issue #212 for a tangible use case

As of 3.0.73, Jadeite will use Rowan image symbolList exclusively.


Constructor form displaying fields for other contracts

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
When it comes to the problem with the constructor, it seems like the form is not updating correctly. After deploying the contracts, when I select a new contract, the constructor field stays there. See the screenshots.

Before deploying the first contract with constructor:

After deploying it:

Expected behavior
The constructor form shows correct input fields corresponding to the selected contract.
Starknet Remix Plugin details

default plugin from the marketplace

Thanks for the issue @kwkr

Added a PR in the repo that is causing this.

Done! Thank you for the PR 🎉🙏


Added 'Client-ID' label on Authorization header to match imgur's api …

…changes
The imgur API now requires a Client-ID label before the client ID.
Authorization: Client-ID YOUR_CLIENT_ID
I have added that label to prevent the user from getting a status 403 malformed auth header error when uploading an image.

:+1: @dianafaye17 thanks — released as v0.2.0.

Thanks! :-)
On Wed, Nov 4, 2015 at 3:51 PM, Adam Timberlake<EMAIL_ADDRESS>wrote:

[image: :+1:] @dianafaye17 https://github.com/dianafaye17 thanks —
released as v0.2.0.
—
Reply to this email directly or view it on GitHub
https://github.com/Wildhoney/ngImgur/pull/5#issuecomment-153878567.


Use compute-cluster-name in kuberentes pod label

Changes proposed in this PR

Use the compute cluster name in Kubernetes cook pod label

Why are we making these changes?
Makes it less likely for cook clusters to stomp on each other.

@pschorf Don't we need to add compute-cluster-name to the call here:
https://github.com/twosigma/Cook/pull/1204/files#diff-bc229d7ae30cc7a52880c221f9438017R137

That link didn't really work, it's the call to initialize-pod-watch inside the finally block


README: better document questions and react children APIs

Even though i'm not really happy about the type/key API situation, i think questions and react API deserve a bit more doc.

Yes, thanks again!
Regarding the input types, I'm also not finding adding the types to spans the best approach. We're still in the beginning phases, so we can explore different options to make sure we choose the best one down the road.

maybe the for keyword from the HTML form spec is what we want ? (used in <label for="inputName">)
or we have the name keyword too, (used in <input name="inputName"/>)

What about doing this:
<EurekaForm.TelInput />
<EurekaForm.EmailInput />
<EurekaForm.FileInput />

All those components would extend EurekaForm.Input and inherit common properties, and offer some type specific functionality as well if need be

As a matter of fact I had not seen those comments :)
So right now,  is your <EurekaForm.Input>, I feel we'd still want to have a type key to handle all that are really just HTML5 inputs (that could grow) and have a collection of more complex inputs, so you can have TypeAheadQuestion with type="email" or a MapQuestion with type="tel" (where you'd select the country on a map and type the number, poor ui but it's just to make the point)
Right now I'm needing a kind of TypeAheadMapQuestion where you'd select a complex object filtering by geography and name from a known list.


Add support for dynamic styling similar to L.geoJSON/Leaflet.VectorGrid

Now, this feature is available on v1.1.0.

Now, this feature is available on v1.1.0.

Thank you @iamtekson


change(log): Report compiler version and Zebra features when starting Zebra

Motivation
We want to report the compiler version when Zebra panics, and when it starts up. This helps us diagnose some bugs.
As part of this change, I updated the vergen version, because it changes how the compiler version works.
Close #5422
Close #6421
Specifications
API reference:
https://docs.rs/vergen/latest/vergen/index.html#usage
Complex Code or Requirements
This might break some of Zebra's existing version-handling code, which expects a semver-compliant version number.
It might also cause issues in PR #6601, which manually parses the version number.
Solution

Add the compiler version and compiler date (for nightly compilers) to logs and panics

Related changes:

Add a list of enabled Zebra features to logs and panics
Upgrade to vergen version 8 (ticket #6421)

Rewrite the vergen build script
Remove outdated disabled reproducible build code, vergen now has an env var to activate it
Replace the cargo profile with debug checks and optimization level
Change some environmental variable names

Unrelated changes:

Fix a clippy lint in the tests on nightly

Review
This is a routine change.
Reviewer Checklist

[ ] Will the PR name make sense to users?

[ ] Does it need extra CHANGELOG info? (new features, breaking changes, large changes)

[ ] Are the PR labels correct?
[ ] Does the code do what the ticket and PR says?

[ ] Does it change concurrent code, unsafe code, or consensus rules?

[ ] How do you know it works? Does it have tests?

We have 3 different alternatives for checking the git metadata, I'm just doing some builds to check which one is the fastest to build. (It's probably the C library dependency, unfortunately!)

Using gitoxide adds 80 new crate dependencies to Zebra's build (596 -> 675), which increases our security risk a bit.
So that's another potential issue.

main branch:
real    2m18.323s
user    47m53.429s
sys     2m22.332s
This PR with gitoxide dependency:
real    2m22.013s
user    48m26.231s
sys     2m21.810s
33 seconds more user compile time
This PR with git2 dependency:
real    2m19.301s
user    47m48.147s
sys     2m21.963s
5 seconds less user compile time

Manual testing:
Before this PR, Zebra's startup logs have:
$ zebrad
2023-05-04T00:05:38.784742Z  INFO zebrad::application: Diagnostic Metadata:
version: 1.0.0-rc.7+30.g937d704
Zcash network: Mainnet
state version: 25
branch: report-compiler-version
git commit: 937d704
commit timestamp: 2023-05-03T23:14:22Z
target triple: x86_64-unknown-linux-gnu
build profile: release
2023-05-04T00:05:38.784744Z  INFO zebrad::application: loaded zebrad config ...

After this PR, they have:
$ zebrad
2023-05-04T01:59:18.728120Z  INFO zebrad::application: Diagnostic Metadata:
version: 1.0.0-rc.7-37+gedf51d6.dirty
Zcash network: Mainnet
state version: 25
features: default,getblocktemplate_rpcs,metrics_exporter_prometheus,prometheus,release_max_level_info
branch: report-compiler-version
git commit: 0623ba6
commit timestamp: 1980-01-01T00:00:00.000000000Z
target triple: x86_64-unknown-linux-gnu
rust compiler: 1.71.0-nightly
rust release date: 2023-05-02
optimization level: 3
debug checks: true
2023-05-04T01:59:18.728122Z  INFO zebrad::application: loaded zebrad config ...

The fields removed by vergen 8 are:
build profile: release

The new fields are:
features: default,getblocktemplate_rpcs,metrics_exporter_prometheus,prometheus,release_max_level_info
branch: report-compiler-version
commit timestamp: 1980-01-01T00:00:00.000000000Z
rust compiler: 1.71.0-nightly
rust release date: 2023-05-02
optimization level: 3
debug checks: true

The changed fields are:
version: 1.0.0-rc.7+30.g937d704.modified
to
version: 1.0.0-rc.7-37+gedf51d6.dirty

But in most builds they are unchanged:
version: 1.0.0-rc.7-37+gedf51d6
or
version: 1.0.0-rc.7

I'm going to bump this down to low priority because it doesn't need to go in the upcoming release.


Allow  override --testNamePattern

Description
I want to run some test based on the test case description but for now it was impossible to override the testNamePattern parameter. Would you like to provide this, please?
Steps to Reproduce

Detox, Node, Device, Xcode and macOS Versions

Detox: ^8.0.0
React Native: 0.55.4
Node: v10.5.0
Device: ios
Xcode:
macOS:

Device and verbose Detox logs

can you write package.json? @amrew

i decided to create another bash file to replace detox test command.
then i can fill --testNamePattern param with my custom string.
This is why i can't override the parameter value.
https://github.com/wix/detox/blob/master/detox/local-cli/detox-test.js#L135

I suggest adding an args passthrough feature for  detox test ... -- --testNamePattern similar to npm run <script> -- --args-to-pass-through. @rotemmiz , does it make sense for you? If yes, then pull requests are welcome.

I think that this is still a relevant issue for Detox CLI, just needs rephrasing and volunteers.

Fixed in<EMAIL_ADDRESS>


Improve docfx_project/examples/LogCDPCommunication.md

Explain how would you like this document to be impoved

Explain how would you like this document to be improved

How?


Please implement the "Patreon" plugin.

There seem to be a lot of clients (users, modpack creators) that have outputs missing termination (money they want to pay you).   And I quote:

Stop nagging us, or start paying us to quit our day-jobs and do this full-time.

I suggest some kind of interface that is called on a schedule, say once a month:
public interface DeveloperSupport extends Patreon {
    @Override
    Pizza feedMe(Money benjamins);
    default void moralSupport(Accolades lotsalove) {
        /* Changed argument type to Accolades.  Clients still keep
         * trying to call this with Aggrevation.  I mean, they both
         * begin with "A",  but..
         */
    }
}

Seriously, you've given us years of fun and a conduit system that's become the de-facto standard for transferring things in the most compact way possible.  I'd chip in a dollar a month, and I'd say so would a lot of other of us gamers that appreciate all the hard work this takes, especially with the recently accelerated update schedule and the steep number of breaking changes you now have to deal with.
You've given us something.  Now it's time to let us give you something back.

گیتهاب

It is my feelings, which I believe are shared by @CrazyPants, that we would rather any money directed at us go instead to a charity. We do what we do for the fun of it, and don't expect any monetary compensation.
That said, neither I nor him are currently doing much work on EIO, so it is entirely up to @HenryLoenwind what to do, there are no restrictions on EIO that would prevent him from creating a Patreon or something of the sort, and I certainly would have no problem with him doing so.

I think I need to clarify my statement: I do have a good-paying job and do not hurt for money. However, having that job also means it takes a good chunk of my time doing it. Now if I could live from modding and quit my job, then I could invest that time into Ender IO. But, the chances of that happening, and being a long-term viable thing, are next to nil (unless some crazy billionaire decides he likes my work that much).
Please understand that that comment wasn't a request for funding, but a request for people to stop nagging me because I (can only) do this in my limited free time.


if action log_level == DEBUG and config python_runner_log_level is INFO cannot debug log

https://github.com/StackStorm/st2/blob/007beed8325b397d540d97905af8f2edb14b925f/contrib/runners/python_runner/python_runner/python_runner.py#L128
Cannot log DEBUG from python action if the config setting for python_runner_log_level under actions in st2.log is set to something other than DEBUG.
The reason is the above line checks if the log_level = DEBUG and if so it just uses the config setting in st2.log

A possible solution is to just remove the default setting in this yaml file so that log_level=None on initialization.  It should then default to using what is in the config (due to it being set in the constructor) and skip this line altogether.


Roll Plugins from 8ae850672142 to 77f4a367985e (2 revisions)

https://github.com/flutter/plugins/compare/8ae850672142...77f4a367985e
2021-06-23<EMAIL_ADDRESS>[camera] android-rework part 6: Android exposure- and focus point features (flutter/plugins#4039)
2021-06-23<EMAIL_ADDRESS>[ci] increase auto publish job wait interval to 3 mins (flutter/plugins#4086)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-plugins-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


Update config type definition

mimeType is used to properly decode src, which can be a Buffer or Uint8Array

Hi @andyedinborough ! Thanks for the submission!!
I'm not seeing anything in the code anywhere that references a mimeType parameter.  Was this something that was missed in the commit?  Or is it something that is provided externally somewhere?
Thanks!

Let me check the original code, but for my use-case (passing in raw data), it's necessary for the image data to be properly decoded.

Sorry, it was mime, see https://github.com/serratus/quaggaJS/blob/master/example/node-test-with-buffer.js. I was looking at the variable name that I was passing in.

Aha! ok, i see that in input_stream.ts line 45, where it calls GetPixels with a url and a mime type.
mimeType makes more sense.  I'm going to make a note to make it mimeType in the future, and document it.
Thanks for your submission!


Request for compatibility for portrait mode

Hello brother,
Your work is excellent.
But the only thing that is not letting me use your script to its full potential is that I am not able to use it portrait mode correctly, the banner gets distorted.
If there is a already a command for that please let me know.
I am a normal termux user and I use it in portrait mode only and I think there are also many other people who use it in portrait mode only.
So I request you to add an option to for Portrait mode.
Hope you will look into this.
Thank you

Nevermind found the solution we just have to pinch zoom out to make the banner fit on the screen.

Nevermind found the solution we just have to pinch zoom out to make the banner fit on the screen.


isDetox mock should probably be async

Hi! 👋
Firstly, thanks for your work on this project! 🙂
Today I used patch-package to patch<EMAIL_ADDRESS>for the project I'm working on.

Here is the diff that solved my problem:
diff --git a/node_modules/react-native-is-detox/jestSetup.js b/node_modules/react-native-is-detox/jestSetup.js
index 1b81728..ad12a79 100644
--- a/node_modules/react-native-is-detox/jestSetup.js
+++ b/node_modules/react-native-is-detox/jestSetup.js
@@ -1,4 +1,4 @@
 jest.mock('react-native-is-detox', () => ({
-  isDetox: () => false,
+  isDetox: async () => false,
   isDetoxSync: () => false,
 }));

This issue body was partially generated by patch-package.

Awesome! Thanks so much!


Adding Trivago logo

@guozhangwang Please review

LGTM. Merged to trunk and cherry-picked to 1.0.


Operator-sdk generate bundle creates poorly formatted yaml

Bug Report

What did you do?
A clear and concise description of the steps you took (or insert a code snippet).
When generating an operator-sdk bundle, the yamls that are created have yamllint errors. yamllint is a common linting tool that will be used with operator files and the outputted files should pass yamllint.
$ operator-sdk generate bundle -v 0.0.1 --input-dir deploy/olm-catalog/tron-example-operator/ --operator-name tron-example-operator --overwrite --quiet --default-channel alpha

WARN[0000] ClusterServiceVersion validation: [OperationFailed] provided API should have an example annotation
WARN[0000] ClusterServiceVersion validation: [OperationFailed] provided API should have an example annotation
WARN[0000] ClusterServiceVersion validation: [OperationFailed] provided API should have an example annotation
WARN[0000] ClusterServiceVersion validation: [OperationFailed] provided API should have an example annotation
WARN[0000] ClusterServiceVersion validation: [OperationFailed] provided API should have an example annotation
INFO[0000] Building annotations.yaml
INFO[0000] Writing annotations.yaml in $HOME/tron-example-operator/deploy/olm-catalog/tron-example-operator/metadata
INFO[0000] Building Dockerfile
INFO[0000] Writing bundle.Dockerfile in $HOME/tron-example-operator

$ yamllint .
./manifests/example.watson.ibm.com_backupwatsonexampleservices_crd.yaml
  1:1       warning  missing document start "---"  (document-start)
  17:3      error    wrong indentation: expected 4 but found 2  (indentation)

./manifests/tron-example-operator.clusterserviceversion.yaml
  1:1       warning  missing document start "---"  (document-start)
  13:5      error    wrong indentation: expected 6 but found 4  (indentation)
  16:7      error    wrong indentation: expected 8 but found 6  (indentation)
  23:7      error    wrong indentation: expected 8 but found 6  (indentation)
  30:7      error    wrong indentation: expected 8 but found 6  (indentation)
  37:7      error    wrong indentation: expected 8 but found 6  (indentation)
  44:7      error    wrong indentation: expected 8 but found 6  (indentation)
  51:3      error    wrong indentation: expected 4 but found 2  (indentation)
  56:7      error    wrong indentation: expected 8 but found 6  (indentation)
  69:15     error    wrong indentation: expected 16 but found 14  (indentation)
  70:17     error    wrong indentation: expected 18 but found 16  (indentation)
  103:17    error    wrong indentation: expected 18 but found 16  (indentation)
  106:15    error    wrong indentation: expected 16 but found 14  (indentation)
  109:15    error    wrong indentation: expected 16 but found 14  (indentation)
  113:3     error    wrong indentation: expected 4 but found 2  (indentation)
  122:3     error    wrong indentation: expected 4 but found 2  (indentation)
  124:3     error    wrong indentation: expected 4 but found 2  (indentation)
  127:3     error    wrong indentation: expected 4 but found 2  (indentation)

./manifests/example.watson.ibm.com_modelserves_crd.yaml
  1:1       warning  missing document start "---"  (document-start)
  17:3      error    wrong indentation: expected 4 but found 2  (indentation)

./manifests/example.watson.ibm.com_modeltrains_crd.yaml
  1:1       warning  missing document start "---"  (document-start)
  17:3      error    wrong indentation: expected 4 but found 2  (indentation)

./manifests/example.watson.ibm.com_frontends_crd.yaml
  1:1       warning  missing document start "---"  (document-start)
  17:3      error    wrong indentation: expected 4 but found 2  (indentation)

./manifests/example.watson.ibm.com_watsonexampleservices_crd.yaml
  1:1       warning  missing document start "---"  (document-start)
  17:3      error    wrong indentation: expected 4 but found 2  (indentation)

./metadata/annotations.yaml
  1:1       warning  missing document start "---"  (document-start)

What did you expect to see?
A clear and concise description of what you expected to happen (or insert a code snippet).
The created yaml files should be properly formatted and should pass the yamllint -er. Running yamllint should produce no errors.
What did you see instead? Under which circumstances?
A clear and concise description of what you expected to happen (or insert a code snippet).
Instead, there are multiple warnings and errors. These include document-start and indentation errors (see above example).
Environment

operator-sdk version:

operator-sdk version: "v0.18.1", commit: "7bf7b6886d647dc202525daec16fab67dcc52a3d", kubernetes version: "v1.18.2"

go version:

go version: "go1.13.6 linux/amd64"

Kubernetes version information:

n/a

Kubernetes cluster kind:

Are you writing your operator in ansible, helm, or go?

n/a
Possible Solution

Additional context
Add any other context about the problem here.

It is duplicated of https://github.com/operator-framework/operator-sdk/issues/3545.
Closing as duplicted.

Thanks for the quick response @camilamacedo86 , not sure how I missed that issue! 😅


Token: Allow querying of all owned tokens by pubkey

Problem
There is no way to look up token accounts owned by an account's pubkey.  spl-token accounts requires an --owner <KEYPAIR> argument be provided (or CLI default config).
This is particularly challenging, and I expect will become a louder issue, as we are working out wallet inter-operability issues as more wallets start following our recommended practice of only exposing the system account address to the user.  For debugging, anyone should be able to look up the token accounts owned by a wallet's pubkey.
Proposed Solution
Accept --owner <KEYPAIR or PUBKEY>

cc: @t-nelson @CriesofCarrots not sure whose court this would be in.

I can probably sneak this in

Also, update output:
        USDC_MINT 40
Aux-1*  USDC_MINT  2
Aux-2*  USDC_MINT  5
        SRM_MINT  10

* Please run `spl-token gc` to clean up Aux accounts

spl-token accounts -v outputs token account addresses

Flipping the table around a little makes sort work again too:
USDC_MINT 40
USDC_MINT  2 (Aux-1)
USDC_MINT  5 (Aux-2)
SRM_MINT  10

* Please run `spl-token gc` to clean up Aux accounts


Private Multivalues

Masterless concurrency strategy for private partition
https://whitepaper.fission.codes/file-system/partitions/private-directories/data-layer#multivalues-else-lww
[Note: as an industry, we need a better term than "masterless". "Mainless"? "Egalitarian"?]

Moved to rs-wnfs


HDDS-9093. Appended tailing slash while constructing the bucket key for SSTFilteringService

What changes were proposed in this pull request?
In PR-5008, tailing slash was appended in filterPrefix and filterPrefixFSO so that we don't delete wrong SST files.
For example, SST filtering service is running for a snapshot taken on vol1/bucket, if tailing slash is not appended, it might delete the SST files belongs to vol1/bucket1, vol1/bucket2 or anything with bucket prefix.
The above change caused a regression, because prefix had tailing / but firstBucketKey and lastBucketKey didn't have it.
This change is to append the tailing / on the firstBucketKey and lastBucketKey.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9093
How was this patch tested?
In the interest of time, just added a unit test which checks FILTER_FUNCTION is correct.
Will add end to end test while this is in review or as separate PR.

Added integration test as part of https://issues.apache.org/jira/browse/HDDS-9094


Message Handler Fix: Not Invoking Beans

Fixed a bug where when trying to execute messages annotated with @CbMessageListener or @SbMessageListener when receiving the corresponding packet, this would fail because we weren't passing the parent bean into the method invocation.

Fix works, tested on my laptop


Cloud Architecture Design

Hey there! 👋
I just wanted to drop a quick thank you for sharing your awesome repo and the inspiring article!
I've got a question about your choice of CloudFront over Cloudflare. I noticed Protomaps leans towards Cloudflare because of the cost factor with CloudFront in an AWS setup. So, I'm curious, is there something I'm missing here?
Thanks a bunch for your time!
Cheers,
Filippos

Thanks for reaching out! I don't think the linked comparison is accurate, as CloudFront has a free tier of 1TB outgoing traffic per month, see https://aws.amazon.com/blogs/aws/aws-free-tier-data-transfer-expansion-100-gb-from-regions-and-1-tb-from-amazon-cloudfront-per-month/ In the comparison, the 1000GB traffic is listed with $100...
Also, I'm using AWS because that's where I'm an expert in, other people might use Cloudflare or other providers as they desire.

Hi @tobilg,
I must say, the 1TB of outgoing traffic per month is quite appealing. I overlooked that detail earlier.
Do you believe the current setup of this repository is production-ready? Are there any areas you think could be enhanced to improve its readiness? If you're open to freelancing opportunities, I'd greatly appreciate your assistance for my use case :)
Thank you for your valuable feedback in advance!

I guess the answer is "it depends", as I don't know what you exactly define as "production-ready". Could you elaborate a bit, as well as describing your use case?

I'm trying to use a customizable basemap that can look beautiful, like the screenshot that you have in this repo. Was it like that by default or you customized it to look like that? The map in my use case is to be used on a mobile app where I will just display POIs.
Also, what I meant by production ready, was actually something that is secure in terms of policies that does not allow bots hitting with request indefinitely, or if this repo could be improved further in general, for example, does it follow the practises described by protomaps?.

It's all in the repo, the screenshot is taken from https://github.com/serverlessmaps/serverlessmaps/blob/main/website/index.html once deployed to CloudFront.
Similarly, the IaC is found in https://github.com/serverlessmaps/serverlessmaps/tree/main/iac which you can review yourself in case you want to check if it really adheres to the Protomaps recommendations. Regarding bot mitigation, you can place a WAF in front of CloudFront (see e.g. https://cloudjourney.medium.com/aws-cloudfront-and-waf-97d7314b07af), but this is not built-in to the IaC because potential costs.
Don't get me wrong, but this is an Open Source project that is provided free of charge, meaning that I can't give you free customer support.

That's why I asked previously if you are open to freelancing opportunities :)
Thanks for your detailed reply once again!


Disable black overlay using configure?

For most situations I think the transparent black overlay that dims the screen makes sense, but would it be possible to disable this with the configure. Some situations I don't want the entire screen dimmed on a notification type popup (top and bottom).

Hey @praveenperera,
This is possible, please check the code below:
TopPopup_Notification()
    .showAndStack()
    .hideOverlay()

I can see that this is not very user-friendly, so I will probably move it to a popup declaration in the future versions.
Have a nice day,
T.K.

Oh sorry about that must have missed it in the readme. But ya probably makes more sense in the configure?


adtl as an optional / core dependency

Although InsightBoard can operate without adtl, it is closely associated with the tool and should be at least an optional install option, or even elevated to a core dependency. This would also simplify the user install instructions.

Resolved by #72


Add rescue statements for S3

This so so that S3 does not cause our whole site to not work if S3 goes down.

Coverage decreased (-0.2%) to 97.307% when pulling f57afe6e8bcbd194723dfe9154c5cd41190d7360 on rescueS3Errors into f08c5042dab39ed8abc095e3aefefc3969626752 on master.


[QST] The difference between single-mt-multi-cbg and multi-sste-multi-cbg using node-level-cbg?

Hi, First I want to say this is a great work!
I our practices, we use the typical way of using node-level-cbg with a multi-threaded-executor adding many cbg(sub: cbg is 1:1), and there are many threads inside the executor.
But in your repo I see a noval design with many single-threaded-executor adding only one cbg(executor: sub: cbg is 1: 1: 1), and there are many executors which only spawn one thread.
I think both ways are multi-thread-subscription. What't the big difference between them, and your considerations about this design? Are you finding some issues about the typical usage of what I said above?
more info about node-level-cbg is here https://github.com/clalancette/executor_examples/tree/master/src
more info about executor-level-cbg https://github.com/ros2/examples/tree/rolling/rclcpp/executors/cbg_executor
Thanks.

What I am thinking.

It's difficult to give management to threads inside a multi-threaded-executor.
Its easy to give management to the many thread of a single-threaded-executor.

https://github.com/ros2/examples/blob/1d97c4fc7445554f6f85f63305d424fc017212a0/rclcpp/executors/cbg_executor/src/ping.cpp#L51-L80
But the threads management is covered by your ComponentManagerCallbackIsolated::add_node_to_executor
https://github.com/sykwer/component_container_callback_isolated/blob/3d84f28e7c97239a65ff3706908b1dd3e8c8e215/src/component_container_callback_isolated.cpp#L64-L67
What's your idea?


Missing dependencies

When trying to pip install python-flint, it fails if you don't have Cython or numpy installed. This should be added to the required packages.

I am not sure how to do this from within setup.py. I think that it requires moving from setup.py to pyproject.toml which also then requires choosing a build backend. There are a few related problems in making that change in python-flint.
For now as a workaround you should install Cython and numpy and use --no-build-isolation as described in the README:
pip install 'cython>=3' numpy
pip install --no-build-isolation python-flint

On Windows or OSX this is not necessary because pip install python-flint will install precompiled binaries.
I'm going to close this issue in favour of gh-52 which describes in more detail what needs to be done.


Implement GuzzleHttp\ClientInterface on HttpPlayback\Client

If the Guzzle ClientInterface is implemented, the Playback Client can pass type-hint declarations and maintain API compatibility. Makes DI overriding easier.

Will release with version 1.1.0 next week


Add Environment Variables for API Keys

Ensure that all API keys are stored securely using environment variables.

keys incluided in the .env.sample for group to use


I still don't understand, what is the difference between this and puppeteer extra & stealth? We have too many variations.

I'm impressed with how well this repository's documentation is written, especially with the helpful examples that make it easy to understand. However, I still don't quite grasp the difference between this repository and puppeteer extra/stealth. Can we combine the two to achieve a higher level of anonymity?

I still don't quite grasp the difference between this repository and puppeteer extra/stealth

There are a lot of them:

We patch browser to change fingerprint related properties, puppeteer stealth uses javascript injections.
We obtain fingerprints from real devices. So each profile will look for site like a different user. And the values combination will look natural.
Advanced canvas data replacement is used https://wiki.bablosoft.com/doku.php?id=perfectcanvas . Canvas data will be rendered on real device and then injected into browser.
We use our own proxy implementation which allows to route UDP traffic, use socks5 with passwords. etc.
Headless never used. Instead browser is hidden in different way. For site it seams that browser is visible.
We provide accurate mobile fingerprints emulation.

And many many more, I don't even remember all the stuff.

Can we combine the two to achieve a higher level of anonymity?

No, because puppeteer stealth uses javascript injections, it could detected.

The browser is downloading - this may take some time.

stuck for an hours like that. please help

@putuoka perhaps the download speed is too slow, otherwise an error would occur. You can try downloading the engine manually from this link - then place the archive in the ./data/engine or ./PATH_TO_ENGINE/engine folder, where PATH_TO_ENGINE is the path to the engine if you install it yourself.

@putuoka perhaps the download speed is too slow, otherwise an error would occur. You can try downloading the engine manually from this link - then place the archive in the ./data/engine or ./PATH_TO_ENGINE/engine folder, where PATH_TO_ENGINE is the path to the engine if you install it yourself.

Okay thanks, now I understand why it took so long to download - because the file size was 700mb :(

@putuoka perhaps the download speed is too slow, otherwise an error would occur. You can try downloading the engine manually from this link - then place the archive in the ./data/engine or ./PATH_TO_ENGINE/engine folder, where PATH_TO_ENGINE is the path to the engine if you install it yourself.

What, it is BAS ??

https://bablosoft.com/shop/BrowserAutomationStudio
This software is based on BAS

https://bablosoft.com/shop/BrowserAutomationStudio
This software is based on BAS

I dont like it, it too heavy, i like chromium with 32 bit

@putuoka perhaps the download speed is too slow, otherwise an error would occur. You can try downloading the engine manually from this link - then place the archive in the ./data/engine or ./PATH_TO_ENGINE/engine folder, where PATH_TO_ENGINE is the path to the engine if you install it yourself.

What is PATH_TO_ENGINE? How I can change the data folder location?

We use our own proxy implementation which allows to route UDP traffic, use socks5 with passwords. etc.

Hello. You say that the library supports UDP traffic (if proxies support it), but the UDP test https://networktest.twilio.com/ cannot pass.
My proxies support both UDP and QUIC protocols, but unfortunately, the test failed in this library. At the same time, if I launch BrowserAutomationStudio and check the box to support QUIC, everything works and the test passes

@putuoka perhaps the download speed is too slow, otherwise an error would occur. You can try downloading the engine manually from this link - then place the archive in the ./data/engine or ./PATH_TO_ENGINE/engine folder, where PATH_TO_ENGINE is the path to the engine if you install it yourself.

Still same issue. How to fix?


Tree - nodeDrop - missing parameter

I missing the content of the text field as parameter ('text') by nodeDrop.
It's not possible to check for dublicate without this information server side before saving the changed data, isn't it?

You can use the id parameter in nodeDrop and http://gijgo.com/tree/methods/getDataById method in order to get the text.

Thanks...I've seen this solution after my post.


Fail Fast control for with item task

Currently the incomplete items in a task's with list won't be processed if there are no active task item and parent task has failed status. For example let's consider this workflow
version: 1.0

description: A workflow demonstrating with items and concurrent processing.

input:
  - members  # [0,1,2,3]
  - concurrency: 1

tasks:
  task1:
    with:
      items: <% ctx(members) %>
      concurrency: <% ctx(concurrency) %>
    action: core.local cmd="[ item() -eq  1 ] && exit 1 || exit 0"

If we run above workflow it will process items 0 and 1 only, as on member 1 action will fail which will set the task1 status as fail and the concurrency is set to 1 there for there will be no active task item and state of task1 will be set to failed from running.
And no other item will be processed.
With this change I have introduce a boolean parameter failfast in task which will decide wether to fail the entire task if there are not active task item and task status is failed.
version: 1.0

description: A workflow demonstrating with items and concurrent processing.

input:
  - members  # [0,1,2,3]
  - concurrency: 1

tasks:
  task1:
    failfast: false
    with:
      items: <% ctx(members) %>
      concurrency: <% ctx(concurrency) %>
    action: core.local cmd="[ item() -eq  1 ] && exit 1 || exit 0"

@exp-vkishore would you be able to revisit this? What do you think of @m4dcoder 's suggestion of a flow: keyword under with:?

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


please add bot automerge

@conda-forge-admin

Hi! This is the friendly automated conda-forge-webservice.
I just wanted to let you know that I added bot automerge in conda-forge/pymove-feedstock#20.


fix(versions): update Activiti/activiti-cloud-example-chart versions into master

UpdateBot pushed helm dependency: application to: 0.0.181

UpdateBot commands:
updatebot push-version --kind helm application 0.0.181


⚠ Bofrost.fr has degraded performance

In b9c7edf, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 1094 ms

Resolved: Bofrost.fr performance has improved in ed046fd after 6 minutes.


⚠ ClamAV has degraded performance

In 10bf913, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 63 ms

Resolved: ClamAV performance has improved in af97e9c.


Support hjkl and Ctrl+U/Ctrl+D in Preview mode

It's great to have vim key-binding in Preview mode (HTML preview) as well. Actually my main use case is to navigate around the docs so having hjkl and Ctrl+U/Ctrl+D is good enough.
Thank~

Yeah sounds nice!

landed in v2.0.0🎉

I think this is correct.
diff --git a/src/vim.js b/src/vim.js
index e20e0fb..ceb881d 100644
--- a/src/vim.js
+++ b/src/vim.js
@@ -921,10 +921,10 @@ class Plugin {
         'vim:scroll-half-screen-down': ({ target }) => {
           target.scrollTop += target.clientHeight / 2
         },
-        'vim:scroll-half-screen-up': ({ target }) => {
+        'vim:scroll-full-screen-up': ({ target }) => {
           target.scrollTop -= target.clientHeight
         },
-        'vim:scroll-half-screen-down': ({ target }) => {
+        'vim:scroll-full-screen-down': ({ target }) => {
           target.scrollTop += target.clientHeight
         },
         'vim:move-to-line': ({ target }) => {

@basyura Oops!! Will fix it. Thanks!

landed in v2.0.1🎉


BeginAsync does not exist

In the auth tutorial beginasync is referenced, but I think it was changed to BeginDialogAsync in a recent update?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 615eae0a-7bae-820c-8fda-09fb0899bd0f
Version Independent ID: c2e5b308-760c-1226-ba9b-bef7ae535fca
Content: Enterprise Bot Template Deployment - Bot Service
Content Source: articles/v4sdk/bot-builder-enterprise-template-deployment.md
Product: bot-framework
GitHub Login: @darrenj
Microsoft Alias: darrenj

Also says SignIn.Name doesn't exist either - and not sure what to put here.


Only act on musicbrainz as a source

I have a number of albums with a data_source other than musicbrainz.
If I run beet oldestdate it fails so I have to add a query to the command beet oldestdate data_source:musicbrainz
I have not yet tried it on an import.

What does the data source field do? How do some albums have it and don't? If you could provide some examples I should be able to fix this.

2 ways

Filter with regex mb_albumid, I added this to the mb_sync plugin

I found a new field in beets called data_source if you enable discogs or any other metasources this gets populated with the name of the data_source

I could have a go at either one if you will accept a PR.

I suppose if the data_source is not musicbrainz then we can skip the track? You're welcome to open a PR for this.

I am looking at the code, and I am not sure where to add this. I was going to add it in process_file, but I am getting this error in get_distance. There is no item there to work with. Any ideas?

got something https://github.com/kernitus/beets-oldestdate/pull/13


Use [hidden] on group instead of changing render structure

Fixes #26
I risked this change in c95c9a0 but didn't realize each item would be remounted with a new id when the group is hidden, which breaks a lot of logic that relies on each item having a stable id.

Very nice


Request for support com.victronenergy.evcharger

https://github.com/victronenergy/venus/wiki/dbus#grid-and-genset-meter
Would it be possible to add evcharger devices, so i can include my wallboxes?
Thank you very much!

Hi,
you can edit services.yml according to the evcharger and it should work.
Sebastian

Had the same need:
here is my currend work in progress:
you can see it in vrm and monitor power with this
Unfortunately, i think we need help fom @freakent

How can i set it connected/disconnected? it seems that it does nor work changing this value
MaxCurrent does not work persisdent.
StartStopp thows an error

evcharger:
  ProductId:
    default: 65535 # 0xFFFF
  CustomName:
    default: "MyCharger"
    persist: true
  Ac/Energy/Forward:
    description: "wh"
    format: "{:.2f}W"
  Ac/L1/Power:
    description: "W"
    format: "{} W"
  Ac/L2/Power:
    description: "w"
    format: "{} W"
  Ac/L3/Power:
    description: "w"
    format: "{} W"
  Ac/Power:
    description: "w"
    format: "{} W"
  AutoStart:
#    description: "w"
    default: "0"
    persist: true
#    format: "{}"
  ChargingTime:
    description: "in seconds"
    format: "{} s"
  Current:
    description: "A"
    default: 0
    format: "{}"
#    persist: true
  EnableDisplay:
    description: ""
    format: "{}"
  MaxCurrent:
    description: "A"
    default: 0
#    format: "{}"
    #persist: true   #wird nicht gespeichert reset nach disconnect/reconnect
  Mode:
    description: ""
    default: 0
    format: "{}"
  Model:
    description: "w"
    default: "AC22NS"
    format: "{}"
  Position:
    description: ""
    format: "{}"
    #persist: true   # geht nicht
  Role:
    description: "w"
    format: "{}"
  SetCurrent:
    description: "w"
    format: "{}"
#  StartStop:           #mqtt deamon sbeendet sich fatal wenn gesetzt
#    description: "w"
#    format: "{}"
#    persist: true
  Status:
    description: "w"
    default: 1

This is the StartStop-Errormessage
INFO:device_manager:Received device status message {'clientId': 'ev002', 'connected': 1, 'version': 'v0.9', 'services': {'ev1': 'evcharger'}}
exit_on_error: there was an exception. Printing stacktrace will be tried and then exit
Traceback (most recent call last):
  File "/data/drivers/dbus-mqtt-devices-0.5.1/ext/velib_python/ve_utils.py", line 24, in exit_on_error
    return func(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 1572, in loop_read
    rc = self._packet_read()
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 2310, in _packet_read
    rc = self._packet_handle()
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 2936, in _packet_handle
    return self._handle_publish()
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 3216, in _handle_publish
    self._handle_on_message(message)
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 3444, in _handle_on_message
    self.on_message(self, self._userdata, message)
  File "/data/drivers/dbus-mqtt-devices-0.5.1/device_manager.py", line 60, in _on_message
    if self._status_is_valid(status):
  File "/data/drivers/dbus-mqtt-devices-0.5.1/device_manager.py", line 109, in _status_is_valid
    if services.get(service_id) not in self.service_types: # as defined in services.yml
TypeError: argument of type 'NoneType' is not iterable

Is the services.yml extract you have provided the exact version giving this error?

yes!!

The StartStop section looks OK, apart from having persist: true.
Only use persist: true for static device settings. The problem is now that you have declared this a persisted value I think dbus will always retain the value you last set.

so i have to save the state in nodered and send it after restarts?
Is there any chance to delete this?
i already had to reinstall my production-venusOS because of issues regarding this... ;-)

The AutoStart property does have a problem, the description is commented out. Every section MUST have a description attribute. Also you have comments around the default value, but I think that should be a numeric value not a string.
  AutoStart:
#    description: "w"
    default: "0"
    persist: true
#    format: "{}"

ok, changed this. thank you.
Do you know why Charging Current is greyed out?

Is there any chance to delete this?
Have a look at a victron tool called dbus-spy, it will help you see what is actually help in the dbus.

Do you know why Charging Current is greyed out?
Sorry, I don't

Here are a few resources for you to cross reference:

The Victron modbus-TCP register list. You can request a download of the spreadsheet from Victron, which includes a section for evcharger: https://www.victronenergy.com/panel-systems-remote-monitoring/color-control#technical-information

Victron's dbus-modbus implemtation for evcharger: https://github.com/victronenergy/dbus-modbus-client/blob/master/ev_charger.py

Open_EVSE charger Victron implementation : https://github.com/JuWorkshop/dbus-evsecharger/blob/main/dbus-evsecharger.py

Here you can see /Connected as parameter
https://github.com/victronenergy/venus/wiki/dbus#evcharger
This thets the vehicle to be connected.
But when I try to use it, I get:

#"Can't register the object-path handler for '/Connected': there is already a handler"

Is this not possible to use because of internals of this driver?

Where do you see this message? It's not one I recognise.

When i add " Connected:" to the services.yml for the evcharger profile
and then try to start the driver in the console, i see this error.
I thought that this is because your internal varuable uses "conncted" for the virtual device? but this is just a guess...

I thought that this is because your internal varuable uses "conncted" for the virtual device? but this is just a guess...

I doubt it, very different.
Please show the relevant section from your services.yml, /Connected is not in the extract you provided earlier.

When i add " Connected:" to the services.yml for the evcharger profile
and then try to start the driver in the console, i see this error.

Please also provide full text from the logging so I can see the context.

If you get weird messages like this, change the client id you send in the registration message (currently ev002) so that you create a new, clean set of dbus entries.

I noticed you have lots of
  format: "{}"
```.

Remove these, they add no value. Only use a format if the default format is not what you want. My worry is that you may be trying to apply formats incorrectly.

ok, i did.
here is the full errormessage
INFO:logger:Loglevel set to INFO
INFO:device_manager:Using portalId b827eb20de35
INFO:mqtt_gobject_bridge:[Init] Connecting to local broker
INFO:device_manager:[Connected] Result code 0
INFO:device_manager:Received device status message {'clientId': 'ev003', 'connected': 1, 'version': 'v0.9', 'services': {'ev3': 'evcharger'}}
INFO:device:**** Registering device: ev003, services: {'ev3': 'evcharger'} ****
INFO:device:Registering Service evcharger for client ev003
INFO:device_service_config:About to open config file
INFO:device_service:Registering service evcharger for client ev003 at path com.victronenergy.evcharger.mqtt_ev003_ev3
INFO:settingsdevice:Setting /Settings/MqttDevices/mqtt_ev003_ev3/CustomName does not exist yet or must be adjusted
INFO:settingsdevice:Setting /Settings/Devices/mqtt_ev003_ev3/ClassAndVrmInstance does not exist yet or must be adjusted
INFO:vedbus:registered ourselves on D-Bus as com.victronenergy.evcharger.mqtt_ev003_ev3
exit_on_error: there was an exception. Printing stacktrace will be tried and then exit
Traceback (most recent call last):
  File "/data/drivers/dbus-mqtt-devices-0.5.1/ext/velib_python/ve_utils.py", line 24, in exit_on_error
    return func(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 1572, in loop_read
    rc = self._packet_read()
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 2310, in _packet_read
    rc = self._packet_handle()
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 2936, in _packet_handle
    return self._handle_publish()
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 3216, in _handle_publish
    self._handle_on_message(message)
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 3444, in _handle_on_message
    self.on_message(self, self._userdata, message)
  File "/data/drivers/dbus-mqtt-devices-0.5.1/device_manager.py", line 62, in _on_message
    self._process_device(status)
  File "/data/drivers/dbus-mqtt-devices-0.5.1/device_manager.py", line 143, in _process_device
    self._devices[clientId] = device = MQTTDevice(device_mgr=self, device_status=status)
  File "/data/drivers/dbus-mqtt-devices-0.5.1/device.py", line 30, in __init__
    device_service = MQTTDeviceService(self, id, service)
  File "/data/drivers/dbus-mqtt-devices-0.5.1/device_service.py", line 45, in __init__
    self._set_up_dbus_paths()
  File "/data/drivers/dbus-mqtt-devices-0.5.1/device_service.py", line 103, in _set_up_dbus_paths
    dbus_service.add_path("/"+k, value=value, description=v.get('description'), writeable=True, gettextcallback=textformatcallback, onchangecallback=changecallback)
  File "/data/drivers/dbus-mqtt-devices-0.5.1/ext/velib_python/vedbus.py", line 107, in add_path
    item = VeDbusItemExport(
  File "/data/drivers/dbus-mqtt-devices-0.5.1/ext/velib_python/vedbus.py", line 468, in __init__
    dbus.service.Object.__init__(self, bus, objectPath)
  File "/usr/lib/python3.8/site-packages/dbus/service.py", line 489, in __init__
    self.add_to_connection(conn, object_path)
  File "/usr/lib/python3.8/site-packages/dbus/service.py", line 578, in add_to_connection
    connection._register_object_path(path, self._message_cb,
KeyError: "Can't register the object-path handler for '/Connected': there is already a handler"

But i think in this description and context, this value is used to tell the device that the vehicle is connected to the powerwall.
Is there a workaround possible?

No, it's fundamental. I haven't seen any Victron documentation that suggests that it indicates whether it is connected to the car. It would also mean that the definition of Connected is different for ev chargers than any other service.

ok. But whats about StartStop?
Here is the log
*** starting dbus-mqtt-devices ***
-------- dbus_mqtt_devices, v0.5.1 is starting up --------
INFO:logger:Loglevel set to INFO
INFO:device_manager:Using portalId b827eb20de35
ERROR:device_manager:Unexpected error: <class 'yaml.scanner.ScannerError'>
INFO:mqtt_gobject_bridge:[Init] Connecting to local broker
INFO:device_manager:[Connected] Result code 0
INFO:device_manager:Received device status message {'clientId': 'ev003', 'connected': 1, 'version': 'v0.9', 'services': {'ev3': 'evcharger'}}
exit_on_error: there was an exception. Printing stacktrace will be tried and then exit
Traceback (most recent call last):
  File "/data/drivers/dbus-mqtt-devices-0.5.1/ext/velib_python/ve_utils.py", line 24, in exit_on_error
    return func(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 1572, in loop_read
    rc = self._packet_read()
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 2310, in _packet_read
    rc = self._packet_handle()
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 2936, in _packet_handle
    return self._handle_publish()
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 3216, in _handle_publish
    self._handle_on_message(message)
  File "/usr/lib/python3.8/site-packages/paho/mqtt/client.py", line 3444, in _handle_on_message
    self.on_message(self, self._userdata, message)
  File "/data/drivers/dbus-mqtt-devices-0.5.1/device_manager.py", line 60, in _on_message
    if self._status_is_valid(status):
  File "/data/drivers/dbus-mqtt-devices-0.5.1/device_manager.py", line 109, in _status_is_valid
    if services.get(service_id) not in self.service_types: # as defined in services.yml
TypeError: argument of type 'NoneType' is not iterable

It's definitely a YAML parse error:
<class 'yaml.scanner.ScannerError'>

Interessant, das habe ich sicher drei oder vier mal gecheckt, es dürfte aber mit dem Kommentar dahinter zu tun haben. Als ich das gelöscht habe, gings.
Vielen Dank, und sorry für die häufigen Rückfragen...

Thank you!
With this config, "position" becomes editable:
Position:
persist: true
default: 0
min: 0
max: 1
description: "0=OUTPUT; 1=INPUT"

Now, my chargers are visible in VRM.
(How) could this implementation be used to do "PV-Überschussladen" without other tools?

Here is a current Screenshot: Works great.
Sadly, Victron seems to have no "soc" field for the car.
Would be a nice info aswell.

Why don't you post a suggestion on the Victron community forum to add an SOC field to EVCharger on the dbus. No need to mention this driver, it's the dbus api that needs to be extended.

Thank you! I'm not "into" it , I am just new to victron so i am learning a lot currently.
I need to get a useraccount there, so yes, i will try to do so the next days.

You could also register the car's battery on the system and push SOC to the battery.

@pos-ei-don
I tried your expanded services.yml from your PR.
In venusOS 2.82 this driver did not register a new device. I will try it on a newer version of venusOS.

I have multiple instances running, di I think it should work.
I am so happy with it the at I even migtated another Inverter from the shelly plugin to this one. I live it to be able to manipilulate some values while the GX is up and running

Maybe I did something wrong.
This driver runs smooth since one year, so I'm not in the flow right now... :-D
I would like to monitor my dumpload as evcharger.
I love It too. It's so flexible and easy to use! I don't know why there is not more attention. Or maybe it is, but just a view people have problems. ;-)

I think we should Post easy to use node Red flows to create Devices and fill them with values. This should help others to get startet much faster.
I did some programming in NodeRed, but currently it does not run without manual setting a value, so it needs some more attention 😉

Off topic: Also reading a existing Smartmeter with Tasmota and send it directly to venusOS is on the list. That would eliminate the need of a (relative expensive) EM24.

I use this, and it works well! Also hoymiles can be integrated directly.  (and the EV charger for the car, but dlthis is a different Topic)

I think we should Post easy to use node Red flows to create Devices and fill them with values. This should help others to get startet much faster.
I did some programming in NodeRed, but currently it does not run without manual setting a value, so it needs some more attention 😉

The best place to post sample flows for NodeRed is https://flows.nodered.org. I don't mind adding links in the documentation to to sample flows on https://flows.nodered.org.

@pos-ei-don
what are the minimal information the victron system needs, to show up the charger in the overview?
With just power and energy it didn't.

I've never testes "the minimal" settings.
Here are the values I set from the nodered to this device.
Some other settings i set in the console (charge-current,...) and send it the car.

Are you shure that the registration succeded? You can monitor the calues in MQTT, did the device in the /N/ path (N//evcharger/... ) take the values? be shure to send numbers, not strings!

Here everything looks fine.
But not in the overview. But that's maybe because it's just a RasPi with nothing else connected. I Will try it on my main system.

I dont use the overview.  But i dont See it there either. I use the online vrm.
Do you See it there?

You are right. It shows up in VRM.
Thanks :)

You are charging with 51w? 😉.
Do you plan to connect a solar charger with this?

This is just a test environment.
I Use this driver to send temperatures and power of 6 APSystems inverters to the victron system.
I would like to use the "evcharger" for monitoring a dumpload.
A friend will use it to monitor charging of his motorcycle.
I hope you services.yml will be included in the next version of the driver.
50W is just a example Power. :-D

@pos-ei-don
I have it running now for a few days without problems.
It's nice to see a ev charging but are there other benefits?

I was expecting a calculation of how much energy did go to the ev like the battery (direct use, to battery, to grid, to ev).
Can the Victron system control the charger? Start/stop, power control, ...

You can add an additional battery device, but i am unshure if its worth it.
You can Start and stop, but its on you to Code the backen connection to your charger. My EV-charger works with 0-10V input, so i can change the charge-current within Victron/NodeRed as needed.

Is start/stop and charge current native with VenusOS possible or only with separate logic like NodeRed on VenusOS lage?
I can't start/stop charging using the GUI, the slider is grayed out.

The wallbox can be controlled with VRM or the Cerbo GX via openWB.
However, it doesn't make any sense because openWB controls the wallbox very well and you have a lot more setting options.
Otherwise everything works great. Unfortunately, the SoC from the car is not transmitted, but it is not that important either.
The flow for Node Red openWB - Cerbo GX is available here:
https://openwb.de/forum/viewtopic.php?p=79599#p79599

@stephwe
Thank You Stephan.
But I don't have a real Smart EV charger. I have two use cases.

Electric motorcycle with a Tasmota flashed power plug.
A dump load controller (Tasmota).
Right now I control them from ioBroker. But I think it would be nice if the Victron system could control them. Just to have a "cleaner" system.

The on/off switch does not work for me either.
But Automatic/Manual charging works. So i used this one for turning something on/off.
Its only a workaround, i hope that victron will add several needed features to its EV module.

Is therre any reason why this is closed? The service file still does not contain evcharger or pvcharger.

Looks like the thread went off topic and a working config was not verified for evcharger. Going forward, I am replacing the single services yaml file with separate config files for each service. It's in beta at the moment. I'm not going to try to merge in any more community changes to the services file only to have to migrate them later. I also need contributors to provide tests so that I can verify any new services work as I have no way of testing them.

I already use the 0.7.5-beta which works great for me. I'm happy to test new service files.

I stopped using vrm at the Moment so i am not able to change this to the new Format. I cannot test it.
I hope someone will take over 😉.

Looks like the thread went off topic and a working config was not verified for evcharger. Going forward, I am replacing the single services yaml file with separate config files for each service. It's in beta at the moment. I'm not going to try to merge in any more community changes to the services file only to have to migrate them later. I also need contributors to provide tests so that I can verify any new services work as I have no way of testing them.

I can confirm that this config is working (since a couple of month now)
It would be great to have it included, then the mqtt EV Charger user don't need to inlcude this manually after every upgrade of dbus_mqtt_devices.
# https://github.com/victronenergy/venus/wiki/dbus#evcharger
evcharger:
  ProductId:
    default: 65535 # 0xFFFF
  CustomName:
    default: "MyCharger"
    persist: true
  Ac/Energy/Forward:
    description: "wh"
    format: "{:.2f}W"
  Ac/L1/Power:
    description: "W"
    format: "{} W"
  Ac/L2/Power:
    description: "w"
    format: "{} W"
  Ac/L3/Power:
    description: "w"
    format: "{} W"
  Ac/Power:
    description: "w"
    format: "{} W"
  AutoStart:
    description: "w"
    default: 0
#    persist: true
  ChargingTime:
    description: "in seconds"
    format: "{} s"
  Current:
    description: "A"
    default: 0
#    persist: true
  EnableDisplay:
    description: ""
  MaxCurrent:
    description: "A"
    default: 0
#    persist: true   #wird nicht gespeichert reset nach disconnect/reconnect
  Mode:
    description: ""
    default: 0
  Model:
    description: "w"
    default: "AC22NS"
  Position:
    persist: true
    default: 0
    min: 0
    max: 1
    description: "0=OUTPUT; 1=INPUT"
#  Connected:
#    description: "w"
  Role:
    description: "w"
  SetCurrent:
    description: "w"
#  StartStop:
#    description: "w"
#    persist: true
  Status:
    description: "w"
    default: 1

@dersch81 Thanks for raising this again. This thread went so off topic it got lost. I think this service description still needs refinement. I will open a new issue and will propose some improvements. I would welcome your collaboration (and anyone else interested) on this.


Default Value Attribute more does not work in version 5.1.0

I have enum (img 1)

I use this enum in my model along with DefaultValueAttribute (img 2)

I created a DefaultValueSchemaFilter which allows you to use DefaultValueAttribute with Swagger. (img 3)

After that I registered mine in SwaggerGen (img 4)

When using version 5.0.0 DefaultValueAttribute works correctly (img 5)

But when using version 5.1.0 DefaultValueAttribute works not correctly (img 6)

How can I fix this problem?

When configuring Swagger in your Startup.cs, apply the UseAllOfToExtendReferenceSchemas setting. Long story short, this behavior was made implicit in 5.0.0 and while it's helpful for some situations (including your own), it also caused a range of new issues particularly for people that were using code gen tools to generate clients from the generated Swagger/OpenAPI document. So, I made the behavior opt-in only as of 5.1.0. By opting-in, you should get the behavior you're looking for.


jdk14u aarch64 64k builds show an error while testing on 4k machines

Platform: arm64
Architecture: aarch64
The problem is that if you build jdk14u on an aarch64 machine with 64k pages, and then use it on a machine with 4k pages (and possibly the reverse too), you get this non-fatal error right before the -version output:
[0.011s][error][cds] Unable to map CDS archive -- os::vm_allocation_granularity() expected: 65536 actual: 4096

This was originally reported here.
However, the fix was not backported to 14, nor does the fix appear to be an actual resolution of the issue (but rather something to let the build process tolerate the error).
Since jdk14 is about to go out of service, I'm not convinced that backporting is a good use of anyone's time, however these test failures still need to be resolved for jdk14+ test runs on aarch64.
If the accepted solution for the build issue was to ignore the error message, it seems like the correct action here is to modify the tests to do the same via an OpenJDK bug.
Notes:
64k machine - build-osuosl-aix71-ppc64-2
4k machine - test-aws-ubuntu1804-armv8-1
Useful facts

Correction: This fix appears to claim that this issue is resolved on jdk15, so the tests shouldn't have to be fixed.
It seems a more prudent use of time to simply exclude the tests (from jdk14 - aarch) while we wait for jdk14 to expire. If we see the problem on jdk15, then we can spend time on a fix.

PR to exclude failing tests here: https://github.com/AdoptOpenJDK/openjdk-tests/pull/1905

Closing this for now. If the issue returns on jdk15, we can re-open.

This is an error I am currently having with the latest adoptopenjdk build for aarch64 PCs (https://adoptopenjdk.jfrog.io/artifactory/rpm/centos/7/aarch64/Packages/adoptopenjdk-16-hotspot-16.0.1+9-3.aarch64.rpm). How do I go about reopening and documenting this? The server in question is a free tier machine on oracle cloud, where I install this rpm manually onto a oracle linux OS.


query-handbrake-log appears to have an issue with OS X Automator

Don,
Not quite sure this is a bug, but thought I'd submit it.
Here's the situation, using OS X Automator's Run Shell Script with the query-handbrake-log command:
/usr/local/bin/query-handbrake-log t "/Users/hayward/Desktop/Transcode/Logs" > "/Users/hayward/Desktop/time.txt"
you receive the following error:
The action “Run Shell Script” encountered an error.
/usr/local/bin/query-handbrake-log: invalid byte sequence in US-ASCII
I found this discussion, which seems to be relevant:
https://github.com/CocoaPods/CocoaPods/issues/1538
From that discussion, if I structured the commands in the Run Shell Script as:
export LC_ALL="en_US.UTF-8"
/usr/local/bin/query-handbrake-log t "/Users/hayward/Desktop/Transcode/Logs" > "/Users/hayward/Desktop/time.txt"
everything works as it should.
If you need anything else, please let me know.
Thanks,
Brent

@bmhayward Wow, that is weird. Thanks for opening this.
I'm not sure this is a bug in query-handbrake-log so much as a peculiar interaction between the Ruby interpreter (or libraries) and Automator.
Are there any non-ASCII characters in the filenames you're passing to query-handbrake-log?

Don,
Nothing out of the ordinary as far as non-ASCII characters. What I posted is the exact set of commands I tested with.
When I first tried using query-handbrake-log with Automator nothing would happen, so I just moved on. I had some time today to go back dig in and that is what I found. What I wanted was a way to drop a single log file or a folder of log files on an app and process. Initially, I thought it would just take a few minutes to bang that out with Automator.
Anyway, at least I now know how to use it with Automator and maybe save someone else the trouble in the future.
Thanks,
Brent

@bmhayward OK, I think this is an Automator issue then. Basically, it's requiring input in UTF-8 format. And if it somehow can't figure out whether something is definitely UTF-8, then it just barfs.  And yet another reason I never use Automator.
So, I'll close this as "won't fix" for now and chalk it all up to experience. :) However, I may eventually figure out how to add this to the "FAQ" in the "README" document.
Thanks, again!

@bmhayward Thinking about it again, this is not really a frequently asked question since you're the only one who's ever asked me about this. :) Which is impressive, BTW.
So, maybe this should go in a new "TIPS" sub-section within the larger "Guide" section of the "README" document? I don't know yet but I'll noodle on this some more.

@donmelton TIPS sub-section sounds great.
Here is the completed OS X app to try out.
Transcode Log Analysis.zip
I will post it to github as part of the entire workflow solution sometime in the next week. In the mean time, you can read about the workflow solution in the attached TL;DR.
TL;DR.zip
Any suggestions, corrections or additions are greatly appreciated.
Thanks again,
Brent

@bmhayward Very cool. I look forward to seeing it online. :)

@donmelton closing the loop, Transcode Log Analyzer.app is now available as part of the Transcode workflow solution, https://github.com/bmhayward/Transcode.git
Any thoughts, suggestions, ideas, bugs, etc. would be greatly appreciated :-)

@bmhayward the ReadME alone is pretty awesome, can't wait to give your Transcode app a try. I must've missed the part where it's actually an all-in-one awesomeness for all your transcoding needs. Are you on Gitter? If not, might I suggest syncing up your repo to it so we all can chat on there without clogging up issues?

@hughker I'm now on Gitter, https://gitter.im.bmhayward/Transcode.
Thanks for the tip!


Support for Paging Library

Thank you for this library, it really comes handy with a great set of features. I was wondering if there are any plans for implementing an adapter like the newly introduced in Android Components Paging Library.
Thanks in advance
Cheers

@pkoutsovasilis, I can't say anything on that library yet, but normally the updateDataSet with granular notifications/animations does the job for the differences, so you can use it.
Any library of this kind, will have a place in the extensions part on top of FlexibleAdapter to maintain the same features.
When you receive new list, using updateDataSet, the expected list will substitute the current one!
If you want to add new items and use pagination you should use the "loadingMore" pattern I have exposed.

Thank you @davideas for you response, i surely missed out the "loadingMore" pattern, so i guess the same logic of the Paging Library is sort of already implemented


Print the server's scheme when logging that it's reachable

Not sure if we can make this change - but I think it would be nice to log the scheme as described in #578. We can also assume HTTP (not HTTPS) here, but I'd prefer logging it correctly.

We can also consider detecting if the IP is :: or <IP_ADDRESS> so that we can print more accurate info

Unfortunately no, as TransportServices does not have a handler, and these handlers are only applicable for child channels of the server

@adam-fowler my idea was that I want to (eventually) make it explicit how to reach it from various devices. Should've explained that in the PR.
So <IP_ADDRESS> should also show your LAN IP

@Joannis But bind address <IP_ADDRESS> is different from <IP_ADDRESS>, ie one accepts connections from any address while the other accepts only connections from the localhost. We shouldn't be pretending they are the same.
Maybe if the bind address is set to <IP_ADDRESS> then we can add an additional log that says it is accepting connections from anywhere

I understand what <IP_ADDRESS> does, but it's not common knowledge. By adding logs about the available addresses to connect to, I'm hoping to address that knowledge gap so that people understand how this works.

An excellent example that I've had people ask time and time again:

How do I connect to my backend from an iOS app? <IP_ADDRESS> doesn't work.

You need an additional log item then, because the people who do know what is going on are going to be a little confused when there server logs says the server is listening on <IP_ADDRESS> but accepting connections from anywhere.

An excellent example that I've had people ask time and time again:

How do I connect to my backend from an iOS app? <IP_ADDRESS> doesn't work.

Also can we make it clear which situation we are trying to fix here, as your change doesn't fix this.


Sketch 3.4 not working

For me is not working. It ask for new label but then nothing happens

Hi Gabriel, I'm not having any issues, though I know there are some problems with 3.4 and multiple documents (sometimes the plugin actually targets a file other than the one that's focused in the foreground).  Can you restart Sketch, open only one document then retry?  Was it working for you prior to the 3.4 upgrade?

I will try by uninstalling and re-installing because I see that other plugins are not working, could be something in my configuration and not related your plugin. Sorry Thanks a lot!

Was something with my configuration. I reinstall all and now is working fine ;)

Glad you got it working!