When ServiceEntry are with the identical host name, try to autoAllocateIPs same IP

Signed-off-by: Yann Liu<EMAIL_ADDRESS>provide a description of this PR:
To help us figure out who should review this PR, please put an X in all the areas that this PR affects.

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Please check any characteristics that apply to this pull request.

[ ] Does not have any user-facing changes. This may include CLI changes, API changes, behavior changes, performance improvements, etc.

Hi @yannuil. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/ok-to-test

@yannuil: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test release-notes_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@yannuil: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test release-notes_istio

unit-tests_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@yannuil: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test release-notes_istio

unit-tests_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test unit-tests_istio

lint_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test lint_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@yannuil: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test release-notes_istio

unit-tests_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test unit-tests_istio

lint_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test lint_istio

gencheck_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@yannuil: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test release-notes_istio

unit-tests_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test unit-tests_istio

lint_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test lint_istio

gencheck_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test gencheck_istio

analyze-tests_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test analyze-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

I could not understand what is the caveat currently? In other words what's the problem now?

@howardjohn Thank you for your time to review this PR. I understand the cost of adding a new test suite. I tried to use the suite to create a reproducer. I will remove the test suite.
@hzxuzhonghu I will explain my findings.
In an istio setup with both DNS_CAPTURE and DNS_AUTO_ALLOCATION set,  two SEs with the same host name but different port are applied. This function will try to give each SE a virtual IP. In my test, <IP_ADDRESS> and <IP_ADDRESS> are used.
In a pod with the mesh, when a request is made to istio.io, which defined in the SEs, a DNS look up will be performed first to resolve the domain name to one of these two addresses. Depends on the order in which this "internal" DNS records get added in the istio-agent, the internal address resolved by the DNS is consistent.
In my test cases, there are a 80 normal TCP port and 443 TLS port. Based on my test case, <IP_ADDRESS> is related to the mesh external cluster with a TCP port 80. It seems to be the "default" DNS record for domain istio.io in the mesh, a HTTPs request made to https://istio.io will fail to get routed by istio and try to "escape" the mesh. This will result a 500 upstream error from envoy.  My understanding is when envoy tries to find <IP_ADDRESS> with TLS port 443, it could not find a matching route. In the ALLOW_ANY case, the request will be let go into the wild.
The sidecar_simulation_test seems to be testing if the rules in envoy is doing correctly. In this case, the rules are doing what is supposed to.
What is the best way to test this?

two SEs with the same host name but different ports

Currently one address is allocated to this. Then what's the issue?

@yannuil: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

unit-tests_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test unit-tests_istio

lint_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test lint_istio

gencheck_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test gencheck_istio

analyze-tests_istio
bc171d52893f9708f82a2afedb38aa7b99ea5e2e
link
true
/test analyze-tests_istio

release-notes_istio
bab1e0301c1ed1b846d7641496619a9c7edfc460
link
true
/test release-notes_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu Thank you for your time.

two SEs with the same host name but different ports

Currently, there will be two virtual IPs assigned to these two SEs. It will cause an request to one of the SEs to be blocked. I try to show the problem with this test set. In the test set, one of the sub test will fail.
This PR will give a single virtual IP for these two SEs, then traffic will go through as expected.

Hi everyone,
I am also facing the problem that this PR is trying to resolve.

Currently one address is allocated to this. Then what's the issue?

When I create two distinct ServiceEntry k8s objects :

First referencing www.google.com:443 -> Istio VIP <IP_ADDRESS>
Second referencing www.google.com:80 -> Istio VIP <IP_ADDRESS>

When I try to curl https://www.google.com:443, the VIP resolved is the one associated with the second ServiceEntry object (<IP_ADDRESS>).
Thanks you @yannuil for implementing a resolution.
Thank you @hzxuzhonghu for your time on this PR.

I did not understand this issue well, will try to take a deep look

@hzxuzhonghu thank you for your time.
The existing code, with DNS_AUTO_ALLOCATE configured, allocates multiple virtual IPs for any given hostname, when the same hostname is referenced in multiple ServiceEntry resources.  When using DNS_CAPTURE, only one IP will be returned for the DNS request and this will cause issues with routing. This patch ensures that the same virtual IP is used for any given hostname, regardless of how many ServiceEntry resources reference the hostname.  In other words it ensures there is only ever one virtual IP defined for a hostname.
@howardjohn I have studied the unit test example you provided. If I understand it correctly, it tests the envoy's cluster definition with the istio behavior. I think the cause of the issue is mainly due to DNS.
Do I need to add any release notes?

@yannuil: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
8b2cf38c50744303eccf6b1812f5f41385435698
link
true
/test release-notes

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

BTW, why remove the integration test?

I removed the integration test because John suggested me to explore a different type of test in his review.
I could not figure out how to construct an unit test for this problem. I am happy to add the integration back in.
@howardjohn could you give me some suggestions?

And for release note like this one?

Yes, please add one

@yannuil: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
8b2cf38c50744303eccf6b1812f5f41385435698
link
true
/test release-notes

integ-telemetry-istiodremote_istio
84a55e8be1d9d1aeaf68a4891b0174b8153c23d1
link
true
/test integ-telemetry-istiodremote

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@yannuil: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-telemetry-istiodremote_istio
84a55e8be1d9d1aeaf68a4891b0174b8153c23d1
link
true
/test integ-telemetry-istiodremote

lint_istio
84a55e8be1d9d1aeaf68a4891b0174b8153c23d1
link
true
/test lint

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@yannuil: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-telemetry-istiodremote_istio
84a55e8be1d9d1aeaf68a4891b0174b8153c23d1
link
true
/test integ-telemetry-istiodremote

lint_istio
f7c5d0035d89cea65b17be2d7ff2b92cd1dfba55
link
true
/test lint

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@yannuil: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-telemetry-istiodremote_istio
84a55e8be1d9d1aeaf68a4891b0174b8153c23d1
link
true
/test integ-telemetry-istiodremote

lint_istio
9c0022bbe2e217ad28d479af8797d174ab8f5b1b
link
true
/test lint

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio/release-managers

@ericvn any idea why the cherry-pick PR's weren't created for this one?

@ericvn: new pull request created: #41116

In response to this:

/cherry-pick release-1.15

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@GregHanson Sometimes there is an issue with cherry-picks. We had an issue for that at one point (would have to see if it's still open). I've also noticed that things sometime run much later, ie. I see multiple cherry-picks attempted. I wonder if some backend job gets stalled and then things start up later.
Tried kicking a CP using the command to see if that does something.

/cherry-pick release-1.13

@ericvn: #40165 failed to apply on top of branch "release-1.13":
Applying: Add integration tests for DNS AUTO ALLOCATION
Applying: Reuse the same ip for services with the identical hostname
Using index info to reconstruct a base tree...
A	pilot/pkg/serviceregistry/serviceentry/controller.go
Falling back to patching base and 3-way merge...
Auto-merging pilot/pkg/serviceregistry/serviceentry/servicediscovery.go
CONFLICT (content): Merge conflict in pilot/pkg/serviceregistry/serviceentry/servicediscovery.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0002 Reuse the same ip for services with the identical hostname
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.13

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ericvn: new issue created for failed cherrypick: #41118

In response to this:

/cherry-pick release-1.13

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

I did go back and look at the last page of commits on the main istio/istio branch and any PRs with the cherry-pick labels on them, did attempt to do the cherry-picks.


RE; Possible GDPR issue?

Like already partially discussed in the comments of #19 .
The old data (infraction history, notes, etc) are being stored for users that left and this might go against the GDPR laws.
I ain't a lawyer so might be worth looking into.

Last time we discussed this internally the conclusion was that it's all good.

@daanbreur thank you for raising your concerns regarding data privacy. Hackster doesn't store any PII data (the same as Noahbot before it), we only keep discord ids as you can see for yourself (e.g. https://github.com/hackthebox/Hackster/blob/main/src/database/models/ban.py).
@HeckerBirb thanks for jumping in, I wasn't aware of that discussion.
Please keep in mind that we take data privacy very seriously at Hack The Box so we are constantly evaluating the data we store and process.

I would not care about this cause its a very useful tool to have. I personally thought that a Discord ID would be in the same category as PII. But apparently not.
Thank you for checking for me though 👍


Fix verify tokens with the correct bonus token

The real bonus token should be the first unmatched token. For example, the draft_token_ids is [1, 2, 3, 5, 7], and the target_token_ids is [1, 2, 3, 4, 6, 8]. Then, the matched token should be [1, 2, 3], and the bonus token should be [4] because the target model will output [4] based on the input [1, 2, 3] in the next round generation. So we can select the bonus token [4] in this round without any precision regression.
It will increase the performance from 89 tokens/s to 110 tokens/s in typical_acceptance_sampler in A100 single card with (num_speculative_tokens=2, max_num_seqs=1, model="meta-llama/Llama-2-7b-chat-hf", speculative_model="Felladrin/Llama-68M-Chat-v1"). The outputs are exactly the same before and after my changes.
Do you mind review this PR? @cadedaniel
cc @LiuXiaoxuanPKU

After second thoughts, I feel this is a good optimization. We can actually skip rejection sampling in the greedy decoding case.

After second thoughts, I feel this is a good optimization. We can actually skip rejection sampling in the greedy decoding case, which explains the speedup you get. I also suggestion trying the flashinfer backend and you should also see good speedup.

Thx Xiaoxuan, so do you think this optimization idea is qualified to merge to vllm. We treat this as a platform independent optimization(orthogonal w/ backend optimizations like flashinfer), which can benefit other device backends like CPU/XPU, and we see similar performance issue.

This change is to align the verify token function to the transformers speculative sampling algorithm, it always selectes the next sample token from the target model.

This change is to align the verify token function to the transformers speculative sampling algorithm, it always selectes the next sample token from the target model.

From reading the code, it seems it adjusts the distribution and resamples (line 4195 - 4203)?

After second thoughts, I feel this is a good optimization. We can actually skip rejection sampling in the greedy decoding case, which explains the speedup you get. I also suggestion trying the flashinfer backend and you should also see good speedup.

Thx Xiaoxuan, so do you think this optimization idea is qualified to merge to vllm. We treat this as a platform independent optimization(orthogonal w/ backend optimizations like flashinfer), which can benefit other device backends like CPU/XPU, and we see similar performance issue.

Could you double check the correctness here? If the optimization can pass the rejection sampling tests, yeah happy to review and get it in.

From reading the code, it seems it adjusts the distribution and resamples (line 4195 - 4203)?

Yes, but the point is the output will always contain the first unmatched token (which is the next sample token from the target model).

Could you double check the correctness here? If the optimization can pass the rejection sampling tests, yeah happy to review and get it in.

The error comes from sampling. We can not guarantee the output will be all matched even if the target model is the same as the draft model because sampling will introduce random factors. So I disabled sampling by setting temperature=False so we can make sure all tokens will be matched.

The error comes from sampling. We can not guarantee the output will be all matched even if the target model is the same as the draft model because sampling will introduce random factors. So I disabled sampling by setting temperature=0 so we can make sure all tokens will be matched.

This is the intention because you cannot force users to set temperature=0. That's why @LiuXiaoxuanPKU suggested we could bypass rejected sampling when temperature=0, but we cannot remove rejected sampling.

This is the intention because you cannot force users to set temperature=0. That's why @LiuXiaoxuanPKU suggested we could bypass rejected sampling when temperature=0, but we cannot remove rejected sampling.

We cannot expect the output length of speculative decoding if sampling is applied, it could be in [1, num_speculative_tokens]. How about changing the test to verify the output token number in a reasonable interval instead of finishing after 2 steps?

OK, I removed temperature=0 but changed the max_tokens and num_speculative_tokens to avoid tests failed by sampling. Please let me know your opinion about this test.

I'm confused. It seems this PR removes rejection sampling. Then how do you do speculative decoding with temperature != 0?

I'm confused. It seems this PR removes rejection sampling. Then how do you do speculative decoding with temperature != 0?

I didn't remove rejection sampling, just removed recovered_token since it is not needed. The newly defined bonus_token covers the recovered_token case. Besides, I kept the temperature in default value in the test.

The recovered token should not be from the target token ids if temperature != 0. It should be sampled from a new distribution. Correct me if I misunderstand anything.

The recovered token should not be from the target token ids if temperature != 0. It should be sampled from a new distribution. Correct me if I misunderstand anything.

I see your point, the recovered_token is selected from a new distribution, but I just select it from target_token which is more convenient and also makes sense. Please let me know if you want me to revert it to keep the original recovered_token, thanks.

The recovered token should not be from the target token ids if temperature != 0. It should be sampled from a new distribution. Correct me if I misunderstand anything.

I see your point, the recovered_token is selected from a new distribution, but I just select it from target_token. It is more convenient and also makes sense.
I know adjusting the distribution of target_prob is from the original paper, but I didn't see any advantages compared to just selecting from target_token_ids, and it also introduces a customized multinomial function and some overheads.
Please let me know if you want me to revert it to keep the original recovered_token from the adjusted target probs, thanks.

I don't think change original paper algorithm is a good idea without data proving, and I don't think change behavior is the target of this PR. This PR's target is performance optimization. @jiqing-feng, pls only optimize performance for temperature == 0, and don't change the logic of others.
You can submit another PR or issue if you wanna discuss it, keeping the PR only for one thing is better for cognition burden and make things move fast forward.

The recovered token should not be from the target token ids if temperature != 0. It should be sampled from a new distribution. Correct me if I misunderstand anything.

Yes, you were right. I have fixed the recovered token and selected it from the new distribution (torch.clamp(target_prob-draft_prob), min=0). Please take a review. Thx.

Hi @LiuXiaoxuanPKU . I checked the rejection sampler codes in detail and found there is no need to change it because you can get the correct recovered token ids.  Only 2 things:

The _multinomial function just selects the tokens with the largest probability which is different with torch.multinomial
The recovered token ids can not exist with bonus token ids at the same case because it can only return 1 token from Mp

I am okay with the 2 little differences btw vllm and the original paper because I cannot get speed-up on rejection sampling with this PR, but it could get significant speed-up on typical acceptance sampler. So I opened another PR to only change typical acceptance sampler, see #8562 . Please take a review, thx.


call PasswordBoxOnPasswordChanged after initialization

to support passwordboxes which are initially not empty
fixes #213

:cool: ...I will test over the next couple of days.

This fix doesn't quite work.  There is another issue outstanding with text boxes, which is pretty much the same thing, and after testing your fix I could still reproduce using the steps described here: https://github.com/ButchersBoy/MaterialDesignInXamlToolkit/issues/193

Ups, I have missed this bug report. I have amend the commit which fixes both issues.


[DevOps] Publish Dapr "Hello WASM" app into Meshery Catalog

Current Situation
Meshery Catalog has a relatively small number of published designs.
Desired Situation
Meshery Catalog is populated with a variety of published designs, including Dapr sample apps, representing any number of technologies in the designs and representing different categories of designs.
Difficultly Level: Beginner
This is a beginner-level issue designed to both help introduce you, the contributor, to additional Kubernetes concepts and DevOps practices, and to aid in creating a robust catalog full of cloud native design patterns and sample apps in the Meshery Catalog.
Skills Needed

No coding skills needed.
Awareness of Kubernetes.

Implementation

Identify a Dapr sample app

Take this app - https://github.com/dapr/samples/tree/master/hello-wasm

Import App

Import the sample application using mesheryctl app import command, following this guide.

Publish App

Publish your sample app into the Meshery Catalog. Guides: Using Meshery UI or Using MeshMap.

[Optional] Open App in Playground

Access Meshery Playground and open your app in MeshMap.

[Optional] Share your App

Be social and share your imported apps on twitter or linkedin and the Meshery project accounts will repost.

Repeat

Import and publish one or more sample apps.

Acceptance Criteria

"Hello WASM" has been imported into Meshery and published into the Meshery Catalog.

Contributor Guides and Handbook

🛠 Meshery Build & Release Strategy
📚 Instructions for contributing to documentation

Meshery documentation site and source

🎨 Wireframes and designs for Meshery UI in Figma
🙋🏾🙋🏼 Questions: Discussion Forum and Community Slack

I'd like to work on this issue.

i would like to work on this

Hey @leecalcote, I'm having trouble with the mesheryctl app import command. It's saying it's not valid. Also, the guide link you gave seems broken. Could you help me find the right direction?

Hey @leecalcote, I'm having trouble with the mesheryctl app import command. It's saying it's not valid. Also, the guide link you gave seems broken. Could you help me find the right direction?

Hi @prasad89 it would be helpful to contributors if you linked a screenshot of the error you encountered also perhaps a step-by-step guide on the process you took that led to the error so it can be reproduced?

@criticic I have assigned this issue to you, if you haven't already do take a look at the CONTRIBUTING.md for steps on how to effectively make a contribution.


refactor(interactive): Make runtime module header only

Move all implementation code from .cc to .h for the runtime_adhoc module.

Reason for refactor: We aim to make the operator's implementation header-only and templatized, enabling other storage systems to easily run ad-hoc queries.
Cost: Converting the runtime module to header-only may slow down compilation?

Fix Issue #4243

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 58.78%. Comparing base (0052971) to head (72402e7).
Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #4244       +/-   ##
===========================================
+ Coverage   34.33%   58.78%   +24.45%
===========================================
  Files         126       12      -114
  Lines       13235      592    -12643
===========================================
- Hits         4544      348     -4196
+ Misses       8691      244     -8447

see 126 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0052971...72402e7. Read the comment docs.

Closed since will not be merged.


improve a star

@soneryaldiz @854768750
The sym factor bug is fixed in this branch. Here is the results:

However, the symmetry net vin, vip in this design is connected with terminal and their pins are parallel, so can not  be routed symmetrically.

Thanks @Lastdayends . This is not really the best example for symmetric net constraint. I suggest to try on the high_speed_comparator which does have more interesting constraints.

Also, the existing routes (aka preroutes) could be ignored for symmetry calculation. The router should really focus on the symmetry of the new wire segments added. The symmetry of preroutes is the responsibility of whatever generated that subcircuit.

@parijatm
CI fails to the limitation of runtime. May I know what testcases are there in those CI testing?
I can run those testcase locally to see what is the issue.

Was able to reproduce segmentation fault on current_mirror_ota and ldo_error_amp locally.
By the way, the worst case runtime of our integration tests was 16 minutes previously. For reference it was < 10 minutes for the (BUFFER or OTA or Powertrain) integration test. So 30 minutes of no output should be considered a bug. Please profile.


ValueError when running make clean install

in clean new Node.js environments when running the command:
make clean install

it fails with the error:
ValueError: "<path_to_parent_directory>/elyra/packages/application" is not a valid npm package

you can get past this issue by globally installing some NPM packages:
npm install -g<EMAIL_ADDRESS>npm install -g<EMAIL_ADDRESS>npm install -g<EMAIL_ADDRESS>npm install -g<EMAIL_ADDRESS>
however, this only works around the problem.
the problem appears to stem from the clean script in the NPM packages which is deleting the  package's node_modules directory.
clean is probably not needed in the package.json since the Makefile is already cleaning up the packages/directories. however, if clean is needed it probably shouldn't be removing the node_modules directory.

Fixed by #420


[Bug]: Assume Role Region is not valid for GovCloud Regions

Steps to Reproduce

prowler aws --region us-gov-east-1 --role arn:aws-us-gov:iam:::role/ --log-level DEBUG
aws
kubernetes
An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid

Expected behavior
Assume role works properly with an applicable govcloud region. This is occurring because the AWS region being set does not use the region being set as a CLI argument.
Actual Result with Screenshots or Logs
{"timestamp": "2024-06-11 20:04:10,218", "filename": "aws_provider.py:872", "level": "CRITICAL", "module": "aws_provider", "message": "ClientError[855] -- An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid"}
How did you install Prowler?
From pip package (pip install prowler)
Environment Resource
EKS
OS used
Linux
Prowler version
Prowler 4.2.4
Pip version
pip 23.0.1
Context
No response

Hi @collin-miller I'm actively checking this issue. I'll get back to you soon.
Thanks for using Prowler 🚀

Hi @collin-miller could you please test it again with the above PR? You can clone the branch PRWLR-3913-bug-assume-role-region-is-not-valid-for-gov-cloud-regions-4244 and let us know if this works for you.
Thanks!


Website defaced

hello,
I have a website called https://unemundem.al and on 23rd of november
it has been defaced by the ip <IP_ADDRESS>
Basically the attacker has been able to perform a sql injection on the
wordpress installed and the website was forwarding to another
location.
On wp_options table it has change the values of home and site url to
the follwing url
https://port.lovegreenpencils.ga/det.php?pit=222&sid=5&yuid=243634&
I have restored it but i'm sure it will try again.
you can find the log from  mod_sec here https://pastebin.pl/view/raw/842d1d82
and it seems this kind of attack is not handled very well
can you please advice?
thank you

Hi @albgen
Is not the first time that I see this technique on WordPress. Reading your logs it seems that your Nginx reply always with 403 Forbidden for each attempt except for the first log. As you can see, in the first log, Nginx reply with 500 but the request seems to come from the authenticated user "bbamiokjopu" (am I correct?). Is it possible that someone has guessed the password of bbamiokjopu via a brute-force attack and then replaced your front page?

it seems this kind of attack is not handled very well

In your logs I don't see any response different than 403 or 500, am I wrong? I think that we should see a different response from your Nginx on a successful SQLi attack.
Could you please provides more information about your modsecurity configuration and CRS version and configuration?
I'm going to close this issue just because it seems not related to a specific CRS problem, but please give me more information to understand better what happened.

Hi MiddleBlue,
yes that user exists and was also an administrator but i have not created it. Most probably is a user that the attacker has created. I have deleted it.
I think that also the audit log has not recorded the requests that has caused the issues...
The version of modsec is the latest one from the main branch and the nginx is the 1.19.4.
The crs ruleset is the one from https://github.com/SpiderLabs/owasp-modsecurity-crs.git
thank you
the server is a centos 7 vps always updated...
if you need other info let me know. Maybe the log for the website on that precise moment?

Hi @albgen
the log could be useful and if you want I can give it a look. It's better to not make them public in this issue, can you send them to<EMAIL_ADDRESS>? Also, have you got a list of all WordPress plugins+versions active on your website?


fix(broccoli-typescript): properly parse compilerOptions

Previously, the settings in tsconfig would be ignored by
the broccoli-typescript plugin, because the whole config's
JSON was being provided to convertCompilerOptionsFromJson()
instead of just the expected compilerOptions.

Lgtm


Ng Generator does not always save components in correct directory

like #6431
Bug Report or Feature Request (mark with an x)
- [x] bug report -> please search issues before submitting
- [ ] feature request

Versions.
@angular/cli: 1.2.7
node: 6.10.1
os: darwin x64
@angular/animations: 4.3.3
@angular/cdk: 2.0.0-beta.8
@angular/common: 4.3.3
@angular/compiler: 4.3.3
@angular/core: 4.3.3
@angular/flex-layout: 2.0.0-beta.8
@angular/forms: 4.3.3
@angular/http: 4.3.3
@angular/material: 2.0.0-beta.8
@angular/platform-browser: 4.3.3
@angular/platform-browser-dynamic: 4.3.3
@angular/router: 4.3.3
@angular/cli: 1.2.7
@angular/compiler-cli: 4.3.3
Mac OS Sierra
Repro steps.
It's hard to give repo steps because I haven't discovered exactly what I have to do to reproduce.
However, once the behaviour starts for a project, it continues to behave this way. The only way I have found to fix it is to start a completely new project and copy all my code over.
The issue is well described in #6431.
In brief,

I cd to a module directory in my project eg: app/admin

I create a new component, eg ng g c users

Instead of creating the new component inside app/admin/users, it creates it inside app, eg app/users
In addition it creates an incorrect import statement in app.module, eg:
import { UsersComponent } from './admin/src/app/users/users.component';
I have seen this happen twice now. I can be working with a project for months and months with it working correctly, then one day it just stops working the way it should. The only way I have been able to fix it is to generate a completely new project and copy all my code over to the new project.
The log given by the failure.
N/A
Desired functionality.
It should create the component inside the admin directory, eg:
app/admin/users
and it should create a well formed import statement in admin.module, eg:
import { UsersComponent } from './users/users.component';
Mention any other details that might be useful.
When I create a brand new project it does what I want it to. I have no idea what I do to provoke this behaviour. I have gone through the following files with a fine tooth comb and can't find anything wrong:

.angular-cli.json
package.json
tsconfig.json
src/tsconfig.app.json

I can't think of anywhere else to look.

For the wrong import path Issue #7135 and Pull Requests #7284 and #7090 are related here. Already reported since versions 1.2.x. This is just a duplicate of an existing issue. Fixes and new implementation are already on progress being done by @Brocco

The reason it doesn't work as intended is because you're using cd. Use ng in the root of your project and it'll work as intended.

@grizzm0 Yes it will work on the root of the project. But what if you have complex directory structure inside the src/app directory? It should automatically be done the CLI tool itself to correctly resolve the path. This was a regression appeared on what i don't know version.

@exequiel09 I was simply explaining why the error happens and how he can solve it for now. :) This is how ng is supposed to be used. From the looks of the other issues it seems like it'll be "fixed" tho.
I don't see what a complex folder structure has to do with anything other than auto-completing with tab in the terminal.

@grizzm0 this is not how Ng is supposed to work. For the last several months it has worked as I wanted it to in my project. If I start a brand new project, it works how I want it to.
What I don't understand here is why it just starts happening after it has been working perfectly for months on end.
In terms of a complex folder structure, try writing
ng g c contract/subcontract/subcontract-new/supplier-select/new-supplier/new-supplier-with-name/new-supplier-result
I know at least one developer who discarded angular in favour of react on the basis of this issue alone.

@exequiel09 the issue I am having is actually not the same as #7135.
I am finding it does create the wrong path in app.module but not in the same format as above.
I could actually live with this behaviour although it would be annoying.
The bigger issue is that it creates the component in the wrong directory as described above. This is killing me as it used to work and I'm not sure why it stopped.

We just ran into the src in path thing...  Now, I've NEVER used ng generate outside of the root directory, and I've ALWAYS used directory paths.
It won't find the module blah when generating the component unless you tell it to create the module in 'blah/' and refer to the module with its name blah...
Sorry for the blathering on, with the blah blah and all.
Bottom line, if you then start doing cd into directories, it's not a good thing, it'll try relative pathing but then put src/app in there and screw it up.
Is this new? I dunno. But that's the behavior I see.

i have similar issue. when you use ng g c inside the src directory the generated path in the app.module.ts file is not correct.

Any update on this issue?

We'd really like to know how you are going with this issue. When we are working on a complex app it would make it so much easier.
We all know some things can be really difficult to achieve but it was marked as urgent on 24 August.
Any sort of update would be much appreciated.

FWIW: for me the path is duplicated. OS: Win 10, node 8.9.2, ng-cli: 1.6.0:
[duricz@msgn07241 ~/temp/my-new-app]$ ng g m new-module
  create src/main/frontend/src/main/frontend/app/new-module/new-module.module.ts (193 bytes)
[duricz@msgn07241 ~/temp/my-new-app]$

FWIW my root path in .angular-cli.json is set to src/main/frontend;

Hi guys. Can you tell us ANYTHING about progress on this issue.
It was first raised in August. It is no 21st December. It is marked as urgent. But we haven't heard anything???

We have now just ticked over to February 2018.
It would be great if you could tell us SOMETHING, ANYTHING.
This is a priority 1, severity 5 issue

My problem was fixed between 1.6.0 and 1.6.4, not sure which one did it.

That's great @zladuric. Angular CLI is an incredibly good and useful product and I know the team are continually working on it. The problem I have here is the issue was marked a Priority 1 and Severity 5 back in August last year.
Since then we have not heard a peep.
I just find that really strange.
It may well be that the problem is just too hard to solve and I'll just have to get used to creating everything from the root directory – although that would be a real pain. Anyway, if that's the case, it would be nice to hear something about it. The issue is still marked as Open with the highest priority and severity but we hear nothing about it. I just don't get that bit. @filipesilva

@hansl just interested. What's the priority on this issue now?

@chriscurnow Priority is now based differently, so we don't put it ourselves anymore. Since @brocco is assigned to your issue we'll get to it soon.

@hansl Thanks 😄

@hansl :
It's now May 2018. I reported this in August 2017. Last update here was Feb this year. In Feb you said you would get to it soon.
Look I know the CLI team has been working hard and has made some fantastic additions and fixed up a lot of stuff. I'm not complaining but it would be nice to have some sort of update now and again 😄


do not translate words from a given vocabulary

e.g., do not translate LLM to 法学硕士. Leave it  as LLM.
e.g., do not Transformer LLM to 变压器. Leave it  as Transformer.

if we could not set such a non-translating vocab for translators (google, tencent ... )
the only way is to remedy it replace the (wrongly) translated words to the origin EN word after translation ...

An impl could be:
    from functools import reduce
    replace_dict = {"法学硕士": "LLM", "变压器": "Transformer", "代币":"token"}
    text_final = reduce(lambda text, kv: text.replace(*kv), replace_dict.items(), text_final)

Another (downstream way) is to proc the translated main.tex file:
#!/bin/bash

declare -A replace_dict=(["法学硕士"]="LLM" ["变压器"]="Transformer" ["代币"]="token")

while read -r line; do
    for key in "${!replace_dict[@]}"; do
        line=${line//${key}/${replace_dict[$key]}}
    done
    echo $line
done < main.tex

iter all .tex files of directory dir and proc (as we could not in general not know which .tex is the main tex file?):
#!/bin/bash

declare -A replace_dict=(["法学硕士"]="LLM" ["变压器"]="Transformer" ["代币"]="token")

find dir -name "*.tex" | while read -r file; do
    while read -r line; do
        for key in "${!replace_dict[@]}"; do
            line=${line//${key}/${replace_dict[$key]}}
        done
        echo $line
    done < "$file"
done

Thank you for reporting issues to us. Since we are a general translation tool instead of a tool only working for CS or DL, we think it might be better to leave it as what it is temporarily.
We could consider a functionality as a "user dictionary", by asking the users to manually define the "popular vocabulary". The only thing user need is to load a list of vocabulary. Similar to your solution here but more systematic and friendly to users. @SUSYUSTC


Longer loading times for index headers with Thanos Receive compared to Thanos Sidecar

Thanos, Prometheus and Golang version used:
Thanos: v0.36.1
Prometheus: v2.53.1

Object Storage Provider:
Ceph Cluster
What happened:
We noticed that our Thanos Stores, which uses an Objectstorage as storage backend, has significant longer loading times per block/index when the are generated from the Thanos Receiver compared to Thanos Sidecar.
We then checked the two buckets, which both have a similar size and the blocks are also similar in size. For Tenant 1 we have 3112 index files with an average of 348MB. For tenant 2 we have 2993 index files with an average of 309MB per file. But this does not explain why Tenant1 takes on average 2-3 times as long to load per Index Header compared to Tenant2.
What you expected to happen:
Regardless of which data source the blocks are written from, according to the documentation they should make no difference to how the Thanos Store generates the index headers.
With similarly large buckets and a similarly large number of blocks to be loaded, almost the same speeds should be achieved when generating the index header/block.
How to reproduce it (as minimally and precisely as possible):
Compares a bucket which was filled with Thanos Receive data and one which was filled with Thanos Sidecar data. Then start a store for the receive bucket and a store for the sidecar bucket. You should now be able to recognise the differences.
Full logs to relevant components:

Receiver Blocks
Sample Output from thanos tools bucket inspect
|            ULID            |         FROM         |        UNTIL         |     RANGE     |   UNTIL-DOWN   |  #SERIES  |   #SAMPLES    |  #CHUNKS   | COMP-LEVEL | COMP-FAILED |                                                    LABELS                                                    | RESOLUTION |  SOURCE   |
|----------------------------|----------------------|----------------------|---------------|----------------|-----------|---------------|------------|------------|-------------|--------------------------------------------------------------------------------------------------------------|------------|-----------|
| 01J5ADJ0YDNACNJDHHZQ8569R9 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,390,745 | 6,632,884,192 | 53,817,971 | 4          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-1,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5AE1S6B4B3SQT2N2P90RFZE | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,470,793 | 6,822,144,336 | 54,662,046 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-20,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AENP6H3APP8HMAZ4YB1Q69 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,523,468 | 7,296,872,037 | 58,401,675 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-19,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AF6JVAACJHSJKDPD0PWT9S | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,473,601 | 6,940,360,938 | 55,575,188 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-22,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AFKCBDFXHGASHFXX0X53DN | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,472,729 | 6,954,088,819 | 55,707,100 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-16,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AG38NNJH1N6TYF2V1F9XE3 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,499,885 | 6,926,953,022 | 55,516,517 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-0,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5AGFCSBEHH2T4Z4W0GM2575 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,391,613 | 6,727,877,419 | 54,527,334 | 4          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-15,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AGWMQJX829DA1KFV7EQKSY | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,296,077 | 6,450,519,360 | 53,433,198 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-23,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AJ0C392CNMDJH9BEZFY3J4 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,483,324 | 6,940,353,514 | 55,577,142 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-6,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5AJW46CSWWCMF31R32P5JKN | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,520,486 | 7,222,286,006 | 57,850,518 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-5,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5AKJJ831QYYFX0JRCZHDER9 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,513,029 | 7,125,389,232 | 57,103,805 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-18,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AM79Y8R9R7GCKWWVZHEH99 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,447,667 | 6,800,683,513 | 54,452,452 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-13,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5ANEJ1KQ0NR41AG4PEZSFP7 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,507,697 | 7,077,556,168 | 56,662,247 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-8,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5APA35Y4C5R39ASY2T8GYGV | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,510,407 | 7,045,648,983 | 56,417,914 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-4,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5APY3B0XS4YQ16Q2EBWR9VK | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,426,950 | 6,718,865,003 | 53,820,753 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-7,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5AQF2AC5FRZM83AMB7FA4FB | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,479,027 | 6,856,283,653 | 54,953,937 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-10,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5AQZME2KQ702WQZ9WHA9343 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,399,707 | 6,645,819,614 | 53,204,040 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-17,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5ARWPSASD1V3D59NCVC4CSM | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,550,251 | 7,261,090,323 | 58,164,387 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-14,tenant_id=tenant1-storage | 0s         | compactor |
| 01J5ASDK72HZGNM4MBA4J5FSCG | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,515,668 | 7,131,415,720 | 57,106,606 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-2,tenant_id=tenant1-storage  | 0s         | compactor |
| 01J5AT1BGWF7355BJ1KAYDRKZ1 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s        | 1,476,254 | 7,070,666,245 | 56,617,297 | 3          | false       | monitor=storage,receive=true,region=fsn1,replica=thanos-receive-tenant1-storage-3,tenant_id=tenant1-storage  | 0s         | compactor |

Sidecar Blocks
Sample Output from thanos tools bucket inspect
|            ULID            |         FROM         |        UNTIL         |     RANGE     |  UNTIL-DOWN   |  #SERIES  |   #SAMPLES    |  #CHUNKS   | COMP-LEVEL | COMP-FAILED |                                                  LABELS                                                  | RESOLUTION |  SOURCE   |
|----------------------------|----------------------|----------------------|---------------|---------------|-----------|---------------|------------|------------|-------------|----------------------------------------------------------------------------------------------------------|------------|-----------|
| 01J5AAF5HPF2PFW5ABW85SF0WT | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s       | 1,167,658 | 5,209,825,975 | 46,609,953 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-10,prometheus_replica=prometheus-sidecar-cnodes-10-0,region=fsn1 | 0s         | compactor |
| 01J5AAXV5ATB0HMGVJR6W1FZ2V | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | -8h0m0s       | 1,089,725 | 5,059,691,020 | 46,056,643 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-1,prometheus_replica=prometheus-sidecar-cnodes-1-0,region=fsn1   | 0s         | compactor |
| 01J5AB7V3H3J0E5FR1M8GA00GP | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | 192h0m0s      | 1,167,658 | 624,417,950   | 4,694,514  | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-10,prometheus_replica=prometheus-sidecar-cnodes-10-0,region=fsn1 | 5m0s       | compactor |
| 01J5ABQ09FP3JD4QK5F6C3JAX0 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 48h0m0s       | 192h0m0s      | 1,089,725 | 609,356,419   | 4,519,046  | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-1,prometheus_replica=prometheus-sidecar-cnodes-1-0,region=fsn1   | 5m0s       | compactor |
| 01J5AN61W5608MDYQDY4Q2NYWV | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.951s | -7h59m59.951s | 1,466,663 | 6,651,517,461 | 59,629,508 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-11,prometheus_replica=prometheus-sidecar-cnodes-11-0,region=fsn1 | 0s         | compactor |
| 01J5ANN2BX9XNR4M29YXWYBVDB | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.998s | -7h59m59.998s | 1,582,493 | 7,864,742,275 | 69,268,695 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-22,prometheus_replica=prometheus-sidecar-cnodes-22-0,region=fsn1 | 0s         | compactor |
| 01J5APAJ59PFEAZTAMACFPNV98 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.992s | -7h59m59.992s | 1,333,182 | 6,461,683,230 | 57,603,142 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-15,prometheus_replica=prometheus-sidecar-cnodes-15-0,region=fsn1 | 0s         | compactor |
| 01J5AQ1NPR8M6B4NGFMV12EHM3 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.998s | -7h59m59.998s | 1,613,122 | 7,945,622,683 | 70,102,189 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-19,prometheus_replica=prometheus-sidecar-cnodes-19-0,region=fsn1 | 0s         | compactor |
| 01J5AQQ0N1EXJ6X50G1YZSA4AE | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.978s | -7h59m59.978s | 1,330,199 | 6,189,685,122 | 55,563,851 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-12,prometheus_replica=prometheus-sidecar-cnodes-12-0,region=fsn1 | 0s         | compactor |
| 01J5AR5R0KP9FA2KDZP5SBDWVX | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.975s | -7h59m59.975s | 1,308,462 | 6,319,618,992 | 56,376,946 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-2,prometheus_replica=prometheus-sidecar-cnodes-2-0,region=fsn1   | 0s         | compactor |
| 01J5ARHMGKAK7JR90VGPDKGQDR | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.993s | -7h59m59.993s | 1,032,887 | 4,753,217,152 | 43,427,265 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-9,prometheus_replica=prometheus-sidecar-cnodes-9-0,region=fsn1   | 0s         | compactor |
| 01J5ARVN1S460E8G55ZC1VH0J5 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.952s | -7h59m59.952s | 879,900   | 4,269,481,051 | 37,960,012 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-24,prometheus_replica=prometheus-sidecar-cnodes-24-0,region=fsn1 | 0s         | compactor |
| 01J5ASA4NHPFQP41M52CN32SKH | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.993s | -7h59m59.993s | 1,423,181 | 7,007,498,147 | 62,191,611 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-14,prometheus_replica=prometheus-sidecar-cnodes-14-0,region=fsn1 | 0s         | compactor |
| 01J5ASTXDVACMQ8X28Z2WP07PE | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.999s | -7h59m59.999s | 1,319,988 | 6,271,030,141 | 56,402,900 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-3,prometheus_replica=prometheus-sidecar-cnodes-3-0,region=fsn1   | 0s         | compactor |
| 01J5ATBB4N72TSMSB6K3WA8HCM | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.991s | -7h59m59.991s | 1,306,804 | 6,213,659,211 | 55,874,472 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-5,prometheus_replica=prometheus-sidecar-cnodes-5-0,region=fsn1   | 0s         | compactor |
| 01J5ATSCJM8J7N0P2FC3H69QQV | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.998s | -7h59m59.998s | 1,430,156 | 6,383,481,921 | 57,123,055 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-17,prometheus_replica=prometheus-sidecar-cnodes-17-0,region=fsn1 | 0s         | compactor |
| 01J5AVAHJNP1AGX8CETWPDCNHM | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.992s | -7h59m59.992s | 1,230,953 | 5,984,898,253 | 53,706,705 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-13,prometheus_replica=prometheus-sidecar-cnodes-13-0,region=fsn1 | 0s         | compactor |
| 01J5AVPZ5XTCBP0091RV7ABH67 | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.998s | -7h59m59.998s | 1,293,051 | 5,843,361,729 | 52,638,926 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-4,prometheus_replica=prometheus-sidecar-cnodes-4-0,region=fsn1   | 0s         | compactor |
| 01J5AW2NH655C3GZYTHDHPD84N | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.995s | -7h59m59.995s | 1,455,648 | 6,593,190,912 | 58,906,486 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-18,prometheus_replica=prometheus-sidecar-cnodes-18-0,region=fsn1 | 0s         | compactor |
| 01J5AWG07X66J4D30T82505G1V | 2024-08-13T00:00:00Z | 2024-08-15T00:00:00Z | 47h59m59.985s | -7h59m59.985s | 1,073,459 | 4,956,575,766 | 45,101,988 | 3          | false       | monitor=cnodes,prom=prom-tenant2-cnodes-8,prometheus_replica=prometheus-sidecar-cnodes-8-0,region=fsn1   | 0s         | compactor |

thanos_bucket_store_blocks_loaded
sum (thanos_bucket_store_blocks_loaded{region="fsn1", pod="store-test-0") by (namespace, region)
{namespace="monitoring-tenant1-storage", region="fsn1"} 2497
{namespace="scrape-tenant2-cnodes", region="fsn1"} 2491

thanos_bucket_store_indexheader_load_duration_seconds_sum
Green: Tenant1
Blue: Tenant2

thanos_objstore_bucket_operation_transferred_bytes_sum
Green: Tenant1
Blue: Tenant2

Is this always reproducible or you discovered this issue the first time?

For Tenant 1 we have 3112 index files with an average of 348MB

Have you checked the largest index file size?

Hey, yeah this is always reproducible even with new buckets. Currently we scrape and receive the same data for one tenant and also for this tenant this behavior is reproducible.
The largest index file is about 800MB.


Update the CONTRIBUTING.md file

Describe the bug
This commit updates the CONTRIBUTING.md file in the repository to provide clear guidelines for contributing to the project.
Expected behaviour
The changes include the following:

Added a section on how to report bugs and issues, including steps for reproducing them and providing relevant information.
Included instructions on how to suggest new features or improvements, including the preferred format for feature requests.
Clarified the code formatting and style guidelines to ensure consistency across contributions.
Provided examples and explanations on how to submit pull requests, along with the necessary steps for the review process.
Added a section on how to communicate effectively with the project maintainers and other contributors, emphasizing the importance of respectful and constructive discussions.
These updates aim to streamline the contribution process and provide contributors with clear instructions on how to get involved. The goal is to foster a welcoming and collaborative environment for anyone interested in contributing to the project.

Screenshots

Additional context
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Do Star the repo before starting your work.

Sure, thanks!


Uncaught PermissionError in history.py line 299

Aider version: 0.64.1
Python version: 3.12.3
Platform: Linux-6.8.0-49-generic-x86_64-with-glibc2.39
Python implementation: CPython
Virtual environment: Yes
OS: Linux 6.8.0-49-generic (64bit)
Git version: git version 2.43.0
An uncaught exception occurred:
Traceback (most recent call last):
  File "aider", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "main.py", line 861, in main
    coder.run()
  File "base_coder.py", line 741, in run
    self.run_one(user_message, preproc)
  File "base_coder.py", line 784, in run_one
    list(self.send_message(message))
  File "base_coder.py", line 1294, in send_message
    shared_output = self.run_shell_commands()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "base_coder.py", line 2030, in run_shell_commands
    output = self.handle_shell_commands(command, group)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "base_coder.py", line 2059, in handle_shell_commands
    self.io.add_to_input_history(f"/run {command.strip()}")
  File "io.py", line 489, in add_to_input_history
    FileHistory(self.input_history_file).append_string(inp)
  File "history.py", line 76, in append_string
    self.store_string(string)
  File "history.py", line 299, in store_string
    with open(self.filename, "ab") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '.aider.input.history'

Thanks for trying aider and filing this issue.
The fix is available in the main branch. You can get it by installing the latest version from github:
aider --install-main-branch

# or...

python -m pip install --upgrade --upgrade-strategy only-if-needed git+https://github.com/Aider-AI/aider.git

If you have a chance to try it, let me know if it works better for you. I'm going to close this issue for now, but feel free to add a comment here and I will re-open. Or feel free to file a new issue any time.


doesnt run

program will just not run in any capacity, opens cmd and closes instantly

Are you aware on how to locate the directory where the files are using cmd?

In what capacity? I try to run start.bat and i just get a momentary cmd window with "directory + python main.py". I have python installed

If you could open the main directory with cmd and try running the program there then it should give you an error instead of instantly closing.

As shown after you have opened the cmd prompt type in "python main.py"


[DRAFT] Stateful testing setup

This is the skeleton to be used primarily for writing all stateful BNS-V2 tests. See each commit for details.

Hey guys! Thanks for this! This is huge and hopefully will take BNS to the limit, please let me know if you need anything

Hey @Patotking12, thanks! 👍 Given the number of invariant cases, you're more than welcome to add some of them.
Feel free to add invariants like @BowTiedRadone is doing here.
Keep each commit simple, adding just one invariant if possible. Avoid helper functions until you see the rule of three and need to refactor. From PoX-4, we learned to start with simple commits and make things more DRY later.
If you have any questions, we're happy to help. Thank you 🙏


colcon build --symlink-install  error

https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nvblox/blob/main/docs/tutorial-human-reconstruction-realsense.md
Hello, I ran into this problem when executing this example.

Starting >>> isaac_ros_stereo_image_proc
Finished <<< isaac_ros_nitros_detection2_d_array_type [26.0s]
--- stderr: isaac_ros_image_proc
CMake Error at gxf/image_flip/CMakeLists.txt:21 (include):
include could not find requested file:
YamlCpp

CMake Error at gxf/image_flip/CMakeLists.txt:22 (find_package):
No "FindGXF.cmake" found in CMAKE_MODULE_PATH.
CMake Warning (dev) at gxf/image_flip/CMakeLists.txt:22 (find_package):
FindGXF.cmake must either be part of this project itself, in this case
adjust CMAKE_MODULE_PATH so that it points to the correct location inside
its source tree.
Or it must be installed by a package which has already been found via
find_package().  In this case make sure that package has indeed been found
and adjust CMAKE_MODULE_PATH to contain the location where that package has
installed FindGXF.cmake.  This must be a location provided by that package.
This error in general means that the buildsystem of this project is relying
on a Find-module without ensuring that it is actually available.
This warning is for project developers.  Use -Wno-dev to suppress it.

Failed   <<< isaac_ros_image_proc [2.16s, exited with code 1]
Aborted  <<< isaac_ros_stereo_image_proc [2.80s]
Aborted  <<< isaac_ros_visual_slam [27.6s]
Aborted  <<< nvblox [4min 15s]

The latest Isaac ROS Common is needed with updated packages with these additional cmake files. Could you also check whether you have a clean build of your ROS workspace (rm -Rf build install log) to make sure you pick up the latest?

Have you encountered the following problems and how to solve them?

This might be a transient network issue somewhere along the connection between your machine and GitHub's servers. I'm able to successfully clone the repository on my machine.
Please double-check that your machine can access GitHub/the internet from within the container.
Additionally, please create a separate GitHub issue next time. There are lots of potential causes for the build to fail, and separating the issues out helps us track and respond more effectively, while also providing more value for future readers 😃


Weak labeling for token classification

Some conclusions that we can start to discuss about WS4NER
Lite integration with skweak
As far as I know, skweak + spaCy integration is (more or less) better option  to apply weaksupervision for token classification. We should provide  a lite integration without  big abstractions.
For example:

weak_labelled_dataset = apply_rules_to_dataset_in_some_way(dataset="the_dataset", rules=<the list of rules>)
docs =weak_labelled_dataset.to_spacy()
# Here, docs will contain the `.spans`  property including `labelling_functions` annotations dictionary

Agnostic data model for weak labeling
Do we need an agnostic data model representing all weak labels generated by rules?
The current data model for TokenClassification does not accept annotated entities with different agents. So, it's not possible built rules results in terms of rubrix records.
Define a set of basic rules using query text
Here the, search keyword is fundamental and a base for rules match definitions, since they will guide the built-in rules.
For example, the rule:
rule = Rule(query="mark*", label="PERSON", action="exact_match")

will be applied on each record as follows:

For each keyword in record
yield the matched token as a PERSON

We can think about a minimal set to easily define rule actions based on keywords. Some of them could be:
# span will be defined by 2 tokens before of keyword Corp (included). "The Masala Corp." should match as ORG for this rule
Rule(query="Corp",  label="ORG", action="before_token", n_tokens=2)

# Span will be defined by 2 tokens after keyword Mr. and the keyword won`t be part of span.
Rule(query="Mr.",  label="PERSON", action="after_token", n_tokens=2, include_keyword=False)

For more complex rules, you can still define your own labelling function
If you want/must define more complex rules, you can always read the dataset and convert to spacy documents, then apply the normal process described in skweak documentation for that.
Basic UI interaction
A simple way to define rules by using the UI. Rules metrics will be limited in terms of precision, but you should  explore matched records and view the rule coverage as a basic metric.
@dcfidalgo @dvsrepo comments a proposal are really welcome


ROX-16667: Optional encryption key secret mount to Helm and Operator

Description
This PR adds support in the Helm charts and the Operator for an option to encrypt notifier secrets.
This is done by:

creating a secret called central-encryption-key in the Central namespace, and then:
setting central.encryptNotifierSecrets to true in the Helm chart, or
setting spec.central.notifierSecretsEncryptionEnabled to true in the Central CR

This will in turn

export ROX_ENCRYPT_NOTIFIER_SECRETS="true" in the Central container
mount the encryption key at /run/secrets/stackrox.io/central-encryption-key/

Checklist

[x] Investigated and inspected CI test results
[x] Unit test and regression tests added
[x] Evaluated and added CHANGELOG entry if required
[ ] Determined and documented upgrade steps
[ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

As the feature is currently not intended to be used directly by users, I did not document the changes in the Helm charts, and I did not create a documentation PR.
Testing Performed
Here I tell how I validated my change

installed Central in a local colima cluster, using the operator
ran /bin/bash in the Central container, checked that ROX_ENCRYPT_NOTIFIER_SECRETS is not defined
created a secret called central-encryption-key and applied it to the Central namespace
patched to Central CR and added spec.central.notifierSecretsEncryption.enabled: true
verified that the Operator reconciled the Central and restarted the Central pod
ran bin/bash in the Central container again, this time making sure that the env var ROX_ENCRYPT_NOTIFIER_SECRETS exists and is set to true, and that the contents of the secret is available at /run/secrets/stackrox.io/central-encryption-key/

Reminder for reviewers
In addition to reviewing code here, reviewers must also review testing and request further testing in case the
performed one does not seem sufficient. As a reviewer, you must not approve the change until you understand the
performed testing and you are satisfied with it.

/test all

/retest

I took a look and it seems that the operator e2e tests failed due to scanner initialization slowness.
At least in the basic-sc case, it was doing something...  for 9 minutes (!) between finishing DB migrations and starting to load NVD definitions.
[pod/scanner-697f44578b-72w8j/scanner] {"Event":"database migration ran successfully","Level":"info","Location":"pgsql.go:205","Time":"2023-09-26 18:20:51.285129"}
[pod/scanner-697f44578b-72w8j/scanner] {"Event":"Loading NVD definitions into cache","Level":"info","Location":"singleton.go:24","Time":"2023-09-26 18:29:18.650833"}
[pod/scanner-697f44578b-72w8j/scanner] {"Event":"Loading definitions directory","Level":"info","Location":"load.go:19","Time":"2023-09-26 18:29:18.651079","dir":"/nvd_definitions"}

On a retry, this does not look good:
    logger.go:42: 08:23:10 | basic-sc/5-central-cr | [pod/scanner-db-84765f58fc-tcp6j/init-db] ALTER TABLE
    logger.go:42: 08:23:10 | basic-sc/5-central-cr | [pod/scanner-db-84765f58fc-tcp6j/init-db] ALTER TABLE
    logger.go:42: 08:23:10 | basic-sc/5-central-cr | [pod/scanner-db-84765f58fc-tcp6j/init-db] ALTER TABLE
    logger.go:42: 08:23:10 | basic-sc/5-central-cr | [pod/scanner-db-84765f58fc-tcp6j/init-db] 2023-09-27 08:19:15.032 UTC [57] ERROR:  canceling autovacuum task
    logger.go:42: 08:23:10 | basic-sc/5-central-cr | [pod/scanner-db-84765f58fc-tcp6j/init-db] 2023-09-27 08:19:15.032 UTC [57] CONTEXT:  automatic analyze of table "postgres.public.vuln_v2"
    logger.go:42: 08:23:10 | basic-sc/5-central-cr | Error from server (BadRequest): container "db" in pod "scanner-db-84765f58fc-tcp6j" is waiting to start: PodInitializing

@stackrox/scanner what could cause this?
Same error in both test cases. Interestingly the OCP-based tests passed on first attempt

/retry

/retest

/retest

/retest

/retest

/retest

/retest

/retest

/retest

/retest


Add mastodon shortcode

This PR adds support to render mastodon toots.
If you have such post.md:
---
title: "My First Post"
date: 2022-11-20T09:03:20-08:00
draft: true
---
## Introduction

This is **bold** text, and this is *emphasized* text.

Visit the [Hugo](https://gohugo.io) website!

{{< tweet user="SanDiegoZoo" id="1453110110599868418" >}}

{{< mastodon server="hachyderm.io" user="andrii" tootID="109335849364316929" >}}

Rendered version is:

This PR resolves https://github.com/gohugoio/hugo/issues/10434

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@bep could you please take a look?

What is the status of this PR? I overlooked this PR when creating #11370 and will happily contribute here, if it helps to get this merged quicker.

See https://github.com/gohugoio/hugo#contributing

Do not submit a pull request until the project lead accepts the proposal.

The proposal (https://github.com/gohugoio/hugo/issues/10434) has not been accepted.

@weddige I'm here and ready to fix my PR; the problem for now is that we need to wait for the acceptance of the proposal.
https://github.com/gohugoio/hugo/issues/10434

Closing per https://github.com/gohugoio/hugo/issues/10434#issuecomment-2033349559.


Revisit: Retry when Spring tests fails to start a Server with a random port obtained in advance

Motivation:
If a random port (0) for internal services or managed services is
specified in Spring configuration, the Spring integration module
allocate available ports early and use them to configure additional
service later.
https://github.com/line/armeria/blob/948763acc7911c951e37dbd35132edd253ea3934/spring/boot2-actuator-autoconfigure/src/main/java/com/linecorp/armeria/spring/actuate/ArmeriaSpringActuatorAutoConfiguration.java#L258-L260
This is not a problem in production mode.
However, in testing situation, many test servers are started with a
random port and the available ports can be acquired by them. #4391 #4329
Modifications:

Add ArmeriaServerLifecyle interface to create ArmeriaServerGracefulShutdownLifecycle if
RetryableArmeriaServerGracefulShutdownLifecycle is missing.
Add RetryableArmeriaServerGracefulShutdownLifecycle to retry a
server with a backoff at most N times.

Result:

This is not a perfect answer for the failure but it might reduce the
number of failures.

Can you check lints?

Sorry, I will do.

Thanks, @ikhoon!


Crowd Counter Streamlit Not Working

Crowd counter streamlit errors out on the website
FileNotFoundError: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs.

streamlit live links removed


Early September Launch Bug Fixes

Codecov Report

Merging #8 into master will decrease coverage by 0.22%.
The diff coverage is 0.00%.

@@            Coverage Diff            @@
##           master      #8      +/-   ##
=========================================
- Coverage    7.38%   7.16%   -0.23%
=========================================
  Files          76      75       -1
  Lines        1814    1871      +57
  Branches      524     556      +32
=========================================
  Hits          134     134
- Misses       1677    1734      +57
  Partials        3       3

Impacted Files
Coverage Δ

src/components/ActivityIndicator.tsx
10.34% <0.00%> (ø)

src/components/AspectRatioView.tsx
0.00% <0.00%> (ø)

src/components/Button.tsx
62.50% <ø> (ø)

src/components/Card.tsx
4.22% <0.00%> (-1.44%)
:arrow_down:

src/components/Carousel.tsx
3.44% <0.00%> (-0.17%)
:arrow_down:

src/components/Footer.tsx
25.00% <0.00%> (+2.77%)
:arrow_up:

src/components/Grid/web.tsx
3.57% <0.00%> (-0.28%)
:arrow_down:

src/components/Image.tsx
0.00% <ø> (ø)

src/components/Modal.tsx
8.33% <0.00%> (+1.66%)
:arrow_up:

src/components/Navbar.tsx
7.27% <0.00%> (+0.12%)
:arrow_up:

... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 561f225...75436be. Read the comment docs.


Support for Git Bash on Windows

Edited the installation scripts to support Git Bash on Windows.

LGTM, thanks for the patch


Update readme / changelog

@craigpaul maybe you can make me a collaborator on this repo so you don't have to merge every change I make.
Simple changes can be made without a PR, like simple changes in README, but can't do this atm because I dont have write permissions on this repo

@craigpaul You dont have to merge the 2.x into the main branch on every change. We can do that once it is really necessary.
Saves you some time 😉

Good call on both fronts. I sent the collaborator invite 👍

Awesome 🎉


Fix nightly + Port Promotion entrypoint fix

Port https://github.com/MinaProtocol/mina/pull/16085 to develop

!ci-build-me

!ci-build-me

!ci-nightly-me

!ci-build-me

!ci-build-me

!ci-build-me

!ci-build-me

!ci-nightly-me

Bypassing branch protection as @svv232 already approved but his permissions are not allowing to pass review


Basic subscription behavior.

We need to decide on what a common spec will be for basic subscription. What I know so far from my discussions with @jhusain is that the TC39 spec for observables will look a lot like RxJS's Observables.
The basic API looks like this:

function nextHandler(x) {
  console.log('next: %o', x);
}

function errorHandler(err) {
  console.error('error: %o', err);
}

function completedHandler() {
  console.info('done');
}

// subscription
var disposable = observable.forEach(nextHandler, errorHandler, completedHandler);

// disposal
disposable.dispose();

forEach
Actually subscribes to the observable, if the observable is "cold" it sets up the underlying data stream. The RxJS implementation does this synchronously (which I'm not sure I like). It returns a disposable object with a dispose method on it.
The only problem I have with RxJS's forEach implementation is I think it should be asynchronous by default. (I'm not talking about event emission, only subscription or setting up the underlying stream)
disposable return
The reason for the disposable return and not just a function is that we can get better performance out of leveraging prototypal inheritance and calling the super-class' dispose() method that we can with closures (if we were to only return a dispose function), which are more expensive. In other words, I believe (and @mattpodwysocki can correct me) it creates a new Disposable that subclasses the previous disposable which calls _super() inside of it's dispose method.
The disposal of the stream is the biggest divergent thing from one implementation to the next
I'm really not sure what to do about this, honestly. RxJS returns an object, Bacon returns a function, and Most expects composition to explicitly end a stream programmatically (I think?)
What are your thoughts?

The RxJS implementation does this synchronously (which I'm not sure I like). It returns a disposable object with a dispose method on it.

I prefer async start as well, for a couple reasons:

Having some streams emit in the current call stack, and others not, can create hazards for the developer.  It's in the same vein as promises never invoking their callbacks in the current call stack.
It made it easier to ensure that the disposable chain was established before the first event is ever processed.  For example, if the first event causes the stream to end (e.g. if there's a take(1) in the mix), then the its possible to guarantee the disposer chain is in a valid state.

The reason for the disposable return and not just a function is that we can get better performance out of leveraging prototypal inheritance

Yes, exactly.  While I like the idea of just returning a function, using an object helps avoid capturing scope chains.  In my mind that's the primary advantage, rather than inheritance, since my personal preference is composition over inheritance to chain disposables.

Most expects composition to explicitly end a stream programmatically

Most.js encourages declarative over imperative wherever possible.  Callers typically arrange, using take, until, etc. to lose interest when needed.  Imperative event source adapters, like most.create provide a way to for the source to imperatively end an event stream, but otherwise, it must be done declaratively.

var disposable = observable.forEach(nextHandler, errorHandler, completedHandler);

My perspective is that nextHandler has different characteristics than the other two: it is intended to be called zero or more times, while the other two will be called at most once.  I think in both RxJS and most.js, a typical sequence would be zero or more nextHandler calls followed by exactly one call to either errorHandler or completedHandler.  IOW, the final overall outcome of success or failure is much like a promise.
That was the reasoning behind most.js's forEach only taking 1 function argument, and returning a promise for the final outcome.  Callers can observe the promise, using then and catch, to deal with the final outcome.  Obviously, returning a promise means that forEach cannot return a disposable. Hence, the preference for declaratively losing interesting.

Obviously, returning a promise means that forEach cannot return a disposable.

I see this being the biggest point of contention, ongoing. The TC39 spec will be a lot like RxJS and what I outlined above. The open standard should accommodate that spec as much as possible.
So if there were native Observables in the browser that matches this spec, do you think you'd try to have your observables match the interface?
Another side to this is that if ever come up with compelling reasons to change this interface, we can always make suggestions to the tc39 effort.

The open standard should accommodate that spec as much as possible.

It seems like the mission of this group should be to find the best possible interop API, e.g. how do we make things like this work: most.of(123).flatMap(Rx.Observable.of).  At the end of that process, maybe we'll have decided that the TC39 API is the best fit for interop, but maybe not.
I could see coming at it from at least 2 angles:

Start with the current forEach TC39 spec, and think about alternatives and improvements, such as enforcing that no event/error/end may occur in the same call stack as the call to forEach (ie async start), or passing in an object instead of 3 functions in order to avoid scope capturing.

Think about an interop API from first principles.  Besides just bridging event/error/end signals between libs, eg. do we need to consider timestamps, consumer demand (aka backpressure), etc.?

So if there were native Observables in the browser that matches this spec, do you think you'd try to have your observables match the interface?

Since supporting it would be a benefit to users, yeah, I'd consider it.  However, I disagree with the current forEach proposal as a primary public API for the reasons I described above: I think it conflates too much into a single signature, and I think it would be better to encourage declarative limits rather than the imperative use of a returned disposable.

@blesh I have an idea for an alternative interop proposal.  I'll try to put together a quick outline this weekend.

Great! Sorry I wasn't very responsive. I was at ngconf the tail end of last week. Just getting back into the swing of things. @jhusain was at that conference with me as well. I'm very interested in his feedback here, because he's the one working on the TC39 spec draft.
There was a LOT of talk about this particular issue at ngconf. That is, standard interop and desired behaviors of Observables. Angular 2 may (or may not) ship with a lite version of Observables.

@blesh Cool, no worries.  I agree @jhusain's feedback would be hugely valuable here.
I ended up not having time to actually write anything down over the weekend, so everything's still in my head :(  But I'll try to make time this week.

Ok, here's another strawman.  This is def intended for interop, and not necessarily as a friendly public API.  It's similar to the original proposal above, but uses an subscriber object instead of functions.  That removes the need for any capturing when interoperating.
I also tried to capture some of the operational requirements of the subscribe method.
I'm not sold on any of the names here--naming can be another discussion if this proposal seems interesting.  I did borrow some of the naming from the reactive-streams-jvm spec
// subscriber
interface Subscriber {
	void event(x)
	void error(e)
	void end(x)
}

// subscribe
// - observable must not call subscriber.event/error/end
//   in the current call stack
// - observable must not call subscriber.event/error/end after calling
//   subscriber.end or subscriber.error
// - observable may call subscriber.event zero or more times.
// - observable may call subscriber.end at most once
// - observable may call subscriber.error at most once
let subscription = observable.subscribe(subscriber)

// forcibly unsubscribe
// Not sure yet what this should return (undefined?)
subscription.unsubscribe()

Hey Brian,
The proposal I'm working on is nearly identical to yours, the only difference is that generator interface introduced in ES6 for the Observer interface:
{
next(value),
throw(e),
return(v)
}
I think the most logical decision for interop is to use the type that's natively in the language. It looks like we agree on semantics though (with the exception of always subscribing adynchronously).
Do you have time for a hangout today? Say 10:30?

@jhusain Hey, cool, thanks for jumping in here!

I think the most logical decision for interop is to use the type that's natively in the language

That makes sense to me.  Like you said, they only differ in the method names.

(with the exception of always subscribing adynchronously)

I should try to clarify a bit more what I meant.  The way this works in most.js is that a new subscriber is subscribed synchronously (eg literally added to an Array in the multicast case), but no event is ever delivered to that new subscriber in the same call stack.  In fact, all events are delivered in a trampoline outside of normal user space (ie in some platform async call stack, like setTimeout, etc).
So far, it has worked out very well because it allows subscription setup to completed in the current call stack before any events are delivered.
Also, Promises/A+ discussions, and real world promise usage has convinced me (and most every promise implementor) that "sometimes sync, sometimes async" is a pretty big footgun for developers.
What timezone are you in?  I'm EST.  I probably can't do a hangout today, but I can try to make time for one tomorrow if there's a time that works for you.  Let me know, it'd be great to chat.

I'm a proponent of an idea that is somewhere in the middle I think.

Subscription occurs in the next code block.
Events can occur synchronously after that.

This is primarily to reduce the chances of releasing Zalgo.
It's a strong opinion weakly held. I'm willing to be convinced otherwise if the arguments are solid.

@blesh I'm not quite sure what you mean by "subscription occurring on the next codeblock".
I think there are two things at play here:

"subscription" - the act of a caller subscribing to an observables events by calling subscribe
the act of the observable propagating an event by calling subscriber.next

I'll try to explain what I mean by using promises as an analog.  Calling then(f) on a promise causes f to be placed into an internal queue before then returns (ie in the current call stack).  However, f will never be invoked before then returns, even if the promise is already fulfilled.  There's no zalgo in that scenario.  IOW, zalgo is not caused by placing f into an internal promise queue before then returns.
In contrast, a promise-like implementation in which f may or may not be called before then returned would certain unleash zalgo.  The silly, but illustrative classic example is the following, which might log 0 or might log 1.
var i = 0;
promise.then(() => ++i);
console.log(i);

Similarly in the stream/observable case, I don't believe that putting a subscriber into an internal subscription list is a source of nondeterminism (and in fact, may help developers reason about things better--see 2 below).  The key is never calling subscriber.next/end/error before subscribe(subscriber) returns.  That is, never deliver an event to a subscriber in the same microturn in which it subscribed.
I believe this has 2 advantages:

The code to put the subscriber in a list is simpler than the code to wait a tick and then put the subscriber in a list.
Waiting a tick to put a subscriber in a list may create unintuitive behavior: events that are already in a trampoline could be "missed" by subscribers who are also sitting in a trampoline waiting to be added to a subscription list.

Sorry to be a bit long winded.  Did that make sense?

Thinking about it a bit more: I think the only thing that matters is never calling subscriber.next/end/error before subscribe() returns.  That's enough to avoid zalgo.  When the actual subscription "happens" is probably just an implementation detail.

I'm saying that...
var s = 0;
var observable = Observable.from([1,2,3]);
observable.forEach(x => s + x);
console.log(s);

should never log 6. It should always log 0.
To mitigate this, I'm basically saying that the stream setup that occurs when forEach is called should be deferred to the next frame. However, all emitted "next" events from that observable should happen synchronously (in this case) or asynchronously as necessary.
I think we're talking apples and apples. But I'm not expressing my apples very well.

In other words we completely agree.
What I'm most interested in is @jhusain's position that it should not be like this. I've heard him mention it several times, but I haven't heard the reasoning. I assume it has something to do with performance? But is performance worth Zalgo? ¯_(ツ)_/¯

Sorry for the late reply. I'm in Pacific? I can call you at 7:00 AM if you are in Eastern (10AM EST)? If not, can you chat at 7 PM tomorrow? (PST)
JH

On Mar 17, 2015, at 11:41 AM, Brian Cavalier<EMAIL_ADDRESS>wrote:
@jhusain Hey, cool, thanks for jumping in here!
I think the most logical decision for interop is to use the type that's natively in the language
That makes sense to me. Like you said, they only differ in the method names.
(with the exception of always subscribing adynchronously)
I should try to clarify a bit more what I meant. The way this works in most.js is that a new subscriber is subscribed synchronously (eg literally added to an Array in the multicast case), but no event is ever delivered to that new subscriber in the same call stack. In fact, all events are delivered in a trampoline outside of normal user space (ie in some platform async call stack, like setTimeout, etc).
So far, it has worked out very well because it allows subscription setup to completed in the current call stack before any events are delivered.
Also, Promises/A+ discussions, and real world promise usage has convinced me (and most every promise implementor) that "sometimes sync, sometimes async" is a pretty big footgun for developers.
What timezone are you in? I'm EST. I probably can't do a hangout today, but I can try to make time for one tomorrow if there's a time that works for you. Let me know, it'd be great to chat.
—
Reply to this email directly or view it on GitHub.

Yes. :) "Don't release Zalgo" falls into the same category as "don't use threads" and "avoid mutation." These are all good practices in general, but we need to preserve the flexibility to opt out of them in order to achieve an acceptable level of performance.
I think there is a solution where everyone can get (pretty much) what they want. Still working on a larger document, because I figure this will come up in the TC-39 as well. Super busy at the moment, appreciate your patience.
JH

On Mar 17, 2015, at 10:13 PM, Ben Lesh<EMAIL_ADDRESS>wrote:
In other words we completely agree.
What I'm most interested in is @jhusain's position that it should not be like this. I've heard him mention it several times, but I haven't heard the reasoning. I assume it has something to do with performance? But is performance worth Zalgo? ¯_(ツ)_/¯
—
Reply to this email directly or view it on GitHub.

@blesh Great, yep, I think we're effectively saying the same thing.
@jhusain No worries.  I could chat from 7-7:30pm PST.  Maybe the 3 of us can do a google hangout then?

"Don't release Zalgo" falls into the same category as "don't use threads" and "avoid mutation."

I completely agree that these are likely impossible constrains in the general/global case.  However, I think   we're talking about a very specific case, the instant of subscription, in which the spec can remove a developer hazard.
This was discussed at great length in Promises/A+, and those discussions included several TC-39 members, as well as other very smart folks.  The conclusion there was that preventing the hazard at the instant of calling then was worthwhile.  There were similar concerns about performance at the time.  The most popular current implementations (Bluebird, RSVP, and when.js) have proven that it isn't a concern at all.  Similarly, I believe that most.js and kefir prove that it isn't a concern for reactive streams either.
In my experience, the number of calls to subscribe tends to be a drop in the ocean compared to the number of events propagated through a stream.  So making subscribe safe, and making event propagation fast seems like the right combo to me.
All of that said, I kind of forgot that my proposal above was intended as an interop API rather than a public API (tho I'm not convinced we could enforce that in JS!)  IOW, I'm not sure if allowing subscriber.event/end/error before subscribe() returns would be ok purely for interop, but my intuition is to err again on the side of safety.

We still on for hangout?
JH

On Mar 18, 2015, at 4:32 AM, Brian Cavalier<EMAIL_ADDRESS>wrote:
@blesh Great, yep, I think we're effectively saying the same thing.
@jhusain No worries. I could chat from 7-7:30pm PST. Maybe the 3 of us can do a google hangout then?
—
Reply to this email directly or view it on GitHub.

I may be a few minutes late, but I can still do 7pst.

Email me at brian at hovercraftstudios dot com

Sorry for the delayed response here, I've been caught up in a promotion.
May I suggest we first discuss for which reasons the JVM interfaces ended up the way they did (none of it is accidental, a lot of effort was poured into getting the signatures just right, and the spec took a very long time to distill to the essentials). That'd save everyone a ton of time I think.

Any news here?

A lot of this is now being covered at http://github.com/zenparsing/es-observable... Since Observable ES7 spec has some traction. Once that settles down a bit, and RxJS "v Next" is in full swing, this will be the place for either: 1. the interop spec, or 2. a JavaScript client/server implementation of the reactive streams io protocol... thoughts, @viktorklang?

@blesh If we can get the reactive-stream-io protocol and TCK in place then interop via it will be extremely nice. But JS-lib-to-JS-lib RS interop would be preferable too (orthogonal). Wdyt?

It seems like @jhusain's traction with Observable in ES7 will do some of the work of providing an interop spec or starting point between JS libs.  Perhaps we should focus this repo on Reactive Streams IO connectivity for JavaScript? I'm unsure. Actually the delineation between Reactive-Streams, ReactiveX and the Reactive-Extensions orgs are very, very confusing to me personally.

@blesh I spoke with @jhusain in New York ~1 week ago.
The big, open, question for me is how ES7 Observable will deal with back pressure (especially for IO and handoffs between an async boundary). As for reactive-streams-io I think the need for it grows stronger and stronger.

@viktorklang is it your opinion/position that the current ES7 Observable spec doesn't have the ability to be "updated/upgraded" to support a more RxJava-y approach to backpressure? (via the addition of some sort of request mechanism)  I've not looked into it, honestly.

@blesh I am not familiar enough with the current ES7 proposal to be sure.

Since JavaScript has async await, IMHO it makes much mores sense to use async iterables for back pressure (reactive streams are actually a form of async pull that could not leverage async await)

@headinthebox You mean something similar to https://news.ycombinator.com/item?id=9742967 ?

it seems like subscription / disposal issue is solved https://github.com/zenparsing/es-observable/issues/48

This has gone stale forever ago.


bcm283x: Increase read performance up to 13.6MHz

Gains are between 7% and 10%.

Codecov Report

Merging #180 into master will decrease coverage by 0.03%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##           master     #180      +/-   ##
==========================================
- Coverage   86.25%   86.21%   -0.04%
==========================================
  Files          64       64
  Lines        5186     5188       +2
==========================================
  Hits         4473     4473
- Misses        628      629       +1
- Partials       85       86       +1

Impacted Files
Coverage Δ

host/bcm283x/gpio.go
53.77% <33.33%> (-0.33%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d86cdf7...3905786. Read the comment docs.


[ALLUXIO-2292]Fix javadoc in InodeTree.java

https://alluxio.atlassian.net/browse/ALLUXIO-2292

Can one of the admins verify this patch?

Jenkins, ok to test

ok to test

Jenkins add to whitelist

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/11782/
Test FAILed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/11770/
Test PASSed.

Merged build finished. Test PASSed.

Thanks


Who is using Zino?

The purpose of this issue
We are always interested in finding out who is using Zino, what attracted you to using it, how we can listen to your needs and if you are interested, help promote your organization.
What we would like to hear from you
Submit a comment in this issue and include the following information:

Your organization or company
Link to your website
What is your scenario for using Zino?
Are you running your application in testing or production?

Company: 杭州宇算科技
Website: https://cosmoduli.com/
Usage: 后端API开发，桌面应用开发
Status: 生产环境

Company: individual
Website:
Usage: API server
Status: development