Missing pulldown-menu in visualisation

I wanted to change something in my VIS when I noticed that the pull-down menu on the right edge of the screen is no longer visible. Before I could use it to select objects.
I'm using Firefox and Chrome, changed the "zoom" for both and cleared the cache. Even moving the bar does not bring any improvement.

Yes, I can confirm that. Maybe as workaround: downgrad back to VIS 1.2.6 (WEB 3.0.10 / Socket.io 3.0.12) and everything runs again (#292).

Still happening in current version?

Is it some specific widget or all widgets?


Bring master into dev

Looks like Master and Dev are out of sync.

Hi, @lukewrites! Thanks for opening this PR!

In the file tdvt/tdvt/config_gen/datasource_list.py:

Line 168:121: E501 line too long (151 > 120 characters)
Line 167:121: E501 line too long (197 > 120 characters)

In the file tdvt/tdvt/config_gen/tdvtconfig.py:

Line 40:13: E117 over-indented
Line 11:80: E252 missing whitespace around parameter equals
Line 11:79: E252 missing whitespace around parameter equals
Line 9:1: E302 expected 2 blank lines, found 1

In the file tdvt/tdvt/config_gen/test_config.py:

Line 272:72: F632 use ==/!= to compare str, bytes, and int literals
Line 272:45: F632 use ==/!= to compare str, bytes, and int literals
Line 272:16: F632 use ==/!= to compare str, bytes, and int literals
Line 257:42: E226 missing whitespace around arithmetic operator
Line 253:1: E302 expected 2 blank lines, found 1

In the file tdvt/tdvt/tabquery.py:

Line 52:121: E501 line too long (138 > 120 characters)

In the file tdvt/tdvt/tdvt.py:

Line 588:71: E703 statement ends with a semicolon
Line 332:71: E251 unexpected spaces around keyword / parameter equals
Line 332:69: E251 unexpected spaces around keyword / parameter equals
Line 288:121: E501 line too long (131 > 120 characters)
Line 273:51: E226 missing whitespace around arithmetic operator
Line 32:1: E402 module level import not at top of file
Line 31:1: E402 module level import not at top of file
Line 27:121: E501 line too long (125 > 120 characters)
Line 27:1: E402 module level import not at top of file

In the file tdvt/tdvt/tdvt_core.py:

Line 600:43: F405 'TdvtInvocation' may be undefined, or defined from star imports: .resources, .test_results
Line 245:32: F405 'TestErrorOther' may be undefined, or defined from star imports: .resources, .test_results
Line 174:121: E501 line too long (128 > 120 characters)
Line 150:121: E501 line too long (124 > 120 characters)

In the file tdvt/tdvt/test_results.py:

Line 462:23: E711 comparison to None should be 'if cond is None:'
Line 458:23: E711 comparison to None should be 'if cond is None:'
Line 423:26: E711 comparison to None should be 'if cond is None:'
Line 419:26: E711 comparison to None should be 'if cond is None:'
Line 413:25: E711 comparison to None should be 'if cond is None:'
Line 398:121: E501 line too long (121 > 120 characters)
Line 152:123: E251 unexpected spaces around keyword / parameter equals
Line 152:121: E251 unexpected spaces around keyword / parameter equals
Line 152:121: E501 line too long (148 > 120 characters)
Line 152:108: E251 unexpected spaces around keyword / parameter equals
Line 152:106: E251 unexpected spaces around keyword / parameter equals
Line 152:83: E251 unexpected spaces around keyword / parameter equals
Line 152:81: E251 unexpected spaces around keyword / parameter equals
Line 152:53: E251 unexpected spaces around keyword / parameter equals
Line 152:51: E251 unexpected spaces around keyword / parameter equals
Line 152:35: E251 unexpected spaces around keyword / parameter equals
Line 152:33: E251 unexpected spaces around keyword / parameter equals

In the file tdvt/test/tdvt_test.py:

Line 666:121: E501 line too long (143 > 120 characters)
Line 659:121: E501 line too long (142 > 120 characters)
Line 652:121: E501 line too long (142 > 120 characters)
Line 652:65: E231 missing whitespace after ','
Line 595:79: E221 multiple spaces before operator
Line 594:79: E221 multiple spaces before operator
Line 593:80: E221 multiple spaces before operator
Line 590:21: F405 'get_path' may be undefined, or defined from star imports: tdvt.tabquery, tdvt.test_results
Line 582:121: E501 line too long (184 > 120 characters)
Line 348:9: F841 local variable 'test_set' is assigned to but never used
Line 347:9: F841 local variable 'test_file' is assigned to but never used
Line 342:23: F405 'TdvtInvocation' may be undefined, or defined from star imports: tdvt.tabquery, tdvt.test_results
Line 341:35: E226 missing whitespace around arithmetic operator
Line 241:28: F405 'TdvtInvocation' may be undefined, or defined from star imports: tdvt.tabquery, tdvt.test_results
Line 100:28: F405 'TdvtInvocation' may be undefined, or defined from star imports: tdvt.tabquery, tdvt.test_results
Line 67:27: F405 'TdvtInvocation' may be undefined, or defined from star imports: tdvt.tabquery, tdvt.test_results

For reference: Error Code Reference, PEP8, Python code style

Opening a new PR with no merge conflicts.


[Question]Significant differences between deepspeed and torchrun training results

Hello,
I'm performing continual training of Llama-2-7b. I'm using the same hyperparameters to train the model (with flashattn2 and bf16 on A100) using torchrun and deepspeed (zero2) but the results are very different.
Torchrun's results are significantly better than deepspeed trained ones:

My deepspeed configuration is:
{
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },
    "bf16": {
           "enabled": "auto"
    },
    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": "auto",
            "betas": "auto",
            "eps": "auto",
            "weight_decay": "auto"
        }
    },

    "scheduler": {
        "type": "WarmupDecayLR",
        "params": {
            "warmup_min_lr": 5e-6,
            "warmup_max_lr": "auto",
            "warmup_num_steps": "auto",
            "warmup_type": "linear",
            "total_num_steps": "auto"
        }
    },

    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "allgather_partitions": true,
        "allgather_bucket_size": 2e8,
        "overlap_comm": true,
        "reduce_scatter": true,
        "reduce_bucket_size": 2e8,
        "contiguous_gradients": true
    },

    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto"
}

I checked the differences between torchrun and deepspeed. The only possible difference is on the warmup_min_lr (torchrun using 0 but deepspeed using 5e-6) and optimizer (torchrun using adamw_torch, deepspeed using its native AdamW). Except these differences, I can't find any other possible reasons accounting for the difference of the performance. However, I don't think these two configuration differences could lead to such a significant performance gap. Is there anything that I missed? Any suggestion or comment would be very helpful to me.
Thank you

After trying a group of experiments, I find the difference is mainly caused by "offload_optimizer". When I remove offloading, the results become much better and more similar to the torchrun's results.
Does it make sense?

@getao, thanks for the update. This suggests a bug in the optimizer offloading logic. The model performance gap is very significant, and we need to investigate. Are you able to provide more details to help us repro?

Sure.
I continuously train the Llama-2-7b on a domain-specific dataset using 4 80GB A100 GPU. The training scripts of torchrun and deepspeed are:
#torchrun version
#!/bin/bash

lr=6e-5
eval_steps=1000
save_steps=2500
INPUT_DATA=MY_DATA_PATH
batch=1
epoch=2
OUT_DIR=MY_OUTPUT_PATH
max_seq_len=1024

torchrun --nproc-per-node=4 baseline.py --adam_beta2 0.95 --adam_epsilon 1e-6 --num_train_epochs $epoch --per_device_train_batch_size $batch --per_device_eval_batch_size $batch --gradient_accumulation_steps 1 --add_special_tokens \
          --learning_rate $lr --logging_steps 10 --save_strategy 'no' --evaluation_strategy 'steps' --eval_steps $eval_steps --save_steps $save_steps --bf16 --max_seq_length $max_seq_len --output_dir $OUT_DIR --logging_dir $OUT_DIR/logs --data_path $INPUT_DATA | tee $OUT_DIR/train.log

#deepspeed version
#!/bin/bash

lr=6e-5
# train_steps=50
eval_steps=1000
save_steps=2500
INPUT_DATA=MY_INPUT_PATH
batch=1
epoch=2

ds_zero=1
max_seq_len=1024
ds_config=ds_zero${ds_zero}.json

OUT_DIR=$MY_OUTPUT_PATH

deepspeed --num_gpus=4 baseline.py --adam_beta2 0.95 --adam_epsilon 1e-6 --num_train_epochs $epoch --per_device_train_batch_size $batch --per_device_eval_batch_size $batch --gradient_accumulation_steps 1 --add_special_tokens \
          --learning_rate $lr --logging_steps 10 --save_strategy 'no' --evaluation_strategy 'steps' --eval_steps $eval_steps --save_steps $save_steps --bf16 --max_seq_length $max_seq_len --output_dir $OUT_DIR --logging_dir $OUT_DIR --data_path $INPUT_DATA --deepspeed $ds_config | tee $OUT_DIR/train.log

Deepspeed configuration (no offloading version) is:
{
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },
    "bf16": {
           "enabled": "auto"
    },
    "optimizer": {
        "type": "AdamW",
        "params": {
            "torch_adam": true,
            "lr": "auto",
            "betas": "auto",
            "eps": "auto",
            "weight_decay": "auto"
        }
    },

    "scheduler": {
        "type": "WarmupDecayLR",
        "params": {
            "warmup_min_lr": "auto",
            "warmup_max_lr": "auto",
            "warmup_num_steps": "auto",
            "warmup_type": "linear",
            "total_num_steps": "auto"
        }
    },

    "zero_optimization": {
        "stage": 1,
        "allgather_partitions": true,
        "allgather_bucket_size": 2e8,
        "overlap_comm": true,
        "reduce_scatter": true,
        "reduce_bucket_size": 2e8,
        "contiguous_gradients": true
    },

    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto"
}

The offloading version additionally has
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
``` in the "zero_optimization".

Just run a group of experiments using the same version (0.12.3), same hyperparameters and same seed. ZERO-2 with offloading yields a significantly worse result:

I set some trainable parameters in my network and found that some layers update very slowly and hardly update at all. Some update very quickly. It is normal to update without using deepspeed.

Any updates on this? Were you able to find the reason for this?


Issue with k8s.io/docs/concepts/storage/persistent-volumes/

This is a Bug Report

Problem:
Page makes no reference to the claimRef property, or bi-directional binding.
claimRef property is only documented in the
api reference, and includes the following in the description:

More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#binding

It is discouraging to click through and find no additional information!
Proposed Solution:
Add documentation describing bi-directional binding of PVs and PVCs.
Page to Update:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#binding

/language en
/priority backlog

#17914

/lifecycle active


Setup staging and production infrastructure

This allows us to deploy to different environments, Test (or Edge), Development (or Staging) and Production. We've also documented how to deploy to prouction and get the relevant permissions, as there's some weirdness around this.

I've added the changes we made earlier, I just need to update the docs with the changes to deployment process

Yeah, I agree. Although the impression I get is that we're supposed to deploy through the central Service Principle Name. I could be wrong though. There's a reply from David Goodyear on Slack: https://ukgovernmentdfe.slack.com/archives/CKYNU0VG9/p1562767543024000

Actually looking at it we might be able to sign in to the CLI as the service principle? https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli?view=azure-cli-latest#sign-in-with-a-service-principal

In that case, rather than adding infrastructure deployment to CD, let's change the script to log in as the appropriate service principle.

I think for now, if we rollback to before Robbie's commits, this is good to go? We can then do the work to use the service principle in another PR.

That does leave the question, how do we actually deploy using the release pipeline? The default azure Web App deployment task in Pipelines stomps all over our docker-compose.yml, so won't work.
The way I envisaged it was that we continuously deploy to the development envrionment from the master branch, and  to the test environment from the edge branch. Deploying to production will still requires a human to manually kick off the deploy from the pipeline, and we can do manual checks of the config before deploying, and also have confidence that it works because we saw an automatic deploy go to development.

I've removed the two commits from yesterday. Stu's right though it does leave the question of how we're going to deploy code changes via the pipeline though.


add simple example of using fixture for test_structs

verified via print statement that test_state() is only called once

closes #559

thanks for the tip, @williamshen-nz !


B 17663 review expense parsing amount

B-17663
Summary
Updated ReviewExpense component to use shared util function convertDollarsToCents to properly parse dollar/cents amount, and updated the associated test.jsx file, to fix a bug where office users were unable to accept receipts whose dollar amounts exceeded $999.99
Verification Steps for the Author
These are to be checked by the author.

[x] Tested in the Experimental environment (for changes to containers, app startup, or connection to data stores)
[x] Have the Agility acceptance criteria been met for this change?

Verification Steps for Reviewers
These are to be checked by a reviewer.
Setup to Run the Code

Instructions for starting storybook
Instructions for starting the MilMove application
Instructions for running tests

How to test

As a service counselor, view any customer move with a PPM shipment that's at the document review stage which includes a receipt.
Navigate to one of those receipts, and in the "Amount" field, enter an value equal or greater to $1,000 (so that a comma separator displays to the user).
Choose to accept/reject/exclude and press the "continue" button to progress to the next document. No errors should appear and you should navigate to the next document.

Warnings

:warning:
Please add the JIRA issue key to the PR title (e.g. MB-123)

  Generated by :no_entry_sign: dangerJS against a9a9b5211e7d2e2aa04ba5c07184803fe3ab0d07


devon modules migration and spring boot upgrade

Microservices-archtype testing is pending ,just created for sharing.

i did not changed .adoc and .txt files (CODE_OF_CONDUCT.adoc , CONTRIBUTING_GUIDE.adoc ,terms-of-use.txt).

** Please DON'T MERGE **
Tasks related to this PR are in progress.

Hi, as far as I have reviewed seems that the core of the changes is related to versions and naming so I have focused on functionality. I was able to launch all the infrastructure services and a sample microservice without issues, the only thing I missed was a change in the devonfw parent pom <devonfw.version> property  (it is still marked as 2.4.0, what broke other microservices projects I have btw). Regarding the functionality I have noticed that now when no authenticated request is done the error thrown appears in html format and is not too descriptive (I think in previous versions the error message was more concrete). I guess it is not directly related with those particular PRs but a previous changes in the modules, but maybe it could be treated as a future improvement.

Yes we will change <devonfw.version>  in root pom to 3.0.0 while making the release. For other issue mentioned we will create issue and solve it in future releases


typo?

in create application gateway , I had to alter  address1 and address2 to use privateIPAddress not privateIpAddress
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: b1982120-a213-fc4d-7b93-0b09729d48d6
Version Independent ID: 4c2c5aed-7e9f-ecb7-847b-50be8c328278
Content: Quickstart: Direct web traffic using CLI - Azure Application Gateway
Content Source: articles/application-gateway/quick-create-cli.md
Service: application-gateway
GitHub Login: @greg-lindsay
Microsoft Alias: greglin

@richardturnerfos
Thanks for your feedback! We will investigate and update as appropriate.

#please-close


Get pv data from Solar Shefield

Get PV data straight from Solar Sheffield
more details to come

Need to change datetimes, as they are not UTC in the database

ALl times are UTC
but 36150, 36386, 43629, 60211 in pvoutput.org look a bit dodgy

update pv_system
set correct_data=false
where pv_system.pv_system_id in (36150, 36386, 43629, 60211)

they are not UTC in the database

Does that mean the times are not UTC in our database? (Or Sheffield's DB?)
If they're not in UTC, are they stored with an explicit UTC offset? If not, I fear we'll have duplicate "identical" datetimes at one daylight savings transition, and an hour's gap at the other DST transition!

Time are UTC in our database. But there was something funny with the pv systems above .They were reporting information from about 30 mins into the future. Of course this is not possible, so there must be something wrong with that system.
Easy way was to remove them for the moment

When pulling the data, we see no systems going over 1kw in the last week. This is a bit odd.
Pulling all data from 3rd July, the highest yield value was 736 watts. This is was for system ss_id=7759, system_id=35186, which has a installed cpaacity of 3.9 KWp


[Android] Buttons are clickable when React Native Inspector is opened

To date on Android if you open React Native Inspector and try to select a RNGH-button, that button would execute it's click action, making debugging rather inconvenient. On iOS inspector works as expected.

Hi, @angly-cat
Thank you a lot for this issue, however, I'm not able to fix it without changes in react-native code. Trigering inspectind has no side effects and then I'n not able to hook it. Switching inspector, does not imply any rerender or any callback. What's more, I found that as it's is possible to block touches on Touchables components, it's not possible to block them in our Library.
Am I right, @kmagiera?

I decided to close this issue. I understand that problem is present but as it's really minor we are going to have radar on it later

Hi, @osdnk! Thanks for responding!

I understand that problem is present but as it's really minor we are going to have radar on it later

This "clicking through" is very inconvenient when debugging RNGH's buttons' and handlers' styles/hitboxes etc.

Trigering inspectind has no side effects and then I'n not able to hook it. Switching inspector, does not imply any rerender or any callback.

I've dived into react-native's code and indeed found no ways to hook on inspector in java code. However, it's possible in JS-thread (found in Libraries/ReactNative/AppContainer.js)
require('RCTDeviceEventEmitter').addListener(
    'toggleElementInspector',
    () => { /* React Native Inspector is toggled */ }
);

With that, if fixing this can be done in JS without putting too much effort, could you give this a second thought, please?

Ok, but I'm not convinced is it a good idea to use is. Maybe it is good idea to make PR for react native to export some constant with "inspect enabled" state

but I'm not convinced is it a good idea to use is

@osdnk I don't think that hooking to inspector toggling event is the best idea too. Therefore I dug into react-native's sources to find out how they "disable" touchables, so we could think of a non-hacky solution.
Apparently, when Inspector is toggled, react-native cover the entire screen with <View onStartShouldSetResponder={...}>-overlay for intercepting all touches. onStartShouldSetResponder returns true, so this overlay captures all touches before they could reach underneath touchables.
As I didn't develop RNGH I can't say for sure, but it seems that by design RNGH handles gestures before react-native's original gesture handling system, which causes this issue in the first place. However, since that's by design, this issue can only be fixed by somewhat hacky solutions like listening to Instector toggling events.

You're right. Maybe it's better than nothing.


FkProducer.java:49: Fake producer close is not implemented

The puzzle 44-45e88687 from #44 has to be resolved:
https://github.com/eo-cqrs/eo-kafka/blob/d78f949e67ac72229c396ec402568cb6573b91c9/src/main/java/io/github/eocqrs/kafka/fake/FkProducer.java#L49-L49
The puzzle was created by h1alexbel on 04-May-23.
Estimate: 60 minutes,  role: DEV.
If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

@l3r8yJ this is your task now, please go ahead. Deadline (when this ticket should be closed) is 2023-05-14T12:12:30.257390.
Estimation here is 60 minutes, that's how much you will be paid.
Remember, you don't have to solve everything in this ticket - you can solve it partially and leave todo markers in the code, which will become future tasks.
If you have any questions don't ask me, I'm not a technical person. Open new tickets instead.
If you don't have time or simply don't want to work on this, you can always resign.

@l3r8yJ Don't forget to close this ticket before the deadline (2023-05-14T12:12:30). You are past the first half of the allowed period.

@zoeself deregister

@zoeself deregister

@h1alexbel ok, I've removed this task from scope. I'm not managing it anymore.

@l3r8yJ depends on #120

@0pdd the puzzle #373 is still not solved.

@0pdd the only puzzle #373 is solved here.


When an operation is triggered from azure and in progress, any operation triggered from Rancher is omitted

Versions:

Rancher: v2.9-9cec8a218821a7f4b89ff4882425647fe89a2963-head
Operator:  rancher-aks-operator:104.3.0+up1.9.3-rc.1

Steps to reproduce:

For a rancher-provisioned/imported AKS cluster, trigger an update from Azure console, e.g. scaling a nodepool.
While the update is in progress, trigger an update from Rancher Dashboard, e.g. add a nodepool

Actual Results:
The following message appears in the operator logs:
time="2024-10-07T09:38:46Z" level=info msg="Waiting for non-Rancher initiated cluster update for [pvala-aks-sync (id: c-4759x)]"

The update from the Azure console appears on Rancher dashboard after it completes, but the Rancher-triggered update is discarded.
Expected Results:
Once the update from the Azure console completes, the Rancher-triggered update must be put into action. If this is not possible, then an error message must appear on Dashboard saying that user needs to wait until non-rancher initiated update completes before attempting to update.
Other Notes:
If k8s upgrade from Azure is triggered to vX+1, and k8s upgrade from Rancher is triggered to vX+2, the following message appears in logs, but rancher-triggered upgrade goes through once azure-triggered upgrade completes.
time="2024-10-07T10:27:18Z" level=info msg="Updating kubernetes version to 1.28.11 for cluster [pvala-aks-sync (id: c-4759x)]"
time="2024-10-07T10:27:25Z" level=error msg="can't update the AKS cluster with error: PUT https://management.azure.com/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourceGroups/pvala-aks-rg/providers/Microsoft.ContainerService/managedClusters/pvala-aks-sync\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n  \"code\": \"OperationNotAllowed\",\n  \"details\": null,\n  \"message\": \"Operation is not allowed: Another operation (Upgrading) is in progress, please wait for it to finish before starting a new operation. See https://aka.ms/aks-pending-operation for more details. The previous operation started at '2024-10-07T10:25:36Z' and elapsed time is: '00:01:48' (RFC3339 format)\",\n  \"subcode\": \"\"\n}\n--------------------------------------------------------------------------------\n"
time="2024-10-07T10:27:25Z" level=error msg="error syncing 'cattle-global-data/c-4759x': handler aks-controller: failed to update cluster: PUT https://management.azure.com/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourceGroups/pvala-aks-rg/providers/Microsoft.ContainerService/managedClusters/pvala-aks-sync\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n  \"code\": \"OperationNotAllowed\",\n  \"details\": null,\n  \"message\": \"Operation is not allowed: Another operation (Upgrading) is in progress, please wait for it to finish before starting a new operation. See https://aka.ms/aks-pending-operation for more details. The previous operation started at '2024-10-07T10:25:36Z' and elapsed time is: '00:01:48' (RFC3339 format)\",\n  \"subcode\": \"\"\n}\n--------------------------------------------------------------------------------\n, requeuing"

We need a more concise definition of how the sync works between Azure and Rancher. Currently, the behavior is confusing. The behavior is inconsistent, e.g. if Azure triggered action is k8s upgrade, adding a nodepool from rancher seems to trigger k8s upgrade instead.

When a rancher-initiated action is triggered in parallel to a non-rancher-initiated action, the cluster sometimes does not go into the Updating state as it should, even if it succeeds.

Sync operation seems to behave differently when rancher-initiated action is triggered via API.
E.g. for a cluster with multiple nodepools: agentpool, poo1, pool2, and pool3.

Delete pool2 from Azure console.
While the deletion is in progress, delete pool3 from rancher.
Wait for a few minutes

When this is done via UI, pool2 is restored, but when done via API, pool2 stays deleted, and pool3 deletion is ignored.

I was able to reproduce this issue. However, I believe this is related to how syncing in Rancher works. Let me try to explain why I think that is.
In addition to the controller that attempts to reconcile the desired spec with the cloud provider, Rancher does a refresh of the current spec at a specified interval, set by default to 5 mins. This is the indirect mechanism that allows the operator to detect any changes that happen on the cloud provider side, and attempt to reconcile them. You can see this in action, by enabling debug logs for the Rancher pod and then grepping the log output:
kubectl logs -n cattle-system deployments/rancher -f | grep -A 2 -i 'checking cluster'

If there is no change on the cloud provider side, you should see logs similar to these:
2024/10/21 14:15:31 [DEBUG] checking cluster [c-z5d2l] upstream state for changes
2024/10/21 14:15:33 [DEBUG] updating cluster [c-z5d2l], upstream change detected
2024/10/21 14:15:33 [DEBUG] cluster [c-z5d2l] matches upstream, skipping spec sync

The last log line indicates that cluster spec matches upstream so there's nothing to do/update.
However, if there is a change on the cloud provider, Rancher will refresh the spec accordingly.
2024/10/21 14:25:35 [DEBUG] checking cluster [c-z5d2l] upstream state for changes
2024/10/21 14:25:36 [DEBUG] updating cluster [c-z5d2l], upstream change detected
2024/10/21 14:25:36 [DEBUG] change detected for cluster [c-z5d2l], updating spec

At that point, the spec changes, the controller detects the spec has changed and runs its reconcile loop again. Since the spec is up-to-date, the controller typically will do nothing as shown in its logs:
INFO[1162] Checking configuration for cluster [yiannis-aks-01 (id: c-z5d2l)]
INFO[1163] Configuration for cluster [yiannis-aks-01 (id: c-z5d2l)] was verified

So far we have established that the controller begins its reconcile loop when there is a change in the spec (either via a user changing the cluster in Rancher UI or via a change on the cloud provider side at a specified interval) and that the controller uses (almost) up-to-date information about the cluster's state.
Let's explore the scenario when a change happens in Rancher right after a change has happened upstream. Assuming those changes happen at around the same time and Rancher has not refreshed the spec in between those changes, the controller will detect the Rancher change and attempt to reconcile it. Currently, the controller checks the status of the cluster upstream and if it's in progress or updating, it will log a line indicating that a change has happened upstream (Waiting for non-Rancher initiated cluster update ...) and return without any errors.

In short, the last write wins.

Do you mean the first write?

This is very likely the main reason we've put this note in our docs, which I didn't quite fully understand up until now.

It does sum up all that you explained. Thank you for looking into this. I think we will need to modify our tests to take this behavior into account.
My takeaway is to avoid simultaneous change if possible.

If k8s upgrade from Azure is triggered to vX+1, and k8s upgrade from Rancher is triggered to vX+2, the following message appears in logs, but rancher-triggered upgrade goes through once azure-triggered upgrade completes.

What is your take on this @yiannistri?

Do you mean the first write?

I meant the last write because every time the spec gets updated (however this may happen) the controller will start a reconcile loop and use the spec (and status) as the source of truth for applying any changes. So whichever process happens to write to the spec last, it is the changes of that process that will be applied eventually.

My takeaway is to avoid simultaneous change if possible.

Yeah I think that would be sensible. I am very aware that the way this works currently can cause issues for some users but it appears that this was build this way on purpose (whatever that purpose was).

@mjura shouldn't this be scheduled for 2.9.x instead ? 🤔

@mjura shouldn't this be scheduled for 2.9.x instead ? 🤔

@yiannistri  @valaparthvi It is not issue, it is the way how in Rancher we treat source of true. There always some cases when users have to decide in which way they want to apply changes to cluster configuration. Rancher with hosted providers works as orchestration to another orchestration in Public cloud. Public Clouds they automating more and more stuff and we will have to decide which configuration is the correct one. In the Rancher case users have to ensure that there are no changes applied on Cloud Provider site and apply then with Rancher. Eventually repeat this operation once again. For CAPI it will work in this same way.

@yiannistri As solution to this I would go and update documentation comment which we have already.

It is not issue, it is the way how in Rancher we treat source of true. There always some cases when users have to decide in which way they want to apply changes to cluster configuration. Rancher with hosted providers works as orchestration to another orchestration in Public cloud. Public Clouds they automating more and more stuff and we will have to decide which configuration is the correct one. In the Rancher case users have to ensure that there are no changes applied on Cloud Provider site and apply then with Rancher. Eventually repeat this operation once again. For CAPI it will work in this same way.

@mjura We have test cases checking things such as:

editing same/different fields from rancher and cloud console simultaneously
deleting nodepool from cloud console, and editing the same nodepool from rancher
upgrading k8s version simultaneously from both rancher and cloud console to same/different versions

Are these test cases not valid then?

As QA test cases to check if there are some errors, I would say that they are valid. But from production real user experience they are not, since we always tell users to use Rancher for managing downstream cluster if it was created from Rancher and for imported clusters always pick one source of true and don't reconfiguration from both places otherwise you will see a lot of race conditions.

Okay. Keeping in mind that users are advised to choose one source of truth, I think it would make sense for us to test and automate that if changes made from the cloud console are reflected back to Rancher, rather than spending time, fighting race conditions with these simultaneous changes.

Considering that we already warn users about race conditions occurring on simultaneous changes, this is a known issue that we don't intend to fix.
We (QA) have refactored our test suite to include more real-time test cases (make changes from Cloud console and ensure they are synced back to Rancher) and removed tests related to this issue (simultaneous changes from both Rancher and cloud console).
Thanks, Yiannis for your deep dive on this issue, it was enlightening, and Michal for your final input.


Cache body for form requests

Describe the bug
I have a gateway application configured to route requests to some backend service. The app has a single page which sends POST requests to this service. It also has a web filter which tries to read form data from the requests.
The issue is, form POSTs hang and eventually timeout. It appears to be caused by the web filter reading the form data, as AJAX POST requests (using query params instead of form data) do work. See the sample.
Gateway version: 2.2.1.RELEASE
Sample
(originally reported against Spring Security, hence the name, but this is reproducible with only the gateway)
https://github.com/SmithJosh/spring-security-8026/

See 58834daf6271c3ae379f5eb0939689ad6a14036e this is the same problem with the HiddenHttpMethodFilter from webflux.
The framework team decided to not do anything about it. See https://github.com/spring-projects/spring-framework/issues/21824.
Gateway disables the HiddenHttpMethodFilter in anger. Not sure what to do about this.

@rstoyanchev I know framework decided against caching, but is there something gateway could do to mitigate this?

I suppose in the routing filter if it's a form data or multipart data (suffers same problem), gateway could use the writers to serialize back to DataBuffers

We are caching through ServerWebExchange#getFormData(). Yes, gateway could serialize form data but since reading the body (even indirectly via getFormData()) competes with Gateway trying to also read in order to forward, is this supported? Or rather what is the official way  to access the content of a form request?
If it is through ServerWebExchange then yes you'll need to serialize form data. Or perhaps through some other way like  gateway filter or predicate. Either way I imagine gateway needs to be aware since you don't want to automatically assume form data has been accessed which would then require reading and serializing the form data.

We are caching through ServerWebExchange#getFormData() and yes gateway could serialize form data but since reading the body, even indirectly via getFormData(), competes with Gateway trying to also read in order to forward, is this supported?

In my opinion, this should be supported. There are legitimate use cases requiring reading form data at the gateway. Checking for CSRF tokens is my use case, but HiddenHttpMethodFilter needs to read it too, and in general it seems like a reasonable thing to support at the gateway.
I see Spring has the ContentCachingRequestWrapper supporting a multi-read use case for the Servlet API. Is there something similar for a ServerHttpRequest?

We do have caching for the request, but it only starts when something matches a gateway route, after a normal WebFliter may have already accessed the form data and only in certain cases (reading the body or retrying with a body).

I imagine gateway needs to be aware one way or another that the application is accessing form data

How would it know that? This case is shown in a WebFilter before reaching the gateway entry point.

Maybe we could override HttpWebHandlerAdapter and extend DefaultServerWebExchange so we can mark when getFormData() or getMultipartData() are called?

In my opinion, this should be supported

Yes, question is how.

HiddenHttpMethodFilter needs to read it

HiddenHttpMethodFilter has no role to play and should never be needed on a request forwarded by the gateway.

We do have caching for the request, but it only starts when something matches a gateway route

Would it make sense to enhance this to provide access to form data somehow? So that it's done in the context of something that gateway handles and understands.
Or perhaps there could be a gateway-provided base WebFilter for access to form data?

Or perhaps there could be a gateway-provided base WebFilter for access to form data?

Something needs to be provided for normal WebFilters. Should it be transparent (custom ServerWebExchange, etc...) or explicit. The latter will be simpler.

Would that also include the ability to read the body, irrespective of form data?

It certainly could. We have the framework for that already for both serialized body (predicates) and raw body (retry filters).

So then perhaps wrapping the request to intercept getBody and essentially cache it if accessed might be something to try.

@spencergibb Any update on this?

no

@spencergibb
Is this issue still being discussed in the team? We are experiencing the same problem when having an authorization server behind a Spring Cloud Gateway server. If CSRF protection is enabled, the submit of the login form hangs and eventually results in a timeout. Possible workarounds are disabling CSRF for the login which is most probably not a good idea or updating the login screen to use an ajax request instead of a standard form submit.

@SmithJosh
FYI: I was able to fix this by creating a filter that caches the incoming request in case it contains form data.
@Component
@RequiredArgsConstructor
public class CacheRequestBodyFilter implements WebFilter {

    private final ServerCodecConfigurer codecConfigurer;
    private final WebSessionManager sessionManager;
    private final LocaleContextResolver localeContextResolver;

    @Override
    public Mono<Void> filter(ServerWebExchange serverWebExchange, WebFilterChain chain) {
        if (requestContainsFormData(serverWebExchange)) {
            return cacheRequestBody(serverWebExchange, (cachedRequest) -> {
                var exchange = new DefaultServerWebExchange(cachedRequest, serverWebExchange.getResponse(), sessionManager, codecConfigurer, localeContextResolver);
                return chain.filter(exchange);
            });
        }

        return chain.filter(serverWebExchange);
    }

    private boolean requestContainsFormData(ServerWebExchange serverWebExchange) {
        var contentType = serverWebExchange.getRequest().getHeaders().getContentType();
        return APPLICATION_FORM_URLENCODED.isCompatibleWith(contentType);
    }
}

Then I configure this filter to be applied just before the Spring Security CSRF WebFilter
@Bean
    public SecurityWebFilterChain springSecurityFilterChain(ServerHttpSecurity http, CacheRequestBodyFilter cacheRequestBodyFilter) {
        return http
               ....
                .addFilterBefore(cacheRequestBodyFilter, CSRF)
               ....
                .build();
    }

So whenever the getFormData() method of DefaultServerWebExchange is subscribed, it will make use of a cached ServerHttpRequest to parse the body to a MultiValueMap in the DefaultServerWebExchange class.

I think we are seeing direct memory leak when using this workaround (and eventually throwing out of direct memory). After removing our filter for caching request body the direct memory stops leaking.

It looks like ServerWebExchangeUtils.cacheRequestBody is creating composite DataBuffer which seems to be not released correctly. I guess netty will release the original buffer when connection is closed but this composite buffer is not released.
I think spring should handle this (in the cache util maybe?), but as a workaround we could release the cached composite buffer manually when the mono is completed by adding this kind of transformDeferred operation:
    @Override
    public Mono<Void> filter(ServerWebExchange serverWebExchange, WebFilterChain chain) {
        if (requestContainsFormData(serverWebExchange)) {
            return ServerWebExchangeUtils.cacheRequestBody(serverWebExchange, (cachedRequest) -> {
                var exchange = new DefaultServerWebExchange(cachedRequest, serverWebExchange.getResponse(), sessionManager, codecConfigurer, localeContextResolver);
                return chain.filter(exchange);
            }).transformDeferred(call -> {
                return call.doFinally(t -> {
                    DataBuffer cachedBody = serverWebExchange.getAttributeOrDefault(CACHED_REQUEST_BODY_ATTR, null);
                    if (cachedBody != null) {
                        DataBufferUtils.release(cachedBody);
                    }
                });
            });
        }

        return chain.filter(serverWebExchange);
    }

There is RemoveCachedBodyFilter for spring-cloud-gateway (it's GlobalFilter) but this workaround is a WebFilter (and we want to run csrf check before gateway chain..) Also the javadoc anywhere did not really tell you to make sure the cached buffer is released somewhere. So if there is people using this workaround make sure you are handling releasing the buffer somehow. :)

@tuomoa I used your workaround, but the memory still leaks.

@FrankChen021 Interesting. We are not seeing memory leak after doing pretty much the fix proposed above.

So, How to repeatedly read FormData？

The problem still exists in the latest version, is there a solution? @spencergibb @rstoyanchev

@tuomoa In my case, your method also memory leak.

@tuomoa @spencergibb The difference is that I cache body used to get MultipartData. I using your latest code, memory is still leaking. I still hope that the official can provide the method of getMultipartData.

@tuomoa @spencergibb The difference is that I cache body used to get MultipartData. I using your latest code, memory is still leaking. I still hope that the official can provide the method of getMultipartData.

Maybe you should override the getMultipartData in the decorator instead of getFormData because that has the same problem (original exchange has created multipartMono which references the original body.

@tuomoa In my case, your method also memory leak.

@FrankChen021 @tianshuang @lsh1358046425 Our WebFilter has @Order(Ordered.HIGHEST_PRECEDENCE + 2)
It's been quite a while when we did create the workaround, but here is our complete filter code if it helps. Looks like we did end up having a custom decorator for the cachedRequest instead of using the DefaultServerWebExchange. Do note that we only cache when request contains formdata. Your use case might be different.
package demo;

import org.springframework.cloud.gateway.support.ServerWebExchangeUtils;
import org.springframework.core.Ordered;
import org.springframework.core.ResolvableType;
import org.springframework.core.annotation.Order;
import org.springframework.core.codec.Hints;
import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.core.io.buffer.DataBufferUtils;
import org.springframework.http.InvalidMediaTypeException;
import org.springframework.http.MediaType;
import org.springframework.http.codec.HttpMessageReader;
import org.springframework.http.codec.ServerCodecConfigurer;
import org.springframework.util.CollectionUtils;
import org.springframework.util.LinkedMultiValueMap;
import org.springframework.util.MultiValueMap;
import org.springframework.web.server.ServerWebExchange;
import org.springframework.web.server.ServerWebExchangeDecorator;
import org.springframework.web.server.WebFilter;
import org.springframework.web.server.WebFilterChain;
import reactor.core.publisher.Mono;

import static org.springframework.cloud.gateway.support.ServerWebExchangeUtils.CACHED_REQUEST_BODY_ATTR;
import static org.springframework.http.MediaType.APPLICATION_FORM_URLENCODED;

/**
 * Decorate exchange and request to cache body form data.
 * Related to https://github.com/spring-cloud/spring-cloud-gateway/issues/1587
 *
 * Without this Csrf filter would read the body when reading formData and after that
 * when some other component tries to read the body it fails. With this, body is cached.
 *
 * Without decorating the server web exchange the formData reading accesses the original mono initialised
 * in {@link org.springframework.web.server.adapter.DefaultServerWebExchange} and would cause body to be
 * not cached. When decorator overrides it, read will actually cause the body cache to be populated.
 *
 * This filter needs to be before csrf filter in the chain.
 */
@Order(Ordered.HIGHEST_PRECEDENCE + 2)
public class RequestBodyCachingFilter implements WebFilter {

    private final ServerCodecConfigurer configurer;

    public RequestBodyCachingFilter(ServerCodecConfigurer configurer) {
        this.configurer = configurer;
    }

    @Override
    public Mono<Void> filter(ServerWebExchange exchange, WebFilterChain chain) {
        DataBuffer body = exchange.getAttributeOrDefault(CACHED_REQUEST_BODY_ATTR, null);

        if (body != null) {
            return chain.filter(exchange);
        }

        // Only cache if form data (because csrf inside only form data)
        if (requestContainsFormData(exchange)) {

            return ServerWebExchangeUtils.cacheRequestBody(exchange, (serverHttpRequest) -> {

                // don't mutate and build if same request object
                if (serverHttpRequest == exchange.getRequest()) {
                    return chain.filter(exchange);
                }

                return chain.filter(new FormDataServerWebExchangeDecorator(
                        exchange.mutate().request(serverHttpRequest).build(),
                        configurer));

            }).transformDeferred(call -> call.doFinally(t -> {
                /**
                 * See https://github.com/spring-cloud/spring-cloud-gateway/issues/1587
                 * Spring cacheRequestBody does not seem to release cached databuffer correctly and
                 * this leads to memory leak. Release the buffer manually after mono is completed.
                 */
                DataBuffer cachedBody = exchange.getAttributeOrDefault(CACHED_REQUEST_BODY_ATTR, null);
                if (cachedBody != null) {
                    DataBufferUtils.release(cachedBody);
                }
            }));
        }

        return chain.filter(exchange);
    }

    private static boolean requestContainsFormData(ServerWebExchange serverWebExchange) {
        var contentType = serverWebExchange.getRequest().getHeaders().getContentType();
        return APPLICATION_FORM_URLENCODED.isCompatibleWith(contentType);
    }

    private static class FormDataServerWebExchangeDecorator extends ServerWebExchangeDecorator {

        private static final Mono<MultiValueMap<String, String>> EMPTY_FORM_DATA =
                Mono.just(CollectionUtils.unmodifiableMultiValueMap(new LinkedMultiValueMap<String, String>(0))).cache();
        private static final ResolvableType FORM_DATA_TYPE =
                ResolvableType.forClassWithGenerics(MultiValueMap.class, String.class, String.class);

        private final ServerCodecConfigurer configurer;

        public FormDataServerWebExchangeDecorator(ServerWebExchange delegate, ServerCodecConfigurer configurer) {
            super(delegate);
            this.configurer = configurer;
        }

        @Override
        public Mono<MultiValueMap<String, String>> getFormData() {
            try {
                MediaType contentType = getDelegate().getRequest().getHeaders().getContentType();
                if (APPLICATION_FORM_URLENCODED.isCompatibleWith(contentType)) {
                    return ((HttpMessageReader<MultiValueMap<String, String>>) configurer.getReaders().stream()
                            .filter(reader -> reader.canRead(FORM_DATA_TYPE, APPLICATION_FORM_URLENCODED))
                            .findFirst()
                            .orElseThrow(() -> new IllegalStateException("No form data HttpMessageReader.")))
                            .readMono(FORM_DATA_TYPE, getDelegate().getRequest(), Hints.none())
                            .switchIfEmpty(EMPTY_FORM_DATA)
                            .cache();
                }
            } catch (InvalidMediaTypeException ex) {
                // Ignore
            }
            return EMPTY_FORM_DATA;
        }

    }

}

Edit: Now that I think about it, I think the problem with my initial fix in the earlier comment is that it creates a whole new ServerWebExchange so it might end up having the same attribute in the initial exchange and the exchange passed to the chain and only the initial exchange attribute is cleaned.
The filter we are using does not do this as it only mutates the initial exchange request. So we have to do exchange.mutate().request(serverHttpRequest).build() instead of new DefaultWebServerExchange. But now we arrive to the point why we need the FormDataServerWebExchangeDecorator. The initial exchange that is passed to this filter already has formDataMono field (and some filter such as CsrfWebFilter might use it) which has reference to the original body and not the decorated cached body. BUT when we decorate the original exchange here and override the getFormData the subsequent calls will use the decorated cached body and it will work. Probably something like this is the reasoning behind our implementation.. it's been a while so I might not remember all the details... 😅

In fact, Gateway has provided org.springframework.cloud.gateway.filter.RemoveCachedBodyFilter to perform cleaning operations, which has the highest priority, and theoretically we do not need to manually clean up.

In fact, Gateway has provided global org.springframework.cloud.gateway.filter.RemoveCachedBodyFilter to perform cleaning operations, which has the highest priority, and theoretically we do not need to manually clean up.

Yeap. But like I mentioned in https://github.com/spring-cloud/spring-cloud-gateway/issues/1587#issuecomment-862172641 that didn't seem to help when there is for example Csrf filter before the gateway chain and the
RemoveCachedBodyFilter is in the gateway chain. If Csrf block the request it does not go through the gateway chain I think.

@tianshuang Did you get it working by overriding the getMultipartData() similarly as we did with the getFormData() in the decorator?

@tuomoa , In your scenario, you do need to explicitly release the dataBuffer because the WebFilter is executed before all GlobalFilters, but in my scenario, the requsetBody acquisition is executed after RemoveCachedBodyFilter, so I don't need to explicitly release it.

@tianshuang glad you got it working! Yeah I suppose because in our case we have for example the Csrf filter before the gateway chain we do need to make sure databuffer is always released correctly. I suppose this is not so trivial what needs. to be done and how so it would be great if there was some implementation by the Spring Cloud Gateway project supporting these usage scenarios also.

@tianshuang Did you get it working by overriding the getMultipartData() similarly as we did with the getFormData() in the decorator?

There is a little difference, I did not get the request body in WebFilter, but in GlobalFilter, and I did not observe direct memory leaks, my GlobalFilter source code is as follows:
@Component
public class CacheRequestBodyAndRequestFilter implements GlobalFilter, Ordered {

    public static final String CACHED_REQUEST_BODY_X_WWW_FORM_URLENCODED_MAP_KEY = "cachedXWwwFormUrlEncodedMap";
    public static final String CACHED_REQUEST_BODY_FORM_DATA_MAP_KEY = "cachedRequestBodyFormDataMap";

    public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
        Supplier<? extends Mono<? extends Void>> supplier = () -> chain.filter(exchange);

        MediaType contentType = exchange.getRequest().getHeaders().getContentType();
        if (MediaType.APPLICATION_FORM_URLENCODED.isCompatibleWith(contentType)) {
            return ServerWebExchangeUtils.cacheRequestBodyAndRequest(exchange,
                    (serverHttpRequest) -> ServerRequest
                            .create(exchange.mutate().request(serverHttpRequest).build(), HandlerStrategies.withDefaults().messageReaders())
                            .bodyToMono(new ParameterizedTypeReference<MultiValueMap<String, String>>() {
                            })
                            .doOnNext(map -> {
                                exchange.getAttributes().put(CACHED_REQUEST_BODY_X_WWW_FORM_URLENCODED_MAP_KEY, map);
                                log.info("x-www-urlencoded: {}", toStringToStringListMap(map));
                            })).then(Mono.defer(supplier));
        } else if (MediaType.MULTIPART_FORM_DATA.isCompatibleWith(contentType)) {
            return ServerWebExchangeUtils.cacheRequestBodyAndRequest(exchange,
                    (serverHttpRequest) -> ServerRequest
                            .create(exchange.mutate().request(serverHttpRequest).build(), HandlerStrategies.withDefaults().messageReaders())
                            .bodyToMono(new ParameterizedTypeReference<MultiValueMap<String, Part>>() {
                            })
                            .doOnNext(map -> {
                                exchange.getAttributes().put(CACHED_REQUEST_BODY_FORM_DATA_MAP_KEY, map);
                                log.info("from-data: {}", toStringToStringListMap(map));
                            })).then(Mono.defer(supplier));
        } else {
            return chain.filter(exchange);
        }
    }

    private Map<String, List<String>> toStringToStringListMap(MultiValueMap<String, ?> multiValueMap) {
        Map<String, List<String>> map = new LinkedHashMap<>();
        for (Map.Entry<String, ? extends List<?>> entry : multiValueMap.entrySet()) {
            String key = entry.getKey();
            List<?> valueList = entry.getValue();
            map.put(key, valueList.stream().map(value -> {
                if (value instanceof String) {
                    return (String) value;
                } else if (value instanceof FormFieldPart) {
                    return ((FormFieldPart) value).value();
                } else {
                    return "";
                }
            }).collect(Collectors.toList()));
        }

        return map;
    }

    @Override
    public int getOrder() {
        return Ordered.HIGHEST_PRECEDENCE + 1; // After RemoveCachedBodyFilter, before AdaptCachedBodyGlobalFilter
    }

}

I didn't release the dataBuffer explicitly because RemoveCachedBodyFilter would do it for me.

I tried this approach and I still dont see formData sent to my underlying rest services
Im facing this issue
https://github.com/spring-projects/spring-security/issues/11620

https://github.com/spring-projects/spring-security/issues/11620

https://github.com/spring-cloud/spring-cloud-gateway/pull/2838 https://github.com/spring-cloud/spring-cloud-gateway/pull/2842 I see that many memory leaks have been fixed recently. It is recommended to upgrade to the corresponding version. @FrankChen021 @tianshuang @tuomoa

Hello guys, I still can't find solution above how to get key & value from Multipart Form Data.
Is there any solution for that?
Thanks

@tianshuang Did you get it working by overriding the getMultipartData() similarly as we did with the getFormData() in the decorator?

There is a little difference, I did not get the request body in WebFilter, but in GlobalFilter, and I did not observe direct memory leaks, my GlobalFilter source code is as follows:
@Component
public class CacheRequestBodyAndRequestFilter implements GlobalFilter, Ordered {

    public static final String CACHED_REQUEST_BODY_X_WWW_FORM_URLENCODED_MAP_KEY = "cachedXWwwFormUrlEncodedMap";
    public static final String CACHED_REQUEST_BODY_FORM_DATA_MAP_KEY = "cachedRequestBodyFormDataMap";

    public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
        Supplier<? extends Mono<? extends Void>> supplier = () -> chain.filter(exchange);

        MediaType contentType = exchange.getRequest().getHeaders().getContentType();
        if (MediaType.APPLICATION_FORM_URLENCODED.isCompatibleWith(contentType)) {
            return ServerWebExchangeUtils.cacheRequestBodyAndRequest(exchange,
                    (serverHttpRequest) -> ServerRequest
                            .create(exchange.mutate().request(serverHttpRequest).build(), HandlerStrategies.withDefaults().messageReaders())
                            .bodyToMono(new ParameterizedTypeReference<MultiValueMap<String, String>>() {
                            })
                            .doOnNext(map -> {
                                exchange.getAttributes().put(CACHED_REQUEST_BODY_X_WWW_FORM_URLENCODED_MAP_KEY, map);
                                log.info("x-www-urlencoded: {}", toStringToStringListMap(map));
                            })).then(Mono.defer(supplier));
        } else if (MediaType.MULTIPART_FORM_DATA.isCompatibleWith(contentType)) {
            return ServerWebExchangeUtils.cacheRequestBodyAndRequest(exchange,
                    (serverHttpRequest) -> ServerRequest
                            .create(exchange.mutate().request(serverHttpRequest).build(), HandlerStrategies.withDefaults().messageReaders())
                            .bodyToMono(new ParameterizedTypeReference<MultiValueMap<String, Part>>() {
                            })
                            .doOnNext(map -> {
                                exchange.getAttributes().put(CACHED_REQUEST_BODY_FORM_DATA_MAP_KEY, map);
                                log.info("from-data: {}", toStringToStringListMap(map));
                            })).then(Mono.defer(supplier));
        } else {
            return chain.filter(exchange);
        }
    }

    private Map<String, List<String>> toStringToStringListMap(MultiValueMap<String, ?> multiValueMap) {
        Map<String, List<String>> map = new LinkedHashMap<>();
        for (Map.Entry<String, ? extends List<?>> entry : multiValueMap.entrySet()) {
            String key = entry.getKey();
            List<?> valueList = entry.getValue();
            map.put(key, valueList.stream().map(value -> {
                if (value instanceof String) {
                    return (String) value;
                } else if (value instanceof FormFieldPart) {
                    return ((FormFieldPart) value).value();
                } else {
                    return "";
                }
            }).collect(Collectors.toList()));
        }

        return map;
    }

    @Override
    public int getOrder() {
        return Ordered.HIGHEST_PRECEDENCE + 1; // After RemoveCachedBodyFilter, before AdaptCachedBodyGlobalFilter
    }

}

I didn't release the dataBuffer explicitly because RemoveCachedBodyFilter would do it for me.

Thank you. this actually helped me a lot.


[windows] introduce dedicated PowerShell configuration step

Description
new "Configure-PowerShell" step is introduced which takes care of both system wide and
temporary user configuration
Related issue:
https://github.com/github/c2c-actions-akvelon/issues/38
Check list

[ ] Related issue / work item is attached
[ ] Tests are written (if applicable)
[ ] Documentation is updated (if applicable)
[ ] Changes are tested and related VM images are successfully generated

/azp run windows2019,windows2022


Update the Owner and Member Groups in dbt Cloud Enterprise section of the self-service permissions page

Contributions

[X] I have read the contribution docs, and understand what's expected of me.

Link to the page on docs.getdbt.com requiring updates
https://docs.getdbt.com/docs/cloud/manage-access/self-service-permissions
What part(s) of the page would you like to see updated?
We've made a lot of changes to licenses and roles recently, the section regarding Owner and Member Groups in dbt Cloud Enterprise could use refreshing
Additional information
No response

Addressed with https://github.com/dbt-labs/docs.getdbt.com/pull/3462


Wrap stdout/stderr to avoid "Not enough space" in Python 2 on Windows 7

When you set the standard streams to binary you seem to only be able to write up to some limit in a single call to write. So we wrap the streams and write in chunks to avoid this.
A similar hack was also implemented in Python 3 to work around this. But that hack doesn't seem to exist in Python 2.
This might have unintended side effects due to replacing the standard stream objects. See the issue for other possible fixes.
I didn't test that this really fixes the issue yet since I don't have access to Windows 7 currently, if anyone could test I will be glad. Just do (with and without this fix):
import click
click.echo('A' * 100000)

Fixes #825

I was about to merge this, then I realized I just merged #819 which fixed a 16k character limit on Windows. I want to make sure I understand what the difference between the 16k limit there and the 64k limit here is, since I'm not very familiar with Windows. It appears you're implementing a buffered writer, which was handled in the other PR by just using io.BufferedWriter.

I guess the immediate difference is that the other one was for the text stream, and this is the binary stream, but why not use io.BufferedWriter here?

I was about to merge this, then I realized I just merged #819 which fixed a 16k character limit on Windows. I want to make sure I understand what the difference between the 16k limit there and the 64k limit here is, since I'm not very familiar with Windows. It appears you're implementing a buffered writer, which was handled in the other PR by just using io.BufferedWriter.

The other PR was due to TextIOWrapper requiring a buffered stream and it not handling partial writes.
This one is due to some Windows versions (e.g. Windows 7) failing to write to the console past a certain size. StreamWrapper isn't exactly like BufferedWriter as it does no buffering, it merely makes multiple calls to write so as to avoid the arbitrary limit in writing to the console.
But now that I think about it, if the BufferedWriter doesn't ever write more than the limit in one call, it's possible that this is redundant, but I have no Windows 7 installation available to test this.

You can get a VM for testing from MS at https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/. I can try this out later if you can't get to it before me.

I confirmed that this still happens on Windows 7 with the previous PR, and is fixed by this PR. But I'm confused why BufferedWriter didn't take care of it.

I confirmed that this still happens on Windows 7 with the previous PR, and is fixed by this PR. But I'm confused why BufferedWriter didn't take care of it.

Reading the code again more carefully. ConsoleStream has two streams. One for text, and another one for anything else. There is no issue with the text stream because _WindowsConsoleWriter already has the limit in place. The other stream, on the other hand, is just the original object and will happily write whatever size you give it.

It seems like we should just be able to wrap the original stream with a buffered writer as well. I tried
if PY2:
    buffer_stream = io.BufferedWriter(io.FileIO(buffer_stream, "wb"))

but that didn't seem to fix it.

I'd guess than, that when passed a large buffer, BufferedWriter just writes it to the file, without splitting it up to smaller writes.

This is the last issue marked for the 7.0 milestone. It would be great to get this done over the weekend. Let me know if you need anything.

This is the last issue marked for the 7.0 milestone. It would be great to get this done over the weekend. Let me know if you need anything.

I reworked the PR and checked that it works on Windows 7.


Spotify output disapear after the count

After install the last version of lemonbuddy my spotify script does not work anymore. The output disappear after the count (in this case set to 100). Script link: http://pastebin.com/CPCRj4tk

Note: If i try without scroller it works flawless.

Note: after the update, the script stoped working at all. Even calling it from the terminal the crash  happens after the count.

Hi, could you show me the module configuration that uses the script

The script works fine (i removed scroller since I don't have it installed) and the sleep calls since it can be specified in the configuration instead.
[module/spotify]
type = custom/script
exec = ~/sandbox/spotify-test
interval = 10

Hi, without "tail", the module just dont appear .The module is:
[module/Spotify]
type = custom/script
exec = ~/.config/lemonbuddy/Scripts/Spotify/Spotify.sh
tail = true
format-padding = 2
format-foreground = ${bar/Bar.foreground}
format-foreground = ${bar/Bar.background}
format-background = #0d000000
format-overline = ${bar/Bar.background}
format-underline = ${bar/Bar.background}

Can you please add a truncate/ellipsis text function to custom modules ? I only use scroller for this.

Could you try the script without both scroller and the sleep's and tail = false
#!/bin/bash
#
# Show Spotify current song

spotifyon=$(ps -e |grep spotify)
if [ -z "$spotifyon" ]; then
	echo ""
else
artist=`dbus-send --print-reply --dest=org.mpris.MediaPlayer2.spotify /org/mpris/MediaPlayer2 org.freedesktop.DBus.Properties.Get string:'org.mpris.MediaPlayer2.Player' string:'Metadata'|egrep -A 2 "artist"|egrep -v "artist"|egrep -v "array"|cut -b 27-|cut -d '"' -f 1|egrep -v ^$`
title=`dbus-send --print-reply --dest=org.mpris.MediaPlayer2.spotify /org/mpris/MediaPlayer2 org.freedesktop.DBus.Properties.Get string:'org.mpris.MediaPlayer2.Player' string:'Metadata'|egrep -A 1 "title"|egrep -v "title"|cut -b 44-|cut -d '"' -f 1|egrep -v ^$`
	echo "|" $artist "-" $title
fi

The script module already have maxlen and ellipsis options. See the wiki

I removed scroller and sleep but, with tail = false the bar does not appear.

And sadly the ellipsis is not working.

Are you using the latest version?

Yeah, the test module just works fine.

Just in case you dont understand my bad english: https://github.com/kbrgl/scroller

It should be fixed now... Please verify..
Notes:

Use tail = true to tail the scroller output
Remove sleeps
Set interval = N in the module config instead

Thanks for the support, it was awesome. Now i have the module showing, even if its turn on and off all the time (i can live with that) but, the cpu usage increased a lot. Anyway, again thanks, i really appreciate your job.

yes I just noticed the cpu usage... looks like a spin lock that's not, i will patch it

If i set scroller count to 10, for example, cpu usage increase to 20%/30%. This is okay, but the module looks like the christimas lights.

Not really sure how you want it to be displayed.. but this works fine for me.. no noticable cpu usage
[module/spotify]
type = custom/script
; all of this could of course be put in a single script... or just move the piping to the spotify script
exec =  ~/bin/spotify_status.sh %artist% - %title% | while read -r status; do echo $status | scroller -c $((${#status}*2)) -i 0.2 -s "   --   "; do
; tail since scroller will output continuously
tail = true
; when the script has ended, wait this time and then launch it again
interval = 2

#!/bin/sh
main() {
  if ! pgrep -x spotify >/dev/null; then
    echo "spotify is off"; exit
  fi

  cmd="org.freedesktop.DBus.Properties.Get"
  domain="org.mpris.MediaPlayer2"
  path="/org/mpris/MediaPlayer2"

  meta=$(dbus-send --print-reply --dest=${domain}.spotify \
    /org/mpris/MediaPlayer2 org.freedesktop.DBus.Properties.Get string:${domain}.Player string:Metadata)

  artist=$(echo "$meta" | sed -nr '/xesam:artist"/,+2s/^ +string "(.*)"$/\1/p' | tail -1)
  album=$(echo "$meta" | sed -nr '/xesam:album"/,+2s/^ +variant +string "(.*)"$/\1/p' | tail -1)
  title=$(echo "$meta" | sed -nr '/xesam:title"/,+2s/^ +variant +string "(.*)"$/\1/p' | tail -1)

  echo "${*:-%artist% - %title%}" | sed "s/%artist%/$artist/g;s/%title%/$title/g;s/%album%/$album/g"
}

main "$@"

Everything is working fine now, thank you so much. Keep the awesome work.

This breaks if the artist or title has an & in it. For example, The Civil Wars - Poison & Wine would show as "The Civil Wars - Poison %title% Wine" instead. I just changed it to echo $artist - $title and it seems to work, but I get a feeling you used sed for a reason. If there's a reason to use sed or a better way to do it, I'd appreciate an update. Thanks for all the work you've put into this!

That syntax was used to be able to pass the format to the script ./script %title% %artist% %album%
You probably have to escape the ampersand. Try:
... | sed 's/&/\\&/g'


New source from proxyspy.net

@chill117 i've added unit tests and removed regex as you asked

I almost gave up hope :) Thanks for taking the time to do this. I'll have a closer look later today.


Update components.md

Persona link was going to the picker page.

Learn Build status updates of commit 30f64c3:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

power-platform/guidance/creator-kit/components.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel


[3.2] [HTML5] Handle contextmenu, webglcontextlost internally.

This way they are automatically cleaned up when the engine exits,
landing a hand to browsers garbage collectors.
3.2 version of #45883

Thanks!


Sequence Point undefined behavior in c-do.c

This line in c-do.c is an example of something for which the order of operations is not defined:
// Get first block value:
pvs.path = VAL_BLK_DATA(pvs.orig = *path_val);

(Note that VAL_BLK_DATA is a preprocessor macro, and not a function.)
One might think that means that the compiler could choose to do the pvs.orig = *path_val, then call VAL_BLK_DATA on that, then put the result in pvs.path or put the VAL_BLK_DATA (pvs.orig) into pvs.path, and then do pvs.orig = *path_val.  But as pointed out here, it's not an either-or situation in principle.  It's undefined behavior.
http://stackoverflow.com/a/13714644/211160
Happened across this just by trying to build the sole file c-do.c with warning levels turned up, adding things like -Werror=sequence-point, which would be a good idea in general to do.

Closing old issues.


Add a tf.feature_column like sklearn.preprocessing.MultiLabelBinarizer

System information

TensorFlow version (you are using): 2.3
Are you willing to contribute it (Yes/No):
If I can ......

Describe the feature and the current behavior/state.
tf.feature_column doesn't have any column for dealing with multi-hot feature like sklearn.preprocessing.MultiLabelBinarizer now.
Will this change the current api? How?
I guess not
Who will benefit with this feature?
Everyone who has to deal with multi-hot feature
Any Other info.

@DachuanZhao Tensorflow does have an indicator_column which you can see here that is used for multi-hot representation of a categorical column.
Is that the feature you were looking for?

There are two questions :

I should give indicator_column the vocabulary_list  explicitly .

# this is a df column
df_column_list = ["1,2","3"]
df_column_list = [v.split(",") for v in column_list]

# sklearn doesn't raise
from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer()
mlb.fit_transform(df_column_list)

#tf raise error
test_column = tf.feature_column.categorical_column_with_vocabulary_list("category", df_column_list ,num_oov_buckets=0)

I should padding the df_column_list  first in tf .

test_column = tf.feature_column.categorical_column_with_vocabulary_list("category", ["1","2","3","4"],num_oov_buckets=0)

#This line raises error
#tf.keras.layers.DenseFeatures(tf.feature_column.indicator_column(test_column))({"category":[["1","2"],["2"],["3"],["4"],["3"]]})

#I should padding it first , then it doesn't raise error
tf.keras.layers.DenseFeatures(tf.feature_column.indicator_column(test_column))({"category":[["1","2"],["2","0"],["3","0"],["4","0"],["3","0"]]})

Both of them makes  indicator_column  complicated to use ...

@DachuanZhao I agree with @NikhilBartwal as he correct that it is being used for multi-hot represtation of the given categorical column!!
I think it helped You!

@DachuanZhao Tensorflow does have an indicator_column which you can see here that is used for multi-hot representation of a categorical column.
Is that the feature you were looking for?

There are two questions :

I should give indicator_column the vocabulary_list  explicitly .

# this is a df column
df_column_list = ["1,2","3"]
df_column_list = [v.split(",") for v in column_list]

# sklearn doesn't raise
from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer()
mlb.fit_transform(df_column_list)

#tf raise error
test_column = tf.feature_column.categorical_column_with_vocabulary_list("category", df_column_list ,num_oov_buckets=0)

I should padding the df_column_list  first in tf .

test_column = tf.feature_column.categorical_column_with_vocabulary_list("category", ["1","2","3","4"],num_oov_buckets=0)

#This line raises error
#tf.keras.layers.DenseFeatures(tf.feature_column.indicator_column(test_column))({"category":[["1","2"],["2"],["3"],["4"],["3"]]})

#I should padding it first , then it doesn't raise error
tf.keras.layers.DenseFeatures(tf.feature_column.indicator_column(test_column))({"category":[["1","2"],["2","0"],["3","0"],["4","0"],["3","0"]]})

Both of them makes  indicator_column  complicated to use ...

We don't support feature column with Keras anymore (especially after 2.4) -- Please use tf.keras.layers.experimental.preprocessing.CategoryEncoding and its adapt method (which is equivalent to fit_transform here) and let us know if that fulfills your use case

We don't support feature column with Keras anymore (especially after 2.4) -- Please use tf.keras.layers.experimental.preprocessing.CategoryEncoding and its adapt method (which is equivalent to fit_transform here) and let us know if that fulfills your use case

OK . It doesn't help me .
My problem is that I don't know how to deal with column like this :
import pandas as pd
df = pd.DataFrame( )
df["text_split_by_comma"] = ["1","2,3","4,5","1,6,7,9"]

There isn't any function for dealing with it in tf.data and tf.feature_column simply .

We don't support feature column with Keras anymore (especially after 2.4) -- Please use tf.keras.layers.experimental.preprocessing.CategoryEncoding and its adapt method (which is equivalent to fit_transform here) and let us know if that fulfills your use case

Like this (https://github.com/tensorflow/tensorflow/issues/18572)[https://github.com/tensorflow/tensorflow/issues/18572]

You should read this as string and perform split after you read from csv, not during read from csv

You should read this as string and perform split after you read from csv, not during read from csv

It means I separate preprocess data  from model if I do that . So I have to preprocess data  once again when I call model.predict .

KPL can be part of your model:
class MyModel(tf.keras.Model):
  def __init__(self):
    self.preprocess_layer = ...
    ...
  def call(self, inputs):
    preprocessed_inputs = self.preprocess_layer(inputs)
    output = ...
    return output

We don't support feature column with Keras anymore (especially after 2.4) -- Please use tf.keras.layers.experimental.preprocessing.CategoryEncoding and its adapt method (which is equivalent to fit_transform here) and let us know if that fulfills your use case

You should edit this https://www.tensorflow.org/tutorials/structured_data/feature_columns first if you don't  support feature column with Keras anymore ......

We don't support feature column with Keras anymore (especially after 2.4) -- Please use tf.keras.layers.experimental.preprocessing.CategoryEncoding and its adapt method (which is equivalent to fit_transform here) and let us know if that fulfills your use case

You should edit this https://www.tensorflow.org/tutorials/structured_data/feature_columns first if you don't support feature column with Keras anymore ......

That's not the way how this works. We need to move those experimental layers into core API before we can claim an alternative solution to be deprecated. For now, given they still serve a majority of use cases, they need to live.
But I do agree with the confusion, I think we can make this happen soon.

We don't support feature column with Keras anymore (especially after 2.4) -- Please use tf.keras.layers.experimental.preprocessing.CategoryEncoding and its adapt method (which is equivalent to fit_transform here) and let us know if that fulfills your use case

You should edit this https://www.tensorflow.org/tutorials/structured_data/feature_columns first if you don't support feature column with Keras anymore ......

That's not the way how this works. We need to move those experimental layers into core API before we can claim an alternative solution to be deprecated. For now, given they still serve a majority of use cases, they need to live.
But I do agree with the confusion, I think we can make this happen soon.

You can refer to https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing for tf.keras.layers.experimental.preprocessing  , especially MultiLabelBinarizer and LabelBinarizer . It's really convenient~~~

KPL can be part of your model:
class MyModel(tf.keras.Model):
  def __init__(self):
    self.preprocess_layer = ...
    ...
  def call(self, inputs):
    preprocessed_inputs = self.preprocess_layer(inputs)
    output = ...
    return output

I'm working on include my data processing code into the model, this is exactly what I'd like to use. But I just looked at functions in tf.keras.layers.experimental.preprocessing, it only has around 20 classes, seems very limited. My data processing code is written all in tf-functions, and I'm not able to convert all tf-functions in my data processing code to tf.keras.layers.experimental.preprocessing. Some of my tf functions in data processing code include: tf.strings.split, tf.lookup.StaticHashTable, tf.strings.to_number, tf.sparse.SparseTensor, tf.compat.v1.sparse_to_dense, tf.fill, tf.concat, tf.not_equal, tf.cast. Thanks.    @tanzhenyu @DachuanZhao

We don't support feature column with Keras anymore (especially after 2.4) -- Please use tf.keras.layers.experimental.preprocessing.CategoryEncoding and its adapt method (which is equivalent to fit_transform here) and let us know if that fulfills your use case

You should edit this https://www.tensorflow.org/tutorials/structured_data/feature_columns first if you don't support feature column with Keras anymore ......

That's not the way how this works. We need to move those experimental layers into core API before we can claim an alternative solution to be deprecated. For now, given they still serve a majority of use cases, they need to live.
But I do agree with the confusion, I think we can make this happen soon.

Hi , I have read the newest document at https://www.tensorflow.org/tutorials/structured_data/preprocessing_layers . It's very useful , but I have another question :
How to serve the model in that document by tf-serving ? Is there a tf-serving document ready for this new feature ?
I add an issue at https://github.com/tensorflow/tensorflow/issues/44002  and you can answer it there .

KPL can be part of your model:
class MyModel(tf.keras.Model):
  def __init__(self):
    self.preprocess_layer = ...
    ...
  def call(self, inputs):
    preprocessed_inputs = self.preprocess_layer(inputs)
    output = ...
    return output

Hi , there are two issues about KPL :
https://github.com/tensorflow/tensorflow/issues/44005
https://github.com/tensorflow/tensorboard/issues/4530
Do you know anyone who may solve them ?  Could you please @  them in the issues ?


keyword search

Static website searching is expensive, in favor of keeping the website lite for mobile usage, I'm not including this.


CDN for Bitbucket / Gitlab doesn't support partial request (Range header)

I'm trying to stream audio files, which requires the request to support partial response by specifying the "Range: " header.
Partial request should already be supported by Cloudflare, and it works for Github (rawcdn.githack.com, gistcdn.githack.com). However I found that both Bitbucket (bbcdn.githack.com) and Gitlab (glcdn.githack.com) does not support partial response.
In this case Content-Type doesn't matter, it still works if the content-type is application/octet-stream instead of audio/x-m4a, as long as partial request is supported.
Is that an intentional behaviour or configuration problem?
Test for Github CDN, which returns 206 Partial Content:
curl --header 'Range: bytes=0-10' -v -o /dev/null 'https://gistcdn.githack.com/arcnavier/725bc575222fe8e9931618c2d5b6580f/raw/Lovely%20Day%20-%20Most%20Beautiful%20Anime%20Music%20%281%20Hour%20BGM%29.m4a'

  Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to gistcdn.githack.com (<IP_ADDRESS>) port 443 (#0)
...
< HTTP/2 206
< date: Thu, 19 Apr 2018 13:17:43 GMT
< content-type: application/octet-stream
< content-length: 11
...
< cf-cache-status: HIT
< content-range: bytes 0-10/57847658
< server: cloudflare
<
{ [11 bytes data]

Test for Gitlab CDN, returns 200 with the entire content
curl --header 'Range: bytes=0-10' -v -o /dev/null 'https://glcdn.githack.com/maomihz/bigmusic/raw/master/2 Hour Beautiful Piano Music - Romantic Love Song 【BGM】.m4a'

  Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to glcdn.githack.com (<IP_ADDRESS>) port 443 (#0)
...
< HTTP/2 200
< date: Thu, 19 Apr 2018 13:17:23 GMT
< content-type: application/octet-stream
< content-length: 114571793
...
< cf-cache-status: HIT
< server: cloudflare
< cf-ray: 40dfa6afbec5586d-DFW
<
{ [783 bytes data]

Bitbucket, returns 200
curl --header 'Range: bytes=0-10' -v -o /dev/null 'https://bbcdn.githack.com/pypy/pypy/raw/51718fe91a11a641de8dfd2e54a7df4876ceca20/get_externals.py'

  Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to bbcdn.githack.com (<IP_ADDRESS>) port 443 (#0)

< HTTP/2 200
< date: Thu, 19 Apr 2018 13:33:45 GMT
< content-type: text/plain

Thank you for your effort!

Yeah, caching media is not the thing I would like to have on my server. But anyway, if you find a way how to do that, I'll consider merging.

Should be fixed by #25


NH-67051 Add report_init_event tests

Adds report_init_event tests!! Tricky mocks for this one.
Tests pass with c-lib 14 (5d7dcee)  before revert

One future possibility is to have an explicit check of the "extension" (whether full oboe with reporter, or minimal Lambda api) init/loading status rather than as part of sending Init event.

Thank you @cheempz ! New ticket here: https://swicloud.atlassian.net/browse/NH-68251


Create an overload ParserUtil#parseTags(String...) method to replace SampleDataUtil#getTagSet(String...)

The method getTagSet should not be in SampleDataUtil since it does not provide any sample data and also what it is doing is similar to the method ParserUtil.parseTags(Collection<String> tags).
Create an overload parseTags(String...) method in ParserUtil to replace the getTagSet.

@eugenepeh

The method getTagSet should not be in SampleDataUtil since it does not provide any sample data and also what it is doing is similar to the method ParserUtil.parseTags(Collection tags).

SampleDataUtil is in the model package, and thus should not be referring to the logic package, though?

https://github.com/se-edu/addressbook-level4/blob/a3c88b17d2001ee5e5c1e2e13a8c1d28d8b85f92/src/test/java/seedu/address/testutil/PersonBuilder.java#L63-L70
hmm that's true but the following implement in Person builder looks weird because we're calling getTagSet in SampleDataUtil yet it doesn't provides SampleData.

SampleDataUtil is in the model package, and thus should not be referring to the logic package, though?

In that case, would renaming the method to something more relevant be more appropriate?

A related question is should SampleDataUtil be in the Model? Looks like it's not being used by other classes in the Model.

I think its fine for SampleDataUtil  to be in model since it provides initial data for model in MainApp
https://github.com/se-edu/addressbook-level4/blob/a3c88b17d2001ee5e5c1e2e13a8c1d28d8b85f92/src/main/java/seedu/address/MainApp.java#L88-L106
Only the method getTagSet, however, is being used in the PersonBuilder in a non-SampleData manner.

I think its fine for SampleDataUtil to be in model since it provides initial data for model in MainApp

If we can also think of SampleDataUtil as a client of the Model (i.e., it instantiates a model and populates it with some data), it is OK for it to be outside of Model too.

@eugenepeh

hmm that's true but the above implement in Person Builder looks weird because we're calling getTagSet in SampleDataUtil yet it doesn't provides SampleData.

Oh, didn't notice that getTagSet is public :-/ I thought it was just a private helper method (which would have then been totally fine).
imo, there's no need to aggressively remove redundancy in the code base like what PersonBuilder did at the expense of causing strange dependencies between architecture layers or classes. Functions like getTagSet are extremely trivial and are just fine being private helper methods.
Maybe if it was rewritten into a Streams one-liner people would be less tempted to do this :-P Our "require braces" code style makes some code seem more verbose/complex than they are.

In that case, would renaming the method to something more relevant be more appropriate?

What do you have in mind?

@damithc Where do you think SampleDataUtil should be moved to? logic, ui, MainApp?

@pyokagan

What do you have in mind?

I think renaming it to maybe "createTagSet" (to make it sound more relevant) then change it to a private method, disallowing other classes such as PersonBuilder to utilize it.
Then for the usage in PersonBuilder, I was also wondering, why not just allow have an overload method for setTags that takes String array input?

@damithc Where do you think SampleDataUtil should be moved to? logic, ui, MainApp?

MainApp? While it is OK for MainApp to depend on Logic to parse the tags string, It's still not nice for MainApp to reach deep into Logic. This situation requires MainApp to reach inside Logic because parse methods are not exposed in the Logic API.

Oh, didn't notice that getTagSet is public :-/ I thought it was just a private helper method (which would have then been totally fine).

Looks like it can be private. In fact only one method from that class is used from outside. Perhaps this is the easiest fix?

Then for the usage in PersonBuilder, I was also wondering, why not just allow have an overload method for setTags that takes String array input?

PersonBuilder is a test util. On the one hand it's not nice to add production code to serve test code. On the other hand, it's nice to put code where they belong.
Options:

Move SampleDataUtil to MainApp and move the parseTags(String...) to Logic. Implications: The dependency is in an acceptable direction but it goes too deep.
Make 2 private parseTags(String...) methods in the two call sites. Implications: No odd dependencies, but there is code duplication.


add stat_prefix support for virtual service api

Please provide a description of this PR:
adds support for https://github.com/istio/api/pull/2405

Hi @varks. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/ok-to-test

@varks: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
64a760fe4484ea38f5987bb125e3d38dba573d74
link
true
/test release-notes

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@varks: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
64a760fe4484ea38f5987bb125e3d38dba573d74
link
true
/test release-notes

integ-pilot-multicluster_istio
64a760fe4484ea38f5987bb125e3d38dba573d74
link
true
/test integ-pilot-multicluster

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test release-notes

/test integ-pilot-multicluster

LGTM, with minor test request.

@varks: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

unit-tests-arm64_istio
20c0baaf905ba766eef5b054cb938c2d2bfa9eb0
link
true
/test unit-tests-arm64

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


[DO NOT MERGE] Add AB test for electric cars

This test compares click through rates of people visiting the two done pages:

/done/vehicle-tax
/done/check-vehicle-tax

We append the variant name on to the end of the URL that leads to the GoUltralow website so that we they can do some measurement based on what users have seen here.
Measurement on GOV.UK only occurs if users have opted in to cookies (I can't speak for what happens on the destination website). People who haven't opted in will get the control variant which does not add a tracking query parameter to the link.
The config in en.yml will be changing when we get the final list from DfT.
https://trello.com/c/In6XcrQU/1399-set-up-dft-multi-variant-a-b-test-test-dev-work
Also hooks in with https://trello.com/c/EpuznchJ/1398-dft-for-the-multivariate-test-analytics-tasks

Thanks for the review @bilbof!


onPinch ViroVideo not working on IOS

I use ViroVideo inside ViroARImageMarker and then can not zoom in/out with onPinch method on IOS. Pls help me!

closing due to not following issue template.


What's the maximum size of EMQ publish payload?

Environment

OS: Centos
Erlang/OTP: 20
EMQ: 2.3.11

Description
What's the maximum size of EMQ publish payload?

It depends your hardware performance.

Sorry , I misunderstood your issue just now.  The Maximum size of mqtt packet could be configured in emq.conf which also means that the maximum size of EMQ  publish payload could also be configured in emq.conf

Hi Gilbert,
So I can change mqtt.max_packet_size to any value? Please check this article:
https://blog.csdn.net/chancein007/article/details/77996544
It said mqtt max payload size should < 268435455 bytes (about 256M).
BR
Jack Zhang

No, you can't , the article you referred is right. You can configure max packet size in emqttd but the size shoud be less than 268 435 455 bytes.

Hi Gilbert,
OK, thanks!
BR
Jack Zhang


Timeout Controller

By design, Playbook has a lifecycle, from creation, initialization, compilation build, deployment, running to destruction.
Currently, we need to implement a new controller to perform periodic resource recovery (destruction) of Playbook. The following is a technical description:

The administrator can set a resource recovery duration, such as 2 weeks (default);
this duration is annotated on each Playbook resource definition;
Timeout Controller periodically scans for Playbooks that are about to expire and performs a series of actions, such as sending reminder emails or notifications (e.g. via Slack) 3 days or 2 days before the expiration date, and performs Playbook deletion on the same day;
Additional consideration is given to support Webhook notification of events in the future;

Reference:

Rust Kubernetes client and controller runtime https://github.com/kube-rs/kube
Kube controller-rs example: https://github.com/kube-rs/controller-rs
Amphitheatre Controllers: https://github.com/amphitheatre-app/amphitheatre/tree/master/controllers
k8s-ttl-controller https://github.com/fpetkovski/k8s-ttl-controller

Note: Contributors are requested to provide technical design solutions and implementation plans in the comments, as we need to communicate more fully.

By design, Playbook has a lifecycle, from creation, initialization, compilation build, deployment, running to destruction.
Currently, we need to implement a new controller to perform periodic resource recovery (destruction) of Playbook. The following is a technical description:

The administrator can set a resource recovery duration, such as 2 weeks (default);
this duration is annotated on each Playbook resource definition;
Timeout Controller periodically scans for Playbooks that are about to expire and performs a series of actions, such as sending reminder emails or notifications (e.g. via Slack) 3 days or 2 days before the expiration date, and performs Playbook deletion on the same day;
Additional consideration is given to support Webhook notification of events in the future;

Reference:

Rust Kubernetes client and controller runtime https://github.com/kube-rs/kube
Kube controller-rs example: https://github.com/kube-rs/controller-rs
Amphitheatre Controllers: https://github.com/amphitheatre-app/amphitheatre/tree/master/controllers
k8s-ttl-controller https://github.com/fpetkovski/k8s-ttl-controller

Note: Contributors are requested to provide technical design solutions and implementation plans in the comments, as we need to communicate more fully.

Additional information:
The Kubernetes platform itself has a ttl-controller that is only used for Job resource recovery, the code should be in the K8s repository and can also be viewed as a reference

#235 This issue has been done.

#235 This issue has been done.

Cool，thanks very much！


[ClockPicker] Fix typos

Hi. Thanks for opening the PR. Can you please run yarn proptypes and yarn docs:api?

I was excited with the web way of submitting changes and didn't see this coming. :)
As soon as I get some time I will actually clone the repo on my system and do as you said.

@rejetto Thanks!


Install command should also activate extension preloading.

This is required to properly enforce restrictions.

I looked into a couple alternatives here, with various pros and cons, but couldn't find one that was satisfactory.
Best approach seems to be to add the extension to the per-session preload for the current database only, during create extension. But there is no method called when the extension is dropped, so we don't have a chance to do any cleanup (I have looked into event triggers, they are not a good fit). The setting will be left lingering, potentially creating problems later. If the cleanup has to be explicit, I think it is better to keep the setup explicit also.