Install the App Service extension Step Fails

Getting the following error on creating the App Service Extension:
(ExtensionCreationFailed)  error: Unable to get the status from the local CRD with the error : {Error : Retry for given duration didn't get any results with err {status not populated}}
Code: ExtensionCreationFailed
Message:  error: Unable to get the status from the local CRD with the error : {Error : Retry for given duration didn't get any results with err {status not populated}}

Attaching image for the same.

Got the same issue when I followed the steps for AKS Cluster. Then I tried using a kind cluster on my laptop just to double check but the issue persists.
Kindly assist !! Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 492acd2f-5d4e-4be2-b1a2-8b2c83a5a2fc
Version Independent ID: 8e44670d-d75e-1185-b3c3-b46604713a42
Content: Create and manage custom locations on Azure Arc-enabled Kubernetes - Azure Arc
Content Source: articles/azure-arc/kubernetes/custom-locations.md
Service: azure-arc
Sub-service: azure-arc-kubernetes
GitHub Login: @shashankbarsin
Microsoft Alias: shasb

Thank you for your feedback! We will review and update as appropriate.

@raisaurabh777 if I understand the issue correctly you have attempted to install the Application services extension on two or three different clusters - AKS, KIND and potentially another?
We were investigating an issue with our partners in Arc K8s Extensions and identified an issue during extension provisioning, which can cause a false negative to be reported indicating the extension deployment failed but then actually completes after approximately 7-10 minutes.  Can you run az k8s-extension show and confirm the provisioningState please?  The command is of the form:
az k8s-extension show -n $extensionName -g $groupName -c $clusterName -t connectedClusters
We are working with our partners to fix the issue.
Regarding your KIND cluster, the Application services extension requires a) that the Kubernetes Cluster supports a loadBalancer service type and b) that there is some attached storage available.  To support the loadBalancer service type in KIND you would need to install and configure MetalLB - here's some documentation regarding this - https://kind.sigs.k8s.io/docs/user/loadbalancer/ . To confirm the available storage classes you have configured in KIND you need to run kubectl get storageclass and this will list applicable storage classes to specify.  If you have no configured storage classes you will need to attach a storage class to your cluster and specify the name in the buildService.storageClassName property in the script.

I've also gotten the same issue when I followed the steps for EKS Cluster as well

@youngjoven can you also share the provisioningState from az k8s-extension show -n $extensionName -g $groupName -c $clusterName -t connectedClusters please?
I suspect your extension ultimately failed to deploy as EKS clusters do not have a storage class attached to them named default.  In my test eks cluster, I have gp2 available.  Please run kubectl get storageclass against your eks cluster and then deploy the extension with a valid storage class for the property - buildService.storageClassName
Also note - the property envoy.annotations.service.bet.kubernetes.io/azure-load-balancer-resource-group is only required when the connected cluster is an AKS cluster - https://docs.microsoft.com/azure/app-service/manage-create-arc-environment?tabs=bash#install-the-app-service-extension

It seems that the error is solved after I fixed the property from "buildService.storageClassName" and removed the property of "envoy.annotations.service.bet.kubernetes.io/azure-load-balancer-resource-group"
But I got a new error as this screenshot showed,

I suspected that the Elastic Load Balancer for the envoy had a problem in EXTERNAL-IP

Here is "az k8s-extension show -n $extensionName -g $groupName -c $clusterName -t connectedClusters"
{
"aksAssignedIdentity": null,
"autoUpgradeMinorVersion": true,
"configurationProtectedSettings": {
"logProcessor.appLogs.logAnalyticsConfig.customerId": "",
"logProcessor.appLogs.logAnalyticsConfig.sharedKey": ""
},
"configurationSettings": {
"Microsoft.CustomLocation.ServiceAccount": "youngserviceaccount",
"appsNamespace": "appservice-ns",
"buildService.storageAccessMode": "ReadWriteOnce",
"buildService.storageClassName": "youngstorageclasses",
"clusterName": "youngKubeEnvironmentspecial",
"customConfigMap": "appservice-ns/kube-environment-config",
"keda.enabled": "true",
"logProcessor.appLogs.destination": "log-analytics"
},
"customLocationSettings": null,
"errorInfo": null,
"extensionType": "microsoft.web.appservice",
"id": "/subscriptions/--omit--/resourceGroups/young-arc/providers/Microsoft.Kubernetes/connectedClusters/young-eks-arc/providers/Microsoft.KubernetesConfiguration/extensions/appservice-ext",
"identity": {
"principalId": "--omit--",
"tenantId": null,
"type": "SystemAssigned"
},
"location": null,
"name": "appservice-ext",
"packageUri": null,
"provisioningState": "Failed",
"releaseTrain": "stable",
"resourceGroup": "young-arc",
"scope": {
"cluster": {
"releaseNamespace": "appservice-ns"
},
"namespace": null
},
"statuses": [
{
"code": "InstallationFailed",
"displayStatus": null,
"level": null,
"message": "Error: {failed to install chart from path [] for release [appservice-ext]: err [release appservice-ext failed, and has been uninstalled due to atomic being set: timed out waiting for the condition]} occurred while doing the operation : {Installing the extension} on the config",
"time": null
}
],
"systemData": {
"createdAt": "2022-02-23T02:07:19.991198+00:00",
"createdBy": null,
"createdByType": null,
"lastModifiedAt": "2022-02-23T14:28:34.478628+00:00",
"lastModifiedBy": null,
"lastModifiedByType": null
},
"type": "Microsoft.KubernetesConfiguration/extensions",
"version": "0.12.0"
}

@youngjoven can you share the output of the following two commands please?
kubectl get storageclass
and
kubectl get pods -n appservice-ns

@youngjoven - Can you please provide us with the above details  like @apwestgarth has requested?

Sorry for late response,, BUT!@ the issue is finally solved although the Docs made me confused.
https://docs.microsoft.com/en-us/azure/app-service/manage-create-arc-environment?tabs=bash#install-the-app-service-extension
I created a new serviceaccount,"admin-user", because "default" serviceaccount didn't work for my work every single time.
and then, I provided a new clusterrolebinding to the serviceaccount.
Therefore, I just fixed name of ServiceAccount as "admin-user" and name of storageClassName as "appservice-ext-k8se-build-service". It was my first time to deploy appservice-extension. Thank you all!

The reason why I created the App Service Extension is to create a custom location for the next step.
https://docs.microsoft.com/en-us/azure/app-service/manage-create-arc-environment?tabs=bash#create-a-custom-location
However, when I declared to type the customlocation for creating, the App Service Extension was showing "failed".

But, the problem is the next thing to create "customLocation" part. It seems that the customLocation is failed because this procedure is not for EKS. I know that "this tutorial uses Azure Kubernetes Service to provide concrete instructions for setting up an environment from scratch". But there is no choice to create customLocation for EKS in tutorial. I am just wondering how I could introduce multi-cloud services to customers using Azure Arc? :(
https://docs.microsoft.com/en-us/azure/app-service/manage-create-arc-environment?tabs=bash#create-a-connected-cluster

@apwestgarth - Thanks for providing your insights.
@raisaurabh777 & @youngjoven - Thanks again for bringing this to our attention. We will take this note and add it in the backlog to update the doc.  We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

I had exactly the same kind of journey (and exactly the same errors and troubles with default clusterrole, the storageclass, etc. as above) as @youngjoven . Could you please extend/update the documentation accordingly? Thank you!


Fix README translation to Brazilian Portuguese

This PR fixes the Brazilian Portuguese translation in the README.md file. Several sections were updated to ensure the translation is accurate and easy to understand for Portuguese-speaking users. The changes include:

Corrected translation of sections related to contributing, bug fixes and improvements.
Added troubleshooting section.
Fixed minor language inconsistencies and clarified some technical terms to improve comprehension.

Thanks


💅🚧 [WIP] Docusaurus Theming

Motivation
I found the current style configuration lacking just a bit of love 💅❤
At the moment it is super easy to bootstrap Docusaurus.
Less straightforward/clean is to do deeper customizations. Eg.: to change a 9px padding value hardcoded in main.css we need to override that value.
It would be nice to give users much more control over the styling of Docusaurus, keeping the 'zero configuration' paradigm intact.
Test Plan
Add/Fix tests TBD
Related PRs
https://github.com/facebook/Docusaurus/pull/389
Plan
We start by implementing processing with postcss and precss

PreCSS combines Sass-like syntactical sugar — like variables, conditionals, and iterators — with emerging CSS features — like color functions, logical and custom properties, media query ranges, and image sets.

[x] Move styles route logic in separate file
[x] Add postcss + precss + autoprefixer as a basic configuration
[x] Move color variables (or font variables, or any variable for the css) in a stylesConfig.pcss file
[ ] Split the current main.css file in modules
[ ] Expose more css code to the users
[ ] Many other postcss optimizations (rg. cssnano to compress output)
[ ] Manage magic css values (margin or padding px values etc) in the stylesConfig file
[ ] Give more control over the react component (Default component + extends? )
[ ] Add more to options to configure the react components, eg. transitions, block visibility, block order

What do you think?
Extra: the Docusaurus default mascotte
What do you think about this illustration I put together?
Docusaurus

Docusaurus icon

They would replace:

@albinotonnina Hi. This is a big undertaking :) Adding multiple, out-of-the-box themes is a big feature that we wanted to add as part of a major upgrade to Docusaurus. I would be very happy and grateful if you wanted to help in this quest.
A couple of things:

What you have now - is it a breaking change? Particularly, removing color {...} from siteConfig.js? Would that break all our existing sites?
At the end of this, would your goal be to have a choice of a couple of ready-to-go themes for Docusaurus?
Re: your cute logo. I like it :) However, we cannot get rid of our original dinosaur -- we named Frank. That was the first logo we created in early development. That said, I imagine we can get your logo integrated somewhere given how cool it looks. Maybe they can co-exist together.

Hello @JoelMarcey
I'd be happy to help, sure!
About your questions:

Will it break with the current sites? At the moment yes, kind of. User with colors (or fonts) settings in their site config will get a warning, suggesting them to move those colors in this new .pcss configuration file which will be used by precss, a postcss plugin with SASS like features, such as variables, conditionals and iterators. This means that then the possibilities to configure Docusaurus are much much more.
It won't break though, it will fallback to the default .pcss file, basically setting the default values (just colors at the moment)
My goal with this. I'm not sure my goal is to have just some ready themes to apply on Docusaurus. It could definitely be like this at the beginning but I see in the future the chance to go much deeper with customization. Like you already do for the Footer component in the UI lib; maybe the user could be able to override/extends the other components as well, such as the sidebar, or even better the layout of the main documentation pages, a bit like we can do already with the homepage. I think Docusaurus could be an engine to manipulate the data (and translations, and versioning and publishing etc etc) with on top a styling/markup/react layer, more isolated than now, overridable, customizable. Themes for example could be in different repositories, that would be easy for the user to install, switch to or create their own. I imagine some userland themes maybe?
I didn't know about Frank so now I'll back-down immediately :) Although I wouldn't use him like it is used now: when you launch Docusaurus for the first time you see him everywhere in the homepage, in all sizes. It hurts a bit my eyes, with all the due respect. I don't dislike the idea of Frank being designed so raw, a bit childish if you want, but I would show him just once, as the main logo and rewrite a bit the other sections, to be more 'template like', meaning more descriptive of what you can do with Docusaurus. I'm glad you like Larry (I just named the other I proposed 😅) and if you see some use for him I would be definitely glad.

I hope you like what I have in mind and I'll be waiting for your opinions on this.
PS. My first real world project with Docusaurus is a rewrite of the new blazing fast 🚀 web application bundler Parcel 📦.
This is what looks like right now with Docusaurus: http://parcel-website.now.sh/
Goal was to have it looks pretty much like the original one.
I think we will make a decision this with the creator of Parcel next week.
There is an issue with crowdin, docusaurus and icons I need to have a look at (the quotes on the sidebar of the documentation pages should not be there) but nothing that cannot be solved.

The way to probably do it with the custom colours is to create a default sass variable file using them, which can be supplied to the compiler and rolled into the output CSS.
I'm really not up to speed on PreCSS and PostCSS, but there have been very few times that Sass has been unable to do what I need.
I'm not a big fan of Frank either, but ultimately a replacement would have to fit what exists already, I would have to say it doesn't really match the branding imagery, which is pastel colored rounded rectangles. Whilst this isn't exactly a fancy SaaS startup, I still feel that consistent imagery is important.

I'm here people. I'm gonna continue this at some point 😃

@albinotonnina I added cssnano in #569. It's a bit of a hack now so feel free to move it to somewhere more appropriate.
Also, it would be better to break up the PR into smaller ones so that progress can be tracked in smaller milestones (also easier to review).

@albinotonnina hello, I would love to lend a hand. I can split main.css into a module that follows a methodology like ITSCSS and or BEM ... etc

@yangshun @ahmadalfy Should this PR be closed in favor of smaller bite-sized ones? Or will we just use this PR as a starting point for some changes?

I don't think the current code base is ready for theming. There's quite a bit of cleaning up to do in terms of the assets pipeline and CSS structure. I'm afraid we'll have to hold off till then. Let's revisit this in future. Apologies!


Quest Goals Bug with GeyserMC players

What is your Quests version / build number (do not say "latest")?
Quests 4.5.3-b359
Describe the problem and how to replicate it.
When using "Invalid Names" for players the Quest goals and goal progress isn't shown to players as the player's name isn't quoted in the command.
The fix should be easy by putting the player's name in quotes.
Any console errors to share?
[13:04:16] [Server thread/INFO]: Invalid name or UUID
[13:04:16] [Server thread/INFO]: tellraw *BeaDieErste ["(Erledigt) Abbauen ",{"translate":"block.minecraft.nether_quartz_ore", "color":"green"},": 100/100"]<--[HERE]
[13:04:16] [Server thread/INFO]: Invalid name or UUID
[13:04:16] [Server thread/INFO]: tellraw *BeaDieErste ["- Abbauen ",{"translate":"block.minecraft.netherrack", "color":"green"},": 0/500"]<--[HERE]
[13:04:16] [Server thread/INFO]: Invalid name or UUID
[13:04:16] [Server thread/INFO]: tellraw *BeaDieErste ["- Abbauen ",{"translate":"block.minecraft.soul_sand", "color":"green"},": 0/100"]<--[HERE]
[13:04:16] [Server thread/INFO]: Invalid name or UUID
[13:04:16] [Server thread/INFO]: tellraw *BeaDieErste ["- Liefere ",{"translate":"block.minecraft.soul_sand", "color":"green"}," zu Prof. Knox: 0/32"]<--[HERE]
[13:04:16] [Server thread/INFO]: Invalid name or UUID
[13:04:16] [Server thread/INFO]: tellraw *BeaDieErste ["- Killen ",{"translate":"entity.minecraft.skeleton", "color":"aqua"},": 0/3"]<--[HERE]
[13:04:16] [Server thread/INFO]: Invalid name or UUID
[13:04:16] [Server thread/INFO]: tellraw *BeaDieErste ["- Killen ",{"translate":"entity.minecraft.piglin", "color":"aqua"},": 0/1"]<--[HERE]
[13:04:16] [Server thread/INFO]: Invalid name or UUID
[13:04:16] [Server thread/INFO]: tellraw *BeaDieErste ["- Killen ",{"translate":"entity.minecraft.magma_cube", "color":"aqua"},": 0/5"]<--[HERE]
[13:10:19] [Server thread/INFO]: Invalid name or UUID
[13:10:19] [Server thread/INFO]: tellraw *BeaDieErste ["(Erledigt) Abbauen ",{"translate":"block.minecraft.netherrack", "color":"green"},": 500/500"]<--[HERE]
[13:14:42] [Server thread/INFO]: Invalid name or UUID
[13:14:42] [Server thread/INFO]: tellraw *BeaDieErste ["(Erledigt) Abbauen ",{"translate":"block.minecraft.soul_sand", "color":"green"},": 100/100"]<--[HERE]
[13:29:49] [Server thread/INFO]: Invalid name or UUID
[13:29:49] [Server thread/INFO]: tellraw *BeaDieErste ["(Erledigt) Abbauen ",{"translate":"block.minecraft.netherrack", "color":"green"},": 500/500"]<--[HERE]
[13:31:37] [Server thread/INFO]: Quests 4.5.3-b359

If applicable, can you provide an example quest from quests.yml?
No response

This sounds like a LocaleLib issue, but regardless I'm not able to replicate it. Granted, the Bedrock account I used showed as starting with a . (not *) in the console. Could you provide more insight on how to reproduce this?

To reproduce it you need to change the prefix from a . to a * in the floodgate config.
Note:
I changed the . to an * because the dot created problems with another plugin.
So I think the error is caused because of the * as Minecraft cannot recognize it as part of the player's name if you don't put it in quotes.
What I think is weird is that it is only the goals part of the message and everything else is sent without any problems.
Also I didn't check again if it might be fixed now. I can if you cannot reproduce it. (And also probably send a full log too then.)


a error when I use DeepSpeed with accelerate

System Info
my accelerate config:
compute_environment: LOCAL_MACHINE
deepspeed_config:
  gradient_accumulation_steps: 8
  gradient_clipping: 1.0
  offload_optimizer_device: none
  offload_param_device: none
  zero3_init_flag: false
  zero_stage: 2
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: 'no'
num_machines: 1
num_processes: 3
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

#---------------------------------------------------------------------------------------
error as follows:
Traceback (most recent call last):
  File "/users_2/d00477216/3_train_gpus/main.py", line 38, in <module>
    model, optimizer, train_dataloader, scheduler = accelerator.prepare(model, optimizer, train_dataloader, scheduler)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/accelerate/accelerator.py", line 1118, in prepare
    result = self._prepare_deepspeed(*args)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/accelerate/accelerator.py", line 1415, in _prepare_deepspeed
    engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/deepspeed/__init__.py", line 125, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 340, in __init__
    self._configure_optimizer(optimizer, model_parameters)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1298, in _configure_optimizer
    self.optimizer = self._configure_zero_optimizer(basic_optimizer)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1547, in _configure_zero_optimizer
    optimizer = DeepSpeedZeroOptimizer(
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 324, in __init__
    self.flatten_dense_tensors_aligned(
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 867, in flatten_dense_tensors_aligned
    return self.flatten(align_dense_tensors(tensor_list, alignment))
RuntimeError: torch.cat(): expected a non-empty list of Tensors
Traceback (most recent call last):
  File "/users_2/d00477216/3_train_gpus/main.py", line 38, in <module>
    model, optimizer, train_dataloader, scheduler = accelerator.prepare(model, optimizer, train_dataloader, scheduler)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/accelerate/accelerator.py", line 1118, in prepare
    result = self._prepare_deepspeed(*args)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/accelerate/accelerator.py", line 1415, in _prepare_deepspeed
    engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/deepspeed/__init__.py", line 125, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 340, in __init__
    self._configure_optimizer(optimizer, model_parameters)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1298, in _configure_optimizer
    self.optimizer = self._configure_zero_optimizer(basic_optimizer)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1547, in _configure_zero_optimizer
    optimizer = DeepSpeedZeroOptimizer(
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 324, in __init__
    self.flatten_dense_tensors_aligned(
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 867, in flatten_dense_tensors_aligned
    return self.flatten(align_dense_tensors(tensor_list, alignment))
RuntimeError: torch.cat(): expected a non-empty list of Tensors
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 27320) of binary: /user_lc/envs/lc_nlp39/bin/python
Traceback (most recent call last):
  File "/home/anaconda3/envs/lc_nlp39/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
    args.func(args)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/accelerate/commands/launch.py", line 908, in launch_command
    deepspeed_launcher(args)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/accelerate/commands/launch.py", line 647, in deepspeed_launcher
    distrib_run.run(args)
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/user_lc/envs/lc_nlp39/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
main.py FAILED

Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[X] My own task or dataset (give details below)

Reproduction
deepspeed_plugin = DeepSpeedPlugin(gradient_accumulation_steps=args.gradient_accumulation_steps)
accelerator = Accelerator(mixed_precision=args.mixed_precision, deepspeed_plugin=deepspeed_plugin)
device = accelerator.device

transformers.set_seed(args.seed)
# 加载模型
model_loader = args.MODEL_LOADER[args.base_model](args)
tokenizer, model = model_loader.load_model()

model.to(device)
# train
if args.do_train:
    # data handle
    train_dataset = args.PROCESSOR_CLASSES[args.task_name](args, 'train', tokenizer)
    train_dataloader = DataLoader(train_dataset, shuffle=True, batch_size=args.train_batch_size,
                                  collate_fn=train_dataset.collate_fn)
    dev_dataset = args.PROCESSOR_CLASSES[args.task_name](args, 'dev', tokenizer)
    dev_dataloader = DataLoader(dev_dataset, batch_size=args.dev_batch_size, collate_fn=dev_dataset.collate_fn)

    optimizer, scheduler = set_optimizer(args, model, train_dataset)
    accelerator.wait_for_everyone()
    print('-----------------------------------', len(train_dataloader))
    model, optimizer, train_dataloader, scheduler = accelerator.prepare(model, optimizer, train_dataloader, scheduler)

    # train
    train_gpus(args, model, train_dataloader, dev_dataloader, optimizer, scheduler, tokenizer, accelerator, device)

Expected behavior
I wonder how to fix it

Hello, can you print number of trainable parameters? Looking at the error, seems like there are no trainable params and hence pass non-empty list to optimizer error

thank you! I use the LoRA method to train my chatGLM models, below are the params:
trainable params: 1572864 || all params:<PHONE_NUMBER> || trainable%: 0.09123437254985815
trainable params: 1572864 || all params:<PHONE_NUMBER> || trainable%: 0.09123437254985815
trainable params: 1572864 || all params:<PHONE_NUMBER> || trainable%: 0.09123437254985815
trainable params: 1572864 || all params:<PHONE_NUMBER> || trainable%: 0.09123437254985815
trainable params: 1572864 || all params:<PHONE_NUMBER> || trainable%: 0.09123437254985815
trainable params: 1572864 || all params:<PHONE_NUMBER> || trainable%: 0.09123437254985815

@pacman100 thank you! I use the LoRA method to train my chatGLM models, below are the params:
trainable params: 1572864 || all params:<PHONE_NUMBER> || trainable%: 0.09123437254985815
trainable params: 1572864 || all params:<PHONE_NUMBER> || trainable%: 0.09123437254985815
trainable params: 1572864 || all params:<PHONE_NUMBER> || trainable%: 0.09123437254985815
trainable params: 1572864 || all params:<PHONE_NUMBER> || trainable%: 0.09123437254985815
trainable params: 1572864 || all params:<PHONE_NUMBER> || trainable%: 0.09123437254985815
trainable params: 1572864 || all params:<PHONE_NUMBER> || trainable%: 0.09123437254985815

LoRA config:
LORA_PARA = {
'r': 8,
'lora_alpha': 32,
'lora_dropout': 0.05,
'target_modules': ['query_key_value'],
'task_type': "CAUSAL_LM",
'inference_mode': False,
}

@xdnjust were you able to resolve the issue? I am also facing the same when using LoRA with any model. Turning off LoRA makes it work correctly.

@xdnjust I also encountered the same issue. Did you figure out how to resolve it? Or is there any update about this issue?

Same issue. 😂

Hello, can you please provide the entire script with the details on how the model is created before passing it to the accelerator.prepare as well as the versions of the torch, transformers, accelerate, peft and deepspeed and the command used for launching the script.

same issue

same issue, anyone have idea how to fix it? 😂

@EnterpriseBCMe please give us a full reproducer to your problem that we can run

Please check all the parameter in optimizer requires gradient. I meet the same issue and filter the parameter by requiring gradient before put them into optimizer seems work.


Fix calculation of # CPU cores (FreeBSD/DragonFly)

On both FreeBSD and DragonFly, sysctl hw.ncpu returns the number of
logical CPUs, NOT the number of physical CPU cores.

FreeBSD has sysctl kern.smp.cores.

DragonFly has sysctl's hw.cpu_topology_core_ids and
hw.cpu_topology_phys_ids. Both multiplied together results in the
number of cores.

If kern.smp.cores (FreeBSD) or hw.cpu_topology_* (DragonFly) is
not available, use hw.ncpu instead.

For example on FreeBSD 12.2p3 running a i7-8565U:
sysctl hw.ncpu # => 8
sysctl kern.smp.cores # => 4

For example on DragonFly 5.9-DEVELOPMENT with Xeon E5-1620:
sysctl hw.ncpu # => 8
sysctl hw.cpu_topology_phys_ids # => 1
sysctl hw.cpu_topology_core_ids # => 4
sysctl hw.cpu_topology_ht_ids   # => 2

I found this by running a HTTP benchmark on a FreeBSD system
(i7-8565U, 4 cores / 8 logical CPUs), where setting --ponymaxthreads=4
improved latency significantly over the default value "8" (hw.ncpu).
And I was wondering why I could set --ponymaxthreads to 8, even so the
documentation explicitly states that it cannot be set to anything
above the number of CPU cores.

Hi @mneumann,
The changelog - fixed label was added to this pull request; all PRs with a changelog label need to have release notes included as part of the PR. If you haven't added release notes already, please do.
Release notes are added by creating a uniquely named file in the .release-notes directory. We suggest you call the file 3707.md to match the number of this pull request.
The basic format of the release notes (using markdown) should be:
## Title

End user description of changes, why it's important,
problems it solves etc.

If a breaking change, make sure to include 1 or more
examples what code would look like prior to this change
and how to update it to work after this change.

Thanks.

@ponylang-main I added an entry in .release-notes. Thanks!

Thanks @mneumann. This will go out in the next ponyc release at the end of the month.


[staging] python38: remove obsolete find_library patch

Motivation for this change
fixes python38 on staging
was missed in: https://github.com/NixOS/nixpkgs/pull/107725
patching file Lib/ctypes/test/test_find.py
Reversed (or previously applied) patch detected!  Assume -R? [n]
Apply anyway? [n]
Skipping patch.
3 out of 3 hunks ignored -- saving rejects to file Lib/ctypes/test/test_find.py.rej

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build python38Packages.requests

@GrahamcOfBorg build python38Packages.requests

python38Packages.requests was built by ofborg. Seeing as all of python3Packages, include stdenv is broken on staging, I'm going to merge

python38Packages.requests was built by ofborg. Seeing as all of python3Packages, include stdenv is broken on staging, I'm going to merge


support relative links with search_index_path

Using search_index_path I always got the absolute path back but for hosting on github a relative path is needed.
I wrote my own helper which uses link_to to get the relative path.
def link_to_search_index
  absolute_link_to_index = '/' + extensions[:search].options[:index_path]
  dom = Nokogiri.HTML(link_to '', absolute_link_to_index)
  node = dom.at_xpath '//a'
  node[:href]
end

Yeah - this is definitely an issue.
The real problem is, I think, that you can have the search box at different path levels (ie, at our site you can search from http://manas.com.ar/, http://manas.com.ar/staff, or http://manas.com.ar/staff/mgarcia). In those cases, if you want to use a relative URL you have to use them differently - or add any way to the page for declaring how many nesting levels to go back.
It's definitely doable, but I'm not that sure your helper works for all the cases. I'll give it a try - and start implementing that if works at least for a nice percentage of the cases. But I think there's more than that for this issue.
Thanks for the feedback! :)

@rtack we've just released middleman-search v0.8.0, which includes #9
That should solve your issue. It doesn't use relative paths, but the absolute paths it uses honor the :http_prefix Middleman setting.
Can you confirm us this works?


Current health, stamina, mana regen

I've added the ability to enter values into the configuration which enable the regeneration of the current stats. This was requested by a few users and I also find it useful so I thought I'd make a pull request.
I hope you don't mind, but I also added a lot of documentation and refactored some areas of the code.

Hey.
That's great. truth be told I've been busy with work and kinda fell off the game. Happy to have anyone continue improving this.
Cheers,
Z.


Prepare for CP4D 4.8 support

In CP4D 4.8 there will be a breaking change: iamIntegration will be defaulted to true for the CP4D control plane.
Make the following changes:

[x] In the sample config for CP4D 4.8, include the use_fs_iam: True property
[x] Support logging in to CP4D using FS IAM
[ ] Allow deploying a demo LDAP server on OpenShift and configure FS IAM for the demo LDAP server

LDAP will follow later.


Admin Router: Document repository structure

High Level Description
This PR documents repository structure and tries to clarify why certain things are structured this way or another.
Related Issues

https://jira.mesosphere.com/browse/DCOS_OSS-1013 Admin Router: Document the way Open/EE repos are structured in order to achieve deduplication

Related PRs
AR-NEXT PR: dcos#1611

Thank you @vespian. I have done a first pass and I will look at this again when my comments are addressed.
In particular, I like the detail and the clear mix of facts and the reasoning behind those facts. Some parts of this would have been very useful when I did the TLS 1.0 work.
That being said, I have not fact-checked the facts presented here. If you would like me to do so, please just ask.

Ugh, GitHub erroneously marked a comment as outdated. See https://github.com/mesosphere/dcos/pull/26#discussion_r120349631:

Ah sorry my question was ambiguous. I meant the other way around. Is it OK to disclose as much as there is here?

@vespian As it is so impactful on this PR I think that answering https://github.com/mesosphere/dcos/pull/26#issuecomment-306498329 is basically a blocker, but I will assume it is fine in a continuing review.


enclave-tls: Implement wolfssl TLS wrapper

Depends on #535

Implement the TLS wrapper  "wolfssl".

Please refer #695 for the detailed information.


Enables Multiple Assignees API

Speccing out the Multi-Assignee API as per https://developer.github.com/changes/2016-5-27-multiple-assignees/
@rauhryan , I don't have the test suite setup on my machine, may need a hand with it Thanks!
Currently this will not work without application/vnd.github.cerberus-preview+json set as the accept headers

LGTM


include spot-pricing #featureRequest

Pulling from the spot-pricing API so that it's possible to query different size instances current prices would be the bomb

See #109


Performance: investigate easy wins for the observation import?

The import script has never been thoroughly inspected for performance, because it was good enough for now.
Now, with 500.000+ observations in the LIFE RIPARIAS instance (which uses a cheap RDS database plan), the full import takes more than 3 hours. We didn't received complains so far because it's running nightly, but it might be good if there are some "easy" optimizations that could already improve this.

Update: I made some good improvements on the import_per_improvements branch (import time halved on my laptop).
The changes were more invasive than what I hoped, so it should be carefully tested before deployment.

Update: since that became more urgent due to #259, I implemented all those changes, import performances seems satisfactory again.


DBZ-770 Typename comparison should not be case dependent

https://issues.jboss.org/browse/DBZ-770

Interesting; I'm curious how this fix is related to the OP's exception which is about the value not being valid? In particular, should we have a more integrative test which reproduces the original exception?

Cool, thanks for the explanation. Applied.


Move header tests to new file

Coverage decreased (-3.7%) to 96.324% when pulling 9f2412ff19e2fdc14cb9e69057bfb34ab90e482e on move-header-tests into f8f4c96d3610660facb597c3d8be3633eeb0e818 on master.

Coverage remained the same at 100.0% when pulling f1087f01838c04bc8d9435f29ef6dde1d8575469 on move-header-tests into f8f4c96d3610660facb597c3d8be3633eeb0e818 on master.


Sign in to confirm you’re not a bot

Describe the bug
i am using @distube/ytdl-core for my project, but its returning error Sign in to confirm you’re not a bot. while i have added cookies but still same issue. but its working on my local machine.
Debug File

Environment

@distube/ytdl-core version: 4.14.4
Node.js version: v21.7.3
Operating system: Heroku/Render

Try this. (I fork @distube/ytdl-core to solve various problems. Also, usage is quite different from this one.)
https://github.com/ybd-project-ver1/ytdl-core


Suggest keys.openpgp.org as keyserver

See https://github.com/bitcoin/bitcoin/pull/23466

ack, thanks


feat: Postgres loader

Description
This PR is based on PR #470, please merge that first
Adds a postgres loader that allows to pass any query. The result is converted to csv and the loaded with the csv loader.
CREATE DATABASE mydatabase;
\c mydatabase;

CREATE TABLE employees (
  id SERIAL PRIMARY KEY,
  name VARCHAR(50),
  age INTEGER
);

INSERT INTO employees (name, age) VALUES ('Alice', 30);
INSERT INTO employees (name, age) VALUES ('Bob', 25);

turns into
[
    {
        "content": "id: 1, name: Alice, age: 30",
        "meta_data": {
            "url": "SELECT * FROM employees",
            "row": 1,
            "database": "mydatabase"
        }
    },
    {
        "content": "id: 2, name: Bob, age: 25",
        "meta_data": {
            "url": "SELECT * FROM employees",
            "row": 2,
            "database": "mydatabase"
        }
    }
]

As you can see, the metadata is adjusted for the data type.

This type is not auto detected. That's because I assume that we will add other databases, and at that point you couldn't tell from the string anyways. SELECT * FROM student is ambiguous.
This also requires setting your connection details in .env.

Both constraints are described in the documentation.
The dependency for this is included as an extra with the name postgres. We need to decide on a policy here, lazy loading makes the code a lot harder to maintain, and means extra work on the user. Including everything in community can't be the way either, because then we're right back to where we started. We need to find a better way for this.
Tests have not been added, because I can't think of a way to effectively test this without including a database.
Fixes # (issue)
Type of change
Please delete options that are not relevant.

[X] New feature (non-breaking change which adds functionality)

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Please delete options that are not relevant.

[ ] Unit Test
[X] Test Script (please provide)

I run the .sql script above to initialize postgres in a docker container.
from embedchain import OpenSourceApp as App
from embedchain.config import OpenSourceAppConfig as AppConfig
from embedchain.models.data_type import DataType

config = AppConfig(log_level="DEBUG")
app = App(config=config)
app.reset()
app.add("SELECT * FROM employees", DataType.POSTGRES)

Checklist:

[X] My code follows the style guidelines of this project
[X] I have performed a self-review of my own code
[X] I have commented my code, particularly in hard-to-understand areas
[X] I have made corresponding changes to the documentation
[X] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[X] New and existing unit tests pass locally with my changes
[X] Any dependent changes have been merged and published in downstream modules
[X] I have checked my code and corrected any misspellings

Maintainer Checklist

[ ] closes #xxxx (Replace xxxx with the GitHub issue number)
[ ] Made sure Checks passed

resolved merge conflicts

Closing this PR since we already added Postgres loader.


Bump version to 2.10.1

Coverage remained the same at 99.656% when pulling a72cd0808d2f62ca1d7cd59f4ba403fe0801b5f0 on Djailla:version into 047ac53ea469bc8eb33b43617878689d44a6a800 on KenKundert:master.

I have automation bumped the version myself.


Sharingservice connect attempt causes app to die on HoloLens

Updated - I reported the issue a few days ago. Having spent a day or two looking at it I think its the issue mentioned here https://forums.hololens.com/discussion/7203/vs2017-u5-6-crash-winrtscriptingobjectwrapper-internaladdref-reference-count-cannot-be-zero

Hi have a problem with the HoloToolKit for Unity. I am using the current version of the HoloToolKit (download yesterday)  running in a Unity 5.60f3, built and debugging using VS 2017.
When I add the ImportExportAnchorManager to a simple test app the app dies as it loads on HoloLens with no debug output. Tracing it through it appears to execute line 225 in ImportExportAnchorManager which in my build reads SharingStage.Instance.SharingManagerConnected += Connected; (see below for the full function)
and returns - it appears to die just after the return.
Sorry for the lack of precision here I don't seem to be able trace it any closer as the debugger is jumping and missing the exact line that is tripping it up.
private void Start()
{
// SharingStage should be valid at this point, but we may not be connected.
if (SharingStage.Instance.IsConnected)
{
Connected();
}
else
{
SharingStage.Instance.SharingManagerConnected += Connected;
}
}
The only debug I get after this line is in SharingStage.cs in HandleLog which is an Assert with the log message logString = "WinRTScriptingObjectWrapper::InternalAddRef Reference Count cannot be zero"
When I remove ImportAnchorManager   and just run the sharingstage and autojoin scripts I can see the Hololens connecting to the sharing service (running via WIFI on my local machine with no firewall) just fine.
The project is a brand new one with a single empty  parent game object and a single child with only AnchorText  for debuging.
The sharing service log records
SharingService: Created Session "Default" with ID 0 on port 20601
SharingService:  ** Logging Session Began at 22:25:36, 5-18-2017
SharingService: ***********************************
SharingService: ****** Sharing Service OnStart ******
SharingService: ***********************************
SharingService: Server Info:
Build Version: <IP_ADDRESS>
Schema Version: 17
SharingService: Listening for session list connections on port 20602 of all network devices of the local machine.
SharingService: Local IP addresses are:
SharingService: 	<IP_ADDRESS>
SharingService: 	<IP_ADDRESS>
SharingService: 	<IP_ADDRESS>
SharingService: Created Session "Default" with ID 0 on port 20601
SharingService: ListServer: Handshake failed with error 1
Any ideas gratefully received

This issue might need to be moved into the main HoloToolkit.

See https://github.com/Microsoft/HoloToolkit-Unity/issues/623

I cannot get the sharing services to work either, Hololens never finds anything.

ImportExportAnchorManager was replaced by AutoJoinSessionAndRoom


Upgrade Chart to apiVersion v2

Move CustomResourceDefinitions from conf folder to crds folder
Remove crds from templates
Update docs

Feature: HCPCFS-2729

Pull Request Test Coverage Report for Build 3804

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.008%) to 89.053%

Totals

Change from base Build 3802:
0.008%

Covered Lines:
9175

Relevant Lines:
9944

💛  - Coveralls


Provide feedback after signup

It should show some message after successful signup and change to the login form. Perhaps it could even login automatically after the signup if auth0 returns enough information/tokens.

#10  I added a message, notifying the user just before logging him in. Hope it's ok.

@cvetkovskin Thanks, looks good! For the next time, you can add Closes #9 to the PR body and it will close this issue automatically :)


Intermittent failure of EipTest.throttle

See failure stack trace below.

I'd want to work on this.

Seems I totally misread the CI build logs. The actual failure was in EipTest.throttle:
2024-01-25T10:09:30.7914371Z [ERROR] Tests run: 22, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 15.71 s <<< FAILURE! -- in org.apache.camel.quarkus.eip.it.EipTest
2024-01-25T10:09:30.7917689Z [ERROR] org.apache.camel.quarkus.eip.it.EipTest.throttle -- Time elapsed: 10.12 s <<< ERROR!
2024-01-25T10:09:30.7919831Z org.awaitility.core.ConditionTimeoutException: Condition with org.apache.camel.quarkus.eip.it.EipTest was not fulfilled within 10 seconds.
2024-01-25T10:09:30.7921922Z 	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
2024-01-25T10:09:30.7923262Z 	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
2024-01-25T10:09:30.7924468Z 	at org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
2024-01-25T10:09:30.7925667Z 	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
2024-01-25T10:09:30.7929861Z 	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
2024-01-25T10:09:30.7931228Z 	at org.apache.camel.quarkus.eip.it.EipTest.throttle(EipTest.java:488)
2024-01-25T10:09:30.7932375Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2024-01-25T10:09:30.7933750Z 	at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:1013)
2024-01-25T10:09:30.7935757Z 	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:827)
2024-01-25T10:09:30.7937637Z 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
2024-01-25T10:09:30.7938647Z 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

Seems the mock endpoint expectation is not fulfilled in time.

This failure just happened on the Quarkus Platform CI build too.


Created build-a-blog-generator-with-openai

Added two gifs

Renamed fille from build-a-blog-generator-with-openai to create-a-blog-generator-with-openai

I'm merging this in to get this tutorial live and later, we can edit it if there are more edits.


.NET 5 RC.1 - UITypeEditor Class

https://docs.microsoft.com/en-us/dotnet/api/system.drawing.design.uitypeeditor?view=net-5.0
The UITypeEditor class is not used for NET5. They say it's supported, but I don't agree with that.
I can't use it in a library that used NET5.
Examples:

It works fine for me in a .Net 5.0 library whose csproj looks like this:
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
    <UseWindowsForms>true</UseWindowsForms>
  </PropertyGroup>

</Project>

Can you share your csproj?

Bu csproj bu gibi görünüyor bir .Net 5.0 kütüphanede benim için iyi çalışıyor:
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
    <UseWindowsForms>true</UseWindowsForms>
  </PropertyGroup>

</Project>

Csproj'unuzu paylaşabilir misiniz?

<UseWindowsForms>true</UseWindowsForms> It was fixed when I added the code :)


Suggest to changeAccess specifier in PoseWithCovarianceDisplay class

Hi,
I'm trying to make a new type of Rviz Display plugin class which is derivied from rviz_default_plugins::displays::PoseWithCovarianceDisplay class.
But as we can see below line, I cannot override the function since it is private virtual function.
The function's name is processMessage().
https://github.com/ros2/rviz/blob/82538e5b4453b1b85f0b49a206d8b225f65b2424/rviz_default_plugins/include/rviz_default_plugins/displays/pose_covariance/pose_with_covariance_display.hpp#L103
As far as I know, most of this function in each dispaly plugin class are written as protected access specifier.
Could we modify to 'protected:'??

I requested PR https://github.com/ros2/rviz/pull/981

Looks like there is some inconsistency across various display plugins with regards to the scope of processMessage() so it might be good to standardize across all of them in the same effort.
In the meanwhile, another approach could be to write your own plugin that internally instantiates an object of the existing PoseWithCovarianceDisplay type. Then implement the new plugin methods such that they simply calls the respective methods of this object except for processMessage() which you implement with custom logic.

Looks like there is some inconsistency across various display plugins with regards to the scope of processMessage() so it might be good to standardize across all of them in the same effort.
In the meanwhile, another approach could be to write your own plugin that internally instantiates an object of the existing PoseWithCovarianceDisplay type. Then implement the new plugin methods such that they simply calls the respective methods of this object except for processMessage() which you implement with custom logic.

Thanks for you comment :)
I'm digging other way instead just for temporarily.

@Yadunund
Yadu, it looks there was some issue on jenkin though build result looks good(?).
https://build.ros2.org/job/Hpr__rviz__ubuntu_jammy_amd64/19/
Do you know why it failed? :(

According to log(https://build.ros2.org/job/Hpr__rviz__ubuntu_jammy_amd64/19/console), code style? and cpplint check may failed.


Case issues

Let's say we have a model like this on the server side:
class Person
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
}

A Asp.Net Core Controller serializes this to a JSON with 'firstName' and 'lastName'. So all my client-side classes use this convention.
I realized that SignalR serializes this to 'FirstName' and 'LastName'. This means I can't re-use my client-side classes.

Is there an easy way to configure this behaviour in SignalR?
If not, what's the recommended workaround?
It would be nice if the default behaviour of SignalR was the same as for Controllers.

The workaround described here does not seem to work.

I found a very ugly workaround. In my hub before returning any object I call the following function on it:
private object ToCamelCaseObject(object obj)
{
	var json = JsonConvert.SerializeObject(obj, Formatting.None, SerializerSettings);
	return JsonConvert.DeserializeObject(json);
}
private readonly static JsonSerializerSettings SerializerSettings = new JsonSerializerSettings() { ContractResolver = new Serialization.CamelCasePropertyNamesContractResolver() };

PS (not related to this issue but FYI): issue number 71 seems to be one of the biggest show-stoppers at the moment. Would be nice if arbitrary large objects can be transferred soon!

@rhegner What version aof SignalR are you using? The article you pointed to was based on the old SignalR repo you can find here - https://github.com/signalr/signalr
Still, could you provide something I can take a look at?
Thanks!

@rhegner - the problem is that at the moment we just new up JSonSerializer/JSonTextWriter without allowing passing any options (or maybe resolving these from the DI). We are discussing the best way to resolve #71.
Everything here is pretty much work in progress.

I worked around this issue by using my own IInvocationAdapter.  It is simply a copy of JsonNetInvocationAdapter (from the repo) but it uses constructor injection to resolve the JsonSerializer rather than newing one up.
public class JsonInvocationAdapter : IInvocationAdapter
{
        public JsonInvocationAdapter(JsonSerializer serializer)
        {
            _serializer = serializer ?? throw new ArgumentNullException(nameof(serializer));
        }

then registrating as follows..
var jsonSerializerSettings = new JsonSerializerSettings();
jsonSerializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver();
jsonSerializerSettings.Formatting = Formatting.Indented;
jsonSerializerSettings.Converters.Add(new StringEnumConverter());
services.AddSingleton(JsonSerializer.Create(jsonSerializerSettings));

services.AddSingleton<JsonInvocationAdapter>();
services.AddSignalR(options =>
{
     options.RegisterInvocationAdapter<JsonInvocationAdapter>("json");
});

Tracked by https://github.com/aspnet/SignalR/issues/261


lusearch calibration

re-calibrate lusearch's default input to match DaCapo's running guidelines for 'default' ('each benchmark executes for between 5 and 20 seconds on contemporary hardware and JVMs.')
determine whether it is amenable to a huge input (> GB heap)
if amenable, assemble a suitable huge input

We should use the wikipedia workload that is commonly used to benchmark tail latency.   Also see the lucerne nightly regressions, which includes wikipedia, among others.
We should also be facilitating reporting of query latencies.


[functions][state] Python state support

Motivation
Add state support in python functions
Changes

Bump bookkeeper version, so the table service has the changes to support python functions
Add state to python function

@srkukarni counters should be supported. but just realized that I missed that in bookkeeper python client, so going to add that in bk.
example will come with a test. when I am ready, I will remove "WIP" and that should be ready for review.

bk related changes is moved to #2906

run cpp tests


Fix compilation of the BLAS extension utests for NO_CBLAS=1

fixes #4684

I can confirm that the compilation completes without error when passing NO_CBLAS. Thank you.

Without passing NO_CBLAS, we are seeing unit tests failure in lapack-netlib/TESTING.
zdmd.out: :( .................ZGEDMD FAILED!Check the code for implementation errors

Same for CGEDME

That has to be unrelated - different testsuite. Can you tell when (and where) you started seeing them ?

Sorry, these failures we are seeing - ZGEDMD & CGEDMD in lapack-netlib/TESTING has been in a while.

@ChipKerchner not immediately reproducible on power8 (ppc64le/CentOS). also fairly certain that any PPC BLAS kernel changes since the introduction of the LAPACK DMD tests in early december came from you or your colleagues. there's also https://github.com/Reference-LAPACK/lapack/issues/960


Adding new API to get Public key by address

Describe the problem
Recently, we have indexed public keys by their addresses: #437
This helps reduce the size of blocks by removing duplicated public keys. Now, we can add a new API in gRPC to get the public key associated with a specific address.
Ideal Solution
We should add a new RPC function named GetPublicKey in blockchain.proto. It will accept an address as input and return the corresponding Public Key as output. If the public key is not indexed yet, it can return a NotFound error.

I will work on this

I will work on this

This issue depends on  #437. Please wait until that pull request is merged.

yes, now we can fix this issue.

Don't forget to add this API to grpc-gateway.config.yaml


[Tracker] Launcher program-related issues

This is the master tracking issue for program-related problems.
Third-party:

Issue #
Title
PR #

~#3425~
~PowerToys Run does not index Steam games~
fixed

#4200
environment path by default
fixed

#3792
PowerToys Run not showing indexed .exe
needs feedback

Unable to find exe but lnk is found

Issue #
Title
PR #

~#3223~
~doesn't find file explorer~
#3969 fixed

~#3599~
~Does not find Evernote~
#3969 fixed

~#3707~
~PowerToys Run doesn't find exe for putty~
#3969 fixed

Installing apps while PT is running:

Issue #
Title
PR #

#3553
PowerToys Run unable to find apps even after updating indexer

#2407
Decide how we update each plugins data

#2729
investigate monitoring shell:appsfolder for updates

Others:

Issue #
Title
PR #

~#3532~
~No camera in run~
#3842 fixed

~#3180~
~PowerToys run shows Notepad en Notepad++ twice~
#3969 fixed

#3637
PowerRun fails to find apps from Store after 0.18.1 upgrade

#3146
Office Applications Not Found

~#3120
missing results in launcher~
fixed with #3223

#3974
Unable to find windows terminal
fixed

#3976
Unable to find apps in other drive

Closing this tracker since only https://github.com/microsoft/PowerToys/issues/3146 is still open.

Closing this tracker since only https://github.com/microsoft/PowerToys/issues/3146 is still open.


Announcing React-RETINA ( for React v15 & also Preact compatible, and supports variable sized ListView)

This library is an alternative Canvas implementation for React v15 (which react-canvas has not yet been updated to use).  It is also compatible with Preact.
https://github.com/xphung/react-retina
Features:

Gallery component supports variable sized pages (whereas react-canvas ListView is limited to fixed size pages)
React-RETINA only uses standard/public React APIs.
The React Retina design avoids numerous React v15 anti-patterns (which React Canvas/React ART are guilty of using).  There are no mixins, string Refs, or even Contexts.  It does not use non standard lifecycle methods.
The React RETINA component library source code is much easier to understand as a result and is leaner and cleaner.
The View base component is extremely lightweight and specialised components (Image, Text, etc) are built on top of View via a custom draw callback.  This is very flexible and allows virtually any drawable shape or component to be built on top of View.
The React RETINA Backing Store implementation is simpler and cleaner to use - instead of requiring specialised hooks/lifecycle to manage invalidation and repaints, just assign a unique ID to the useBackingStore property to make the View the cache a repaint.  Simply assign a new ID whenever an invalidation and repaint into Backing Store is required ... no need for a special Javascript calls or API to invalidate cache data!
Keyboard events are supported!!

@xphung is this meant to be a production ready alternative to react-canvas, and if so do you plan to add documentation and an installable npm package?

Hi fredguest,
React-RETINA is the more lightweight & efficient (in terms of code size and simplicity) than React-Canvas but there are no production-ready guarantees.  The tight code base is much easier to test, understand and modify.
My target & testing for it has been Chrome and OS X WKWebKit only, and it makes no attempt whatsoever to support quirky older browsers.

It sounds very interesting, I'd like to try it out but it's a bit messy to import at the moment. Are you planning to publish it to https://www.npmjs.com/ so people can simply npm install react-retina?

My priority at this stage is getting v1.1 out, before publishing it more widely
Will keep you posted on when v1.1 is out and may consider packaging it to npm then...

Currently, v1.0 is available as a "traditional"

Ok cool sounds good. Looking forward to trying it out!

Interested in using it, but haven't seen any updates since initial commit or an npm package (which is kinda of a must). Are we gonna see more development?

Hi Kungfoolabs, an NPM package isn't my priority right now.  I'm working on v1.1 release first, which will feature major improvements to the Gallery component (essentially breaking it down into a series of separate components).  NPM packaging will be considered after v1.1 has been done.

@xphung Thanks.
Ya, I understand your priorities. Seems like ours don't align. Would love to see a native react extensible canvas framework. Without adoption & support, it'll be unlikely to be used and kind of in the deadpool. I'll look into getting something setup.

@xphung any estimate release date on v1.1? I would also be interested in an npm package.

Hi turnerniles, v1.1 will be about improving the Gallery component (restructuring it into a collection of several smaller components), and is a few mths away.

@xphung Are you still working in it?

@xphung  Impressed and intrigued by the concept of React-RETINA as well as react-canvas. Do you know of any sites using this lib in production yet?


[netcdf-c] Update to 4.7.3 and switched to use targets to generate valid targets

Describe the pull request

netcdf-c is not correctly configuring due to missing includes and library dependencies (zlib/szip)
the generated target was not linking correctly to libcurl (Link interface included both debug and optimized version of libcurl without a generator expressions. Reason for that was a remove duplicates call in the cmakelists.txt which removed duplicated optimized/debug keywords)
the targets were installed into the wrong directory so that find_package was unable to find netcdf (old directory was netcdf-c which does not work if the usage is find_package(netCDF) ) The targets/config are also called netCDF and not netCDF-c. Checked CMake docs again and it searches in name* so its ok

@Neumann-A , thanks for the PR!
I double checked, It involved in regressions when 'set(VCPKG_BUILD_TYPE release)', the failures like:
CMake Error at scripts/cmake/vcpkg_prettify_command.cmake:19 (if):
if given arguments:
"-DSZIP_LIBRARY:STRING=debug" "SZIP_DEBUG-NOTFOUND" "optimized" "F:/VCPKG/numan/vcpkg/installed/x64-windows/lib/libszip.lib" "MATCHES" "( )"

Unknown arguments specified
Call Stack (most recent call first):
scripts/cmake/vcpkg_execute_required_process.cmake:71 (vcpkg_prettify_command)
scripts/cmake/vcpkg_configure_cmake.cmake:312 (vcpkg_execute_required_process)
ports/netcdf-c/portfile.cmake:35 (vcpkg_configure_cmake)
scripts/ports.cmake:94 (include)

So I updated the library and switched everything to use targets instead of library names. So it should link correctly if the other targets are correct. Lets see what CI tells us

You fixed the static build but broke the dynamic build : )

/azp run

/azp run

/azp run


WEB-5922 Move JSON Select to Github

What are the links to relevant tickets?
https://hxshortbreaks.atlassian.net/browse/WEB-5922
What does this PR do?
I have moved JSON Select from Bitbucket to Github
What functional/unit tests does this PR have?
none
How should a developer review this?
glance
Any background context you want to provide?
I don't know the circumstances why we have our own JSON Select - think it is 3rd party code.
Need to investigate after this Jira
Screenshots (if appropriate)

Quality checklist (for PR author to complete BEFORE code review):

[x] I've checked there is appropriate unit test coverage.
[ ] I've updated documentation with any changes to procedures.
[x] I've checked this work against the requirements of the Jira.
[x] This change passes all necessary tests (cross browser, automated or otherwise).
[x] I am ready for this to be code reviewed, merged and tested.

Reviewer 1 @lukefrake

[x] I agree with the assumptions made in the quality checklist.
[ ] I’ve witnessed the work behaving as expected.
[x] I’ve checked for appropriate test coverage.
[x] I’ve checked for coding anti-patterns.
[x] I've checked this work against the requirements of the Jira.
[x] I don’t believe this work introduces unnecessary technical debt.
[x] +1 from me!

Reviewer 2

[ ] I agree with the assumptions made in the quality checklist.
[ ] I’ve witnessed the work behaving as expected.
[ ] I’ve checked for appropriate test coverage.
[ ] I’ve checked for coding anti-patterns.
[ ] I've checked this work against the requirements of the Jira.
[ ] I don’t believe this work introduces unnecessary technical debt.
[ ] +1 from me!

Readme has bitbucket link


Toolbar buttons for Service Orchestration moved to separate class

Purpose or Intent
Purpose is to move toolbar buttons from helper (app/helpers/application_helper/toolbar_builder.rb) to separate classes.
In this PR is converted a Reconfigure this Service button for for Services.
Classes for one button

class ServiceReconfigure contains logic for service_reconfigure button in Service detail page

Toolbar screenshots

service_center.rb

@martinpovolny @PanSpagetka can you review?
Links
Parent issue: #6259
Related issue: #6554
Pivotal Tracker: https://www.pivotaltracker.com/story/show/127545673

@miq-bot remove_label wip

The similarities between the classes will be deduplicated in a separate follow-up PR as discusses f2f.
Merging this as it is now.


Add path to learn spam/ham

POST text/plain to http://localhost/texts/spam to learn a text as spam.
POST text/plain to http://localhost/texts/ham to learn a text as ham.

text/plain replaced by JSON, because there are always some information to transmit...


Refactor/profanity checks

Fix/refactor based on user feedback on Discord:

After upgrading shuttle to 0.5.0, I get an error when trying to do anything (deploy, status, local run, etc.). It tells me, that the name I set is an invalid project name. It does only contain lowercase letters, however, it contains the letters "ass" inside the Word ('dachterasse') so it might trigger the profanity check by mistake. Could this be fixed?

I've removed the Censor::Zealous words as they didn't seem overly terrible in my opinion? I've removed some of the shorter words that might easily be within other words or phrases.
I'm not sure what the below line is/was for, it could be that it's a performance thing and my alternate way isn't ideal for some reason. I'll leave that to someone who knows more than I do on that front! Happy to make the changes though.
static INSTANCE: OnceCell<HashSet<String>> = OnceCell::new();
INSTANCE.get_or_init(|| HashSet::from(["Shuttle.rs".to_string()]));

Nice! I agree that the Zealous word list is a bit much, causing too many false positives.
Regarding the OnceCell, it's used so that the reserved word HashSet only gets instantiated the first time the is_profanity_free_and_not_reserved function is called, not every time. Here is a relevant thread thread from the Rust users forum (lazy_static offers the same functionality we used once_cell for here, but for a few reasons once_cell is recommended now). :slightly_smiling_face:

I've added Rustrict to replace Censor. The reserved 'Shuttle.rs' is covered by other lowercase and punctuation restrictions so I've removed that but left the test in incase it's an issue in future. I've also removed some unused Crates from the common Cargo.toml, they don't seem to be being used anywhere.
Thanks @oddgrd , it's a big help having some references included there to learn more from 😄

Reserved words added back in. I've left "shuttle.rs" in for now as an example. Would it be better to just leave it blank? With us not having any reserved words currently there's no examples to add in for testing purposes either.


Support for LDAP Bind User?

We're looking to integrate Lemur with our LDAP setup (instead of using basic auth), and looking at the configuration: HERE I'm not able to see a way to specify a bind user. Is this not currently supported in lemur, or is there an option I'm missing?

Lemur uses the login credentials of the person trying to authenticate to bind to the ldap service. If ldap bind is successful, then obviously the ldap username/password combination provided are valid. Lemur then looks at the different ldap groups a user is a member of to determine whether access should be granted.

So I worded this incorrectly, and I'm sorry.
Essentially we have users type in sAMAAccountNames (ecoan in my case), but obviously we need the full DN to authenticate. So we have a Service Account that is generated so we can look up the full DN of the User. I'm essentially looking at how I can authenticate with that user to perform those lookups.

No probs. Perhaps I misunderstood.
So you're using AD which is good - because I've only tested against AD. :)
The lemur ldap module uses the user principal name (upn) of the authenticating user to bind. This is done once for each user at login time. The UPN is effectively the email address in AD/LDAP of the user. If the user doesn't provide the email address, it constructs one based on the username supplied (which should normally match the samAccountName) and the value provided by the config LDAP_EMAIL_DOMAIN.
The config LDAP_BASE_DN tells lemur where to search within the AD/LDAP tree for the given UPN (user). If the bind with those credentials is successful - there is a valid user in AD with correct password.
So no other bind user is necessary for authentication. Lemur just uses the upn (or samAccountName and constructs a upn) and the user supplied password.
My lemur config for ldap looks similar to this:
LDAP_AUTH = True
LDAP_BIND_URI='ldaps://domaincontroller.evilcorp.net'
LDAP_BASE_DN='DC=users,DC=evilcorp,DC=net'
LDAP_EMAIL_DOMAIN='evilcorp.com'
LDAP_USE_TLS = True
LDAP_CACERT_FILE = '/opt/app/lemur/cacerts'
LDAP_REQUIRED_GROUP = 'certificate-management-access'
LDAP_GROUPS_TO_ROLES = {'certificate-management-admin': 'admin', 'certificate-management-read-only': 'read-only'}

LEMUR_DEFAULT_ROLE="operator"

So once Lemur is configured similarly and has network connectivity to the server/port defined in LDAP_BIND_URI, you should be good. Try logging in with your email address and valid password.
Perhaps I should update the docs to make this clearer.
Hope this helps and makes sense. Shout out  if it doesn't.

Didn't realize I hadn't closed this! This does indeed work for us, Thanks much!


Configure Jest and write basic tests for src/

The goal is to be able test all of the code in the source folder with a single command.
First,

[x] Install Jest & configure

Then, write some basic tests for:

[x] src/components/GameComponent.jsx
[x] src/contexts/GameContext.js
[ ] src/hooks/useEventEmitter.js
[ ] src/hooks/useEventListener.js
[x] src/hooks/usePhaser.js
[x] src/index.js

Finally,

[x] Configure a test script in package.json

Wrote tests for most files in src directory.


When running on Travis, Karma reports 0 of N tests executed

I'm trying to make my Karma/Angular tests work on Travis as described here, but it keeps saying that 0 tests were executed:
Running "karma:test" (karma) task
INFO [karma]: Karma v0.12.31 server started at http://localhost:9876/
INFO [launcher]: Starting browser Firefox
INFO [Firefox 31.0.0 (Linux)]: Connected on socket C7AVrV2cfPpQquGp3ew4 with id 60880131
Firefox 31.0.0 (Linux): Executed 0 of 4 SUCCESS (0 secs / 0 secs)

There's no such issue when I run the same thing locally, it reports "4 of 4". There's also a blog post by @Swizec where eventually the same result described: "Executed 0 of 114 SUCCESS" (there's a screenshot at the very end).
Here's my set up:
.travis.yml
language: node_js
node_js:
 - '0.11'
before_script:
 - export DISPLAY=:99.0
 - sh -e /etc/init.d/xvfb start

karma.conf.js
module.exports = function(config) {
  var c = {
    basePath: './',

    files: [...],

    autoWatch: true,
    frameworks: ['jasmine'],
    browsers: ['Firefox'],
    plugins: [
      'karma-firefox-launcher',
      'karma-jasmine'
    ]
  };

  config.set(c);
};

Gruntfile.js
module.exports = function(grunt) {
  grunt.initConfig({
    ...
    karma: {
      options: {
        configFile: 'karma.conf.js'
      },
      'test': {
        singleRun: true
      }
    },
    ...
  });

  ...
  grunt.loadNpmTasks("grunt-karma");
  ...

  ...
  grunt.registerTask("fe-test", ["karma:test"]);
  ...
};

Did I miss something?

Hey,
what do your tests look like?
The output from the project I was writing about in that blogpost looks more like this these days:

Nothing special:
describe('AppController', function() {
  var appController;
  var $scope;
  var $location;

  beforeEach(module('app'));

  beforeEach(inject(function($controller, $rootScope, _$location_) {
    $scope = $rootScope;
    $location = _$location_;

    appController = $controller('AppController', {
      $scope: $scope,
      $location: $location
    });
  }));

  it('should say that "notes" is active when location is /notes', function() {
    $location.path('/notes');
    $scope.$apply();

    expect($scope.isNavBarActive('notes')).toBe(true);
    expect($scope.isNavBarActive('categories')).toBe(false);
  });

  // 3 more like this ^^^
});

@loki2302 can you show project or create stub project which reproduce this problem?
Thanks

@maksimr sure:

Project itself: https://github.com/loki2302/nodejs-app-experiment
Tests: https://github.com/loki2302/nodejs-app-experiment/tree/master/fe-test
Travis: https://travis-ci.org/loki2302/nodejs-app-experiment

To run the tests: npm install && bower install && grunt fe-test. Locally, the result is:
...
Running "karma:test" (karma) task
INFO [karma]: Karma v0.12.31 server started at http://localhost:9876/
INFO [launcher]: Starting browser Chrome
INFO [Chrome 41.0.2272 (Linux)]: Connected on socket 0onsddvNPuRq4dm8GUnb with id 97479661
Chrome 41.0.2272 (Linux): Executed 11 of 11 SUCCESS (0.151 secs / 0.146 secs)

When running on Travis, Karma is configured to use Firefox. Outside Travis it uses Chrome. But I've already tried Firefox locally and it worked fine.

@loki2302 thaks
Seems tests are run, but report is broken

@maksimr Oh, that's good news. Any chance I can have it fixed via configuration file only?

Had the same issue (tests running but reporter says 0 out of N). Changed reporter to "dots" now says correctly N out of N.

To me, removing `restartOnFileChange: true' from my karma conf solved this error.


Obsolete Rubocop syntax prevents code linting

There are some minor  Rubocop style issues around which prevents the code from being checked for quality. See below for more details on this issue.
[jonesagyemang:~/Projects/starred_projects/http-2] master+ 2h26m7s ± rubocop
.rubocop.yml: Style/CaseIndentation has the wrong namespace - should be Layout
.rubocop.yml: Style/IndentHash has the wrong namespace - should be Layout
.rubocop.yml: Style/SpaceAroundOperators has the wrong namespace - should be Layout
.rubocop.yml: Style/ExtraSpacing has the wrong namespace - should be Layout
.rubocop_todo.yml: Style/IndentArray has the wrong namespace - should be Layout
.rubocop_todo.yml: Style/MultilineArrayBraceLayout has the wrong namespace - should be Layout
.rubocop_todo.yml: Style/MultilineHashBraceLayout has the wrong namespace - should be Layout
Error: obsolete parameter IndentWhenRelativeTo (for Layout/CaseIndentation) found in .rubocop.yml
`IndentWhenRelativeTo` has been renamed to `EnforcedStyle`
obsolete parameter AlignWith (for Lint/EndAlignment) found in .rubocop.yml
`AlignWith` has been renamed to `EnforcedStyleAlignWith`

This issue should have been resolved.


chore!: remove fixed-length keccak256

Cleaning up some tech debt. We never actually create this opcode so we can just remove it in favour of the variable length version.

Benchmark results
Metrics with a significant change:

note_successful_decrypting_time_in_ms (8): 168 (-37%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Values are compared against data from master at commit a974ec82 and shown if the difference exceeds 1%.
L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
8 txs
32 txs
64 txs

l1_rollup_calldata_size_in_bytes
676
676
676

l1_rollup_calldata_gas
6,424
6,400
6,412

l1_rollup_execution_gas
585,757
585,733
585,745

l2_block_processing_time_in_ms
1,291 (-4%)
4,619 (-2%)
9,124 (-4%)

note_successful_decrypting_time_in_ms
:warning: 168 (-37%)
596 (+5%)
1,018 (-2%)

note_trial_decrypting_time_in_ms
70.0 (+6%)
106 (+39%)
32.4 (-49%)

l2_block_building_time_in_ms
11,669 (-3%)
42,330 (-1%)
83,264

l2_block_rollup_simulation_time_in_ms
11,636 (-3%)
42,212 (-1%)
83,046

l2_block_public_tx_process_time_in_ms
6,051 (-4%)
23,781 (-1%)
47,925 (+1%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric
5 blocks
10 blocks

node_history_sync_time_in_ms
14,098 (-2%)
26,694 (-3%)

note_history_successful_decrypting_time_in_ms
1,194 (-9%)
2,396 (-2%)

note_history_trial_decrypting_time_in_ms
38.3 (-70%)
105 (-3%)

node_database_size_in_bytes
18,612,304
34,881,616 (-1%)

pxe_database_size_in_bytes
29,859
59,414

Circuits stats
Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit
circuit_simulation_time_in_ms
circuit_input_size_in_bytes
circuit_output_size_in_bytes

private-kernel-init
178 (-1%)
44,377
26,164

private-kernel-ordering
160 (-1%)
50,830
39,026

base-parity
3,911 (-2%)
128
311

root-parity
1,059 (+6%)
1,244
311

base-rollup
14,033 (-1%)
110,464
861

root-rollup
48.7 (-2%)
4,359
725

private-kernel-inner
216
71,744
26,164

public-kernel-app-logic
119
47,695
40,661

public-kernel-tail
162
53,372
7,125

merge-rollup
9.66 (+3%)
2,568
861

public-kernel-teardown
116
47,695
40,661

public-kernel-setup
115
47,695
40,661

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
1 leaves
16 leaves
64 leaves
128 leaves
512 leaves
1024 leaves
2048 leaves
4096 leaves
32 leaves

batch_insert_into_append_only_tree_16_depth_ms
9.95 (-1%)
15.9 (-1%)
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.8
31.6
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.579 (-1%)
0.492 (-1%)
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
45.3 (-1%)
71.1 (-2%)
225 (-2%)
444 (-1%)
852 (-3%)
1,693 (-2%)
N/A

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
96.0
159
543
1,055
2,079
4,127
N/A

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
0.464 (-1%)
0.438 (-2%)
0.410 (-3%)
0.415 (-1%)
0.404 (-3%)
0.404 (-2%)
N/A

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
53.6 (-1%)
106 (-1%)
327 (-3%)
660
1,283
2,562 (-2%)
N/A

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
105
207
691
1,363
2,707
5,395
N/A

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
0.473 (-1%)
0.475 (-2%)
0.445 (-3%)
0.455
0.443
0.443 (-2%)
N/A

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
60.6 (-1%)

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
109

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
0.529 (-1%)

Miscellaneous
Transaction sizes based on how many contract classes are registered in the tx.

Metric
0 registered classes
1 registered classes

tx_size_in_bytes
40,372
496,534

Transaction size based on fee payment method

Metric
native fee payment method
fpc_public fee payment method
fpc_private fee payment method

tx_with_fee_size_in_bytes
905
1,161
1,377

Transaction processing duration by data writes.

Metric
0 new note hashes
1 new note hashes
2 new note hashes

tx_pxe_processing_time_ms
1,664 (-1%)
1,074 (-1%)
5,279 (-3%)

Metric
1 public data writes
2 public data writes
3 public data writes
4 public data writes
5 public data writes
8 public data writes

tx_sequencer_processing_time_ms
570
396
1,055 (-1%)
508
1,723
573


[Bug] Leader Election Lost: Kuberay pod restarts every 5mins!

Search before asking

[X] I searched the issues and found no similar issues.

KubeRay Component
ray-operator
What happened + What you expected to happen
Kuberay keep restarting because of leader election lost, I've raised it in Slack couple of times but no luck, we suspect it is causing us issues in some RayServices.
We use kuberay 1.1.1 helm chart version.
Main Issue is here:
{"level":"error","ts":"2024-07-17T13:33:36.127Z","logger":"setup","msg":"problem running manager","error":"leader election lost","stacktrace":"main.exitOnError\n\t/home/runner/work/kuberay/kuberay/ray-operator/main.go:245\nmain.main\n\t/home/runner/work/kuberay/kuberay/ray-operator/main.go:228\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/proc.go:250"}
kuberay-logs.txt
Values configured for helm chart:
  env:
  - name: ENABLE_GCS_FT_REDIS_CLEANUP
    value: "false"
  nodeSelector:
    name: devops
  tolerations:
    - effect: NoSchedule
      key: CriticalAddonsOnly
      operator: Equal
      value: devops
  resources:
    limits:
      cpu: 500m
      # Anecdotally, managing 500 Ray pods requires roughly 500MB memory.
      # Monitor memory usage and adjust as needed.
      memory: 2Gi
    requests:
      cpu: 200m
      memory: 1Gi

replicas: 1
Willing to provide documentation or code if needed :)
Reproduction script
Kuberay 1.1.1, deployed on GKE v1.28.10-gke.107500.
Values configured above (nothing special IMHO).
Anything else
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

How frequently is it happening? I suggest removing the 500m CPU limit in your KubeRay pod spec, it could be leading to leader election requests timing out

Btw, since you mentioend running on GKE, consider using the official GKE addon https://cloud.google.com/kubernetes-engine/docs/add-on/ray-on-gke/how-to/enable-ray-on-gke
However, it's only available on v1.30+ at the moment (you mentioned running v1.28.10)

Thank you @andrewsykim for your quick reply, you're right, limit is no good here.
{"level":"info","ts":"2024-07-17T15:12:06.947Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running; Pod Name: rayservice-raycluster-v44tf-worker-ray-worker-gpu-s9k76; Pod Status.Phase: Failed","RayCluster":{"name":"clip-model-rayservice-raycluster-v44tf","namespace":"clip-model"},"reconcileID":"aa3e136c-fd9a-47ff-8b83-76cf27be3bdc"}
{"level":"info","ts":"2024-07-17T15:12:06.947Z","logger":"controllers.RayCluster","msg":"Environment variable RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV is not set, using default value of 300 seconds","RayCluster":{"name":"clip-model-rayservice-raycluster-v44tf","namespace":"clip-model"},"reconcileID":"aa3e136c-fd9a-47ff-8b83-76cf27be3bdc","cluster name":"clip-model-rayservice-raycluster-v44tf"}
{"level":"info","ts":"2024-07-17T15:12:06.947Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"clip-model-rayservice-raycluster-v44tf","namespace":"clip-model"},"reconcileID":"aa3e136c-fd9a-47ff-8b83-76cf27be3bdc","cluster name":"clip-model-rayservice-raycluster-v44tf","seconds":300}
E0717 15:12:07.440182       1 leaderelection.go:369] Failed to update lock: Put "https://<IP_ADDRESS>:443/apis/coordination.k8s.io/v1/namespaces/devops-ray/leases/ray-operator-leader": context deadline exceeded
I0717 15:12:07.440241       1 leaderelection.go:285] failed to renew lease devops-ray/ray-operator-leader: timed out waiting for the condition
{"level":"error","ts":"2024-07-17T15:12:07.440Z","logger":"setup","msg":"problem running manager","error":"leader election lost","stacktrace":"main.exitOnError\n\t/home/runner/work/kuberay/kuberay/ray-operator/main.go:245\nmain.main\n\t/home/runner/work/kuberay/kuberay/ray-operator/main.go:228\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/proc.go:250"}
@kevin85421 suggested to config kuberay without leader election, do you support env var? RAY_DISABLE_LEADER_ELECTION or something to configure it via values.yaml?
https://ray-distributed.slack.com/archives/C02GFQ82JPM/p1721200745507799?thread_ts=1717181822.513479&cid=C02GFQ82JPM

limit being set to default in kuberay chart:
2024-07-17T15:23:10+00:00   apps                      Deployment                devops-ray      kuberay-operator              OutOfSync  Healthy              Deployment.apps "kuberay-operator" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: "500m": must be less than or equal to cpu limit of 100m
when removing limit and keeping request, getting this error ^

Created this MR to remove default in new version of kuberay:
https://github.com/ray-project/kuberay/pull/2253

Does removing CPU limits resolve the leader election issue though?

@andrewsykim I can't remove the limits currently as we've default in helpm chart, I need to do it manually

Edited manually in stg:

let's see

edited prod too:

but still, the best practice is to avoid setting CPU limits, so I would remove it from the default values in helm chart.

Issue persist, even without cpu limits or downgrade to 1.1.0 kuberay version
any help?

Hmmm, not sure then. Since you're running a single kuberay-operator anyways, you can follow Kai-Hsun's suggestion and just disable leader election

@andrewsykim should I increase replicas? this is prod.
@kevin85421 how to pass that as env var to disable leader election?

seems related: https://github.com/ray-project/kuberay/issues/601

Try setting --enable-leader-election=false in the kuberay-operator flags

can you also share the full pod YAML of the kuberay operator? kubectl get pod <kuberay-operator-pod> -o yaml

@andrewsykim I can do it manually, but not via helm as we don't support this flag: https://github.com/ray-project/kuberay/blob/master/helm-chart/kuberay-operator/templates/deployment.yaml#L57
can I create MR for this?
and here is the yaml:
apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    creationTimestamp: "2024-07-21T09:25:06Z"
    generateName: kuberay-operator-54d77db944-
    labels:
      app.kubernetes.io/component: kuberay-operator
      app.kubernetes.io/instance: kuberay-operator
      app.kubernetes.io/name: kuberay-operator
      pod-template-hash: 54d77db944
    name: kuberay-operator-54d77db944-bsxkt
    namespace: devops-ray
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: kuberay-operator-54d77db944
      uid: f21b2341-a8b5-4856-8816-d1b5dfdb9812
    resourceVersion: "1257112557"
    uid: a40cb305-3bb2-4a22-a69e-8d5506531f1c
  spec:
    containers:
    - command:
      - /manager
      env:
      - name: ENABLE_GCS_FT_REDIS_CLEANUP
        value: "false"
      image: quay.io/kuberay/operator:v1.1.0
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 5
        httpGet:
          path: /metrics
          port: http
          scheme: HTTP
        initialDelaySeconds: 10
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 1
      name: kuberay-operator
      ports:
      - containerPort: 8080
        name: http
        protocol: TCP
      readinessProbe:
        failureThreshold: 5
        httpGet:
          path: /metrics
          port: http
          scheme: HTTP
        initialDelaySeconds: 10
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 1
      resources:
        limits:
          cpu: "8"
          memory: 8Gi
        requests:
          cpu: "1"
          memory: 1Gi
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
          - ALL
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-m56k4
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    imagePullSecrets:
    - name: dv-docker-main-gcp
    nodeName: gke-gp-ms-uc1-k8s-public-mle-devops-1-dab37758-rl5l
    nodeSelector:
      name: devops
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: kuberay-operator
    serviceAccountName: kuberay-operator
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoSchedule
      key: CriticalAddonsOnly
      operator: Equal
      value: devops
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: kube-api-access-m56k4
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
            - key: ca.crt
              path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
            - fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
              path: namespace
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2024-07-21T09:26:15Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2024-07-21T10:28:56Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2024-07-21T10:28:56Z"
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2024-07-21T09:26:15Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://8109abfec6a2c12c0aa452c3cd4fe997632065db3626df7f0874b669b95dc1d8
      image: quay.io/kuberay/operator:v1.1.0
      imageID: quay.io/kuberay/operator@sha256:a535757c28bcc27c06f906146970417cd88769f78df860131625d776608ec309
      lastState:
        terminated:
          containerID: containerd://bebcfb18507c4a237a14eebb85d3356c158155b8d00adaee117138769a2da929
          exitCode: 1
          finishedAt: "2024-07-21T10:28:43Z"
          reason: Error
          startedAt: "2024-07-21T10:05:33Z"
      name: kuberay-operator
      ready: true
      restartCount: 3
      started: true
      state:
        running:
          startedAt: "2024-07-21T10:28:44Z"
    hostIP: <host IP>
    phase: Running
    podIP: <pod IP>
    podIPs:
    - IP: <pod IP>
    qosClass: Burstable
    startTime: "2024-07-21T09:26:15Z"
kind: List
metadata:
  resourceVersion: ""

Created this PR:
https://github.com/ray-project/kuberay/pull/2262
Contributing to this project seems like a big milestone for me, TIA! 🥇

Thank you @andrewsykim for your quick reply, you're right, limit is no good here.
more logs:
{"level":"info","ts":"2024-07-17T15:12:06.947Z","logger":"controllers.RayCluster","msg":"CheckAllPodsRunning: Pod is not running; Pod Name: rayservice-raycluster-v44tf-worker-ray-worker-gpu-s9k76; Pod Status.Phase: Failed","RayCluster":{"name":"clip-model-rayservice-raycluster-v44tf","namespace":"clip-model"},"reconcileID":"aa3e136c-fd9a-47ff-8b83-76cf27be3bdc"}
{"level":"info","ts":"2024-07-17T15:12:06.947Z","logger":"controllers.RayCluster","msg":"Environment variable RAYCLUSTER_DEFAULT_REQUEUE_SECONDS_ENV is not set, using default value of 300 seconds","RayCluster":{"name":"clip-model-rayservice-raycluster-v44tf","namespace":"clip-model"},"reconcileID":"aa3e136c-fd9a-47ff-8b83-76cf27be3bdc","cluster name":"clip-model-rayservice-raycluster-v44tf"}
{"level":"info","ts":"2024-07-17T15:12:06.947Z","logger":"controllers.RayCluster","msg":"Unconditional requeue after","RayCluster":{"name":"clip-model-rayservice-raycluster-v44tf","namespace":"clip-model"},"reconcileID":"aa3e136c-fd9a-47ff-8b83-76cf27be3bdc","cluster name":"clip-model-rayservice-raycluster-v44tf","seconds":300}
E0717 15:12:07.440182       1 leaderelection.go:369] Failed to update lock: Put "https://<IP_ADDRESS>:443/apis/coordination.k8s.io/v1/namespaces/devops-ray/leases/ray-operator-leader": context deadline exceeded
I0717 15:12:07.440241       1 leaderelection.go:285] failed to renew lease devops-ray/ray-operator-leader: timed out waiting for the condition
{"level":"error","ts":"2024-07-17T15:12:07.440Z","logger":"setup","msg":"problem running manager","error":"leader election lost","stacktrace":"main.exitOnError\n\t/home/runner/work/kuberay/kuberay/ray-operator/main.go:245\nmain.main\n\t/home/runner/work/kuberay/kuberay/ray-operator/main.go:228\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/proc.go:250"}

@kevin85421 suggested to config kuberay without leader election, do kuberay supports env var? RAY_DISABLE_LEADER_ELECTION or something to configure it via values.yaml? https://ray-distributed.slack.com/archives/C02GFQ82JPM/p1721200745507799?thread_ts=1717181822.513479&cid=C02GFQ82JPM

Just curious about have you check why leader election timeout?  Did kuberay was killed by OOM-killer? or the networking issue about connecting to the api-server. if you simply disable leader-election, you might  rise of lacking HA.

@Irvingwangjr you're right, it seems like a timeout connecting to the api-server.
any chance we hit API limit?
in Grafana we see 20k request per min, quota in gcp is 3k per min: https://cloud.google.com/kubernetes-engine/quotas
@andrewsykim can you share more knowledge about that?
In stg, after disabling leader election timeout, I see no restarts.
@Irvingwangjr and anyway I'm using 1 replicas for the kuberay, should I increase it and enable leader election? or keep using 1 replica and disable leader election?

Here we can see the big difference between leader election enabled or disabled.
To me it seems that we hit the K8S control plane API server limit, as we see a bunch of timeout.
the question is why? and if anyone else receive it in GKE?

GCP acks that there is a limit of 3k / 1m for API server, this is why we get timeout.

The 3K / minute quota is the default limit for the GKE API and NOT the Kubernetes API server. However, it's possible that your cluster is throttling API requests from kuberay operator. Usually you can figure out if this is happening by looking at the apiserver logs

Closing, moving to regional cluser solved the kuberay restarts.


Maven Based Separate Sample Application

It would be helpful to have a standalone repository with a simple maven example

@tspannhw That is a great idea. We can certainly look into that. In the meantime, did you get to look at these samples?

@tspannhw we have been moving recently toward the trend of keeping samples in the same repo as the main project. This pattern of keeping them close to the code seems to help keep them up-to-date and avoid bit rot. I would prefer to not maintain a separate repo for this.
As @sobychacko pointed out we do have a couple of examples in this repo. However, I just realized that we can not simply point users to those samples as they do have some of our custom Gradle plugins in the build.gradle.
Wdyt about us adding some directions on "Getting started" section of the ref docs (could show a tabbed pane for Gradle / Maven) to help w/ this gap?
Also, we are in the process of making spring-pulsar available on start.spring.io which will give users a simple way to get started w/ a SpringBoot app.

We updated the ref docs to clearly show how to get started w/ gradle or maven. Based on this and the fact that we will be on start.spring.io in the near future I will close this out.


Windows Server 2019 - Password not set

Hello,
with windows 2019 server the admin password defined in config.ini was not set in the image, it asks for new password when first booting on the client machine - any idea how to fix this?
I tried to set "net user Administrator " in "RunAfterSysprep.ps1" CustomResources script, with no effect. The build process didnt show any error
thanks for help beforehand :)

Hello,
The password defined under the [vm] section is used for AutoLogon so that we can perform actions on the VM. The "net user Administrator" will not work, because the Administrator account is being reset by sysprep. During the image generation you can either create another user with a predefined password and administrative rights, or you can specify the user and the password in the unattend.xml used for sysprep (you will find it in the Cloudbase-Init install directory, in the conf folder).
Another method would be to use the "net user Administrator" method inside a script placed in Program Files\Cloudbase Solutions\Cloudbase-Init\LocalScripts directory. Those scripts will be run on first boot. Please check the File Execution paragraph in this article to instruct Cloudbase-init what to do after the script executions is done.
Depending on the environment you are using the image, the password can also be set by Cloudbase-init from the cloud metadata (in Openstack for example), and retrieved later using the nova get-password.
Let us know if this helps or if you face any other issues.

thanks alot for your info, got it working via the LocalScripts Directory! :+1:

Closing this issue, as the problem was solved.


build: sync packages to use same version of jest

Updated packages in monorepo to use the same version of jest related packages
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #3356 into master will decrease coverage by 0.19%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master    #3356     +/-   ##
=========================================
- Coverage   60.09%   59.89%   -0.2%
=========================================
  Files         254      269     +15
  Lines       12451    12566    +115
  Branches     2579     2529     -50
=========================================
+ Hits         7483     7527     +44
+ Misses       4630     4620     -10
- Partials      338      419     +81

Impacted Files
Coverage Δ

packages/amplify-category-hosting/index.js
60% <0%> (-16.93%)
:arrow_down:

...es/graphql-connection-transformer/src/resources.ts
79.74% <0%> (-15.38%)
:arrow_down:

.../amplify-graphql-types-generator/src/validation.ts
85% <0%> (-15%)
:arrow_down:

packages/graphql-auth-transformer/src/resources.ts
77.43% <0%> (-11.42%)
:arrow_down:

packages/amplify-codegen/src/commands/configure.js
86.66% <0%> (-6.67%)
:arrow_down:

packages/amplify-dynamodb-simulator/index.js
71.73% <0%> (-5.44%)
:arrow_down:

packages/amplify-appsync-simulator/src/index.ts
69.14% <0%> (-4.54%)
:arrow_down:

...ages/amplify-graphql-types-generator/src/errors.ts
69.23% <0%> (-3.85%)
:arrow_down:

...ackages/graphql-transformer-core/src/validation.ts
97.67% <0%> (-2.33%)
:arrow_down:

.../graphql-connection-transformer/src/definitions.ts
84.61% <0%> (-2.06%)
:arrow_down:

... and 27 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b74433...589e392. Read the comment docs.