Guacamole creation failed in 3rd step - "Add network firewall rules for guacamole"

Describe the bug
We have successfully deployed v0.4.3 in both DEV and TEST environments.
In DEV environment we deployed the workspace, guacamole, and VM successfully without any errors,
but in TEST, workspace deployment was successful and Guacamole creation failed with this error in the last step:
{
"operations": [
{
"id": "c83d0dbd-71d8-4118-8a02-c5b99efe9e77",
"resourceId": "e7683927-14c0-452f-95f2-d469936eec76",
"resourcePath": "/workspaces/9dc17d2e-6e3b-43ec-bf29-99919d543aab/workspace-services/e7683927-14c0-452f-95f2-d469936eec76",
"resourceVersion": 0,
"status": "deployment_failed",
"action": "install",
"message": "Multi step pipeline failed on step 260421b3-7308-491f-b531-e007cdc0ff46",
"createdWhen":<PHONE_NUMBER>.510277,
"updatedWhen":<PHONE_NUMBER>.765104,
"user": {
"id": "8f110c05-78a6-49d1-9e05-703f530a73b9",
"name": "Dusan Ilic",
"email": "",
"roles": [
"WorkspaceOwner"
],
"roleAssignments": []
},
"steps": [
{
"stepId": "main",
"stepTitle": "Main step for e7683927-14c0-452f-95f2-d469936eec76",
"resourceId": "e7683927-14c0-452f-95f2-d469936eec76",
"resourceTemplateName": "tre-service-guacamole",
"resourceType": "workspace-service",
"resourceAction": "install",
"status": "deployed",
"message": "e7683927-14c0-452f-95f2-d469936eec76: install action completed successfully.",
"updatedWhen":<PHONE_NUMBER>.373291
},
{
"stepId": "12ba0dad-ea6c-4d0d-9255-d316212f5ffa",
"stepTitle": "Add Guacamole URI as AAD redirect URI",
"resourceId": "9dc17d2e-6e3b-43ec-bf29-99919d543aab",
"resourceTemplateName": "tre-workspace-msl",
"resourceType": "workspace",
"resourceAction": "upgrade",
"status": "updated",
"message": "9dc17d2e-6e3b-43ec-bf29-99919d543aab: upgrade action completed successfully.",
"updatedWhen":<PHONE_NUMBER>.685193
},
{
"stepId": "260421b3-7308-491f-b531-e007cdc0ff46",
"stepTitle": "Add network firewall rules for guacamole",
"resourceId": "115f3e59-bce2-4a25-89b0-4d029bd2f8ee",
"resourceTemplateName": "tre-shared-service-firewall",
"resourceType": "shared-service",
"resourceAction": "upgrade",
"status": "updating_failed",
"message": "<ValidationError: "Additional properties are not allowed ('network_rule_collections' was unexpected)">",
"updatedWhen":<PHONE_NUMBER>.510241
}
]
}
]
}
We compared all templates and all versions are the same in both DEV and TEST.
Steps to reproduce

Deploy workspace
Deploy guacamole

Could you please advise how to solve this?
It is a blocker for us now.

@dusan-ilic-mhra I think this could be down to the firewall bundle version in use, and the fact it hasn't got the network_rule_collections property.

Can you get the firewall shared service template via the API and verify it has the appropriate property. Can paste the output here. Thanks.

That's all templates, we need just the firewall shared service template.

Here it is:

And this is a response body:
{
"id": "68a72c51-e601-4a46-b6bd-59cc3899e6ff",
"name": "tre-shared-service-firewall",
"title": "Firewall Shared Service",
"description": "Provides Firewall shared service",
"version": "0.4.3",
"resourceType": "shared-service",
"current": true,
"type": "object",
"required": [
"display_name",
"description"
],
"properties": {
"rule_collections": {
"type": "array",
"title": "rule_collections",
"description": "",
"default": [],
"updateable": true,
"items": {
"title": "items",
"type": "object",
"required": [
"name",
"action",
"rules"
],
"properties": {
"name": {
"title": "name",
"type": "string",
"examples": [
"my-rule"
],
"pattern": "^.$"
},
"priority": {
"title": "priority",
"type": "integer",
"description": "Numeric priority for rule collection - leave blank for auto numbering",
"examples": [
110
]
},
"action": {
"title": "action",
"type": "string",
"examples": [
"Allow"
],
"enum": [
"Allow",
"Deny"
]
},
"rules": {
"title": "rules",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "object",
"required": [
"name"
],
"properties": {
"name": {
"title": "name",
"type": "string",
"examples": [
"rule 1"
],
"pattern": "^.$"
},
"description": {
"title": "description",
"type": "string",
"default": "",
"examples": [
"My rule description here"
],
"pattern": "^.$"
},
"protocols": {
"title": "protocols",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "object",
"required": [
"port",
"type"
],
"properties": {
"port": {
"title": "port",
"type": "string",
"examples": [
"1234"
],
"pattern": "^.$"
},
"type": {
"title": "type",
"type": "string",
"enum": [
"Http",
"Https",
"Mssql"
],
"examples": [
"Http"
]
}
}
}
},
"fqdn_tags": {
"title": "fqdn_tags",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"default": "",
"enum": [
"AppServiceEnvironment",
"AzureBackup",
"AzureKubernetesService",
"HDInsight",
"MicrosoftActiveProtectionService",
"WindowsDiagnostics",
"WindowsUpdate",
"WindowsVirtualDesktop"
],
"examples": [
"AzureKubernetesService"
]
}
},
"target_fqdns": {
"title": "target_fqdns",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"default": "",
"examples": [
"one.two.three.com"
]
}
},
"source_addresses": {
"title": "Source_addresses",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"default": "",
"examples": [
"<IP_ADDRESS>"
]
}
}
}
}
}
}
}
},
"display_name": {
"type": "string",
"title": "Name for the shared service",
"description": "The name of the shared service to be displayed to users",
"updateable": true
},
"description": {
"type": "string",
"title": "Description of the shared service",
"description": "Description of the shared service",
"updateable": true
},
"overview": {
"type": "string",
"title": "Shared Service Overview",
"description": "Long form description of the shared service, in markdown syntax",
"updateable": true
}
},
"actions": [],
"customActions": [],
"uiSchema": {},
"additionalProperties": false,
"system_properties": {
"tre_id": {
"type": "string",
"title": "",
"description": ""
},
"shared_service_id": {
"type": "string",
"title": "",
"description": ""
},
"azure_location": {
"type": "string",
"title": "",
"description": ""
}
}
}

looks like you haven't got the latest firewall template installed? if you see here on main - there's a property called network_rule_collections you're missing... https://github.com/microsoft/AzureTRE/blob/239df38a2a19259897d801806fe0ad12e607e16e/templates/shared_services/firewall/template_schema.json#L169

Maybe a merge issue, looks like that property is in '0.4.0'.

We have this one in main branch. We deployed this from main to DEV environment without problems, but here in TEST we have this issue.
This is main branch:

We first put a version in cosmos DB manually as you can see from the screenshot above, and after that we deployed from CI.

@dusan-ilic-mhra I've edit your message and deleted the history as included people's personal details.
the JSON shared here - https://github.com/microsoft/AzureTRE/issues/2715#issuecomment-1273420914 does not include network_rule_collections as in the JSON here - https://github.com/microsoft/AzureTRE/blob/764b0046edbb6ecaa5b8c18fc10225d44e75c940/templates/shared_services/firewall/template_schema.json
It seems that the latest version of the template has not been correctly registered in the TEST environment. Can you try registering the template again, you might need to increment the version on porter.yaml if it complains that it already exists.
Then run the API call again to get the firewall shared services template, ensure that network_rule_collections is present in the response.

@marrobi
I published and registered version 0.4.4 of tre-shared-service-firewall because it complained about the previous one.
And, I can see now network_rule_collections in JSON here in templates:

Full response body:
{
"id": "73f485e8-cfd8-4c0c-b3f2-5123dfdb8da1",
"name": "tre-shared-service-firewall",
"title": "Firewall Shared Service",
"description": "Provides Firewall shared service",
"version": "0.4.4",
"resourceType": "shared-service",
"current": true,
"type": "object",
"required": [
"display_name",
"description"
],
"properties": {
"rule_collections": {
"type": "array",
"title": "rule_collections",
"description": "",
"default": [],
"updateable": true,
"items": {
"title": "items",
"type": "object",
"required": [
"name",
"action",
"rules"
],
"properties": {
"name": {
"title": "name",
"type": "string",
"examples": [
"my-rule"
],
"pattern": "^.$"
},
"priority": {
"title": "priority",
"type": "integer",
"description": "Numeric priority for rule collection - leave blank for auto numbering",
"examples": [
110
]
},
"action": {
"title": "action",
"type": "string",
"examples": [
"Allow"
],
"enum": [
"Allow",
"Deny"
]
},
"rules": {
"title": "rules",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "object",
"required": [
"name"
],
"properties": {
"name": {
"title": "name",
"type": "string",
"examples": [
"rule 1"
],
"pattern": "^.$"
},
"description": {
"title": "description",
"type": "string",
"default": "",
"examples": [
"My rule description here"
],
"pattern": "^.$"
},
"protocols": {
"title": "protocols",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "object",
"required": [
"port",
"type"
],
"properties": {
"port": {
"title": "port",
"type": "string",
"examples": [
"1234"
],
"pattern": "^.$"
},
"type": {
"title": "type",
"type": "string",
"enum": [
"Http",
"Https",
"Mssql"
],
"examples": [
"Http"
]
}
}
}
},
"fqdn_tags": {
"title": "fqdn_tags",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"default": "",
"enum": [
"AppServiceEnvironment",
"AzureBackup",
"AzureKubernetesService",
"HDInsight",
"MicrosoftActiveProtectionService",
"WindowsDiagnostics",
"WindowsUpdate",
"WindowsVirtualDesktop"
],
"examples": [
"AzureKubernetesService"
]
}
},
"target_fqdns": {
"title": "target_fqdns",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"default": "",
"examples": [
"one.two.three.com"
]
}
},
"source_addresses": {
"title": "Source_addresses",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"default": "",
"examples": [
"<IP_ADDRESS>"
]
}
}
}
}
}
}
}
},
"network_rule_collections": {
"type": "array",
"title": "network_rule_collections",
"description": "",
"default": [],
"updateable": true,
"items": {
"title": "items",
"type": "object",
"required": [
"name",
"action",
"rules"
],
"properties": {
"name": {
"title": "name",
"type": "string",
"examples": [
"my-rule"
],
"pattern": "^.$"
},
"priority": {
"title": "priority",
"type": "integer",
"description": "Numeric priority for rule collection - leave blank for auto numbering",
"examples": [
110
]
},
"action": {
"title": "action",
"type": "string",
"examples": [
"Allow"
],
"enum": [
"Allow",
"Deny"
]
},
"rules": {
"title": "rules",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "object",
"required": [
"name"
],
"properties": {
"name": {
"title": "name",
"type": "string",
"examples": [
"rule 1"
],
"pattern": "^.{5,80}$"
},
"description": {
"title": "description",
"type": "string",
"default": "",
"examples": [
"My rule description here"
],
"pattern": "^.$"
},
"source_addresses": {
"title": "Source_addresses",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"default": "",
"examples": [
"<IP_ADDRESS>"
]
}
},
"source_ip_groups": {
"title": "Source_ip_groups",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"default": "",
"examples": [
"some_ip_group_name"
]
}
},
"destination_addresses": {
"title": "Destination_addresses",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"default": "",
"examples": [
"<IP_ADDRESS>"
]
}
},
"destination_ip_groups": {
"title": "Destination_ip_groups",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"default": "",
"examples": [
"some_ip_group_name"
]
}
},
"destination_fqdns": {
"title": "Destination_fqdns",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"default": "",
"examples": [
"one.two.three.com"
]
}
},
"destination_ports": {
"title": "Destination_ports",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"default": "",
"examples": [
"80",
"443",
"*"
]
}
},
"protocols": {
"title": "protocols",
"type": "array",
"default": [],
"items": {
"title": "Items",
"type": "string",
"enum": [
"Any",
"ICMP",
"TCP",
"UDP"
],
"examples": [
"TCP"
]
}
}
}
}
}
}
}
},
"display_name": {
"type": "string",
"title": "Name for the shared service",
"description": "The name of the shared service to be displayed to users",
"updateable": true
},
"description": {
"type": "string",
"title": "Description of the shared service",
"description": "Description of the shared service",
"updateable": true
},
"overview": {
"type": "string",
"title": "Shared Service Overview",
"description": "Long form description of the shared service, in markdown syntax",
"updateable": true
}
},
"actions": [],
"customActions": [],
"uiSchema": {},
"additionalProperties": false,
"system_properties": {
"tre_id": {
"type": "string",
"title": "",
"description": ""
},
"shared_service_id": {
"type": "string",
"title": "",
"description": ""
},
"azure_location": {
"type": "string",
"title": "",
"description": ""
}
}
}
Now, I guess we need to deploy/update the existing firewall, right?
How to do that through swagger, just by running POST like this:

?
Just to note, that we need to update the existing one.

Now we got this:

I'll try to restart the environment and deploy it again.

I restarted the environment and tried to deploy guacamole again, and I got the same error:

Then, I tried to deploy a new workspace, I'm getting this now:

@dusan-ilic-mhra when you say restarted the environment, how do you mean? The 504 suggests the API app has an issue. Have you looked at the API status in the app gateway backend health?
If you connect to the resource processor, can you carry out az rest calls against management.azure.com?

I run make tre-stop, and then make tre-start.
This is what we get from the health probe:

And, backend pool for API:

The same gateway configuration worked yesterday and today.

I'd avoid running make tre-start and make tre-stop when troubleshooting. It's not really restarting the system, but a way to deallocate resources to save some money.
Can you try manually editing the health probe for app gateway in the portal - say increasing the timeout, hitting Test before you save and see what comes back.

The same:

Now it's healthy without any change, odd:

And, again unhealthy:

Could you please check if is there any change or maintenance on Microsoft backbone, because we didn't change App Gateway settings?

Any errors in the API app logs? Could be the API is restarting.
Actually, check your firewall is healthy. Might be one of the start/stop has caused a problem with outbound connectivity.

We can see these errors:

But, why start/stop would cause this?
The environment is stopped every night automatically and we need to start it in the morning.

Again healthy:

And, we can't see any timeout issues:

Without any change.

And, again:

It looks like an issue with the MSI token endpoint. I feel this is unrelated to the management.azure.com timeout at deploy time. If the timeout still appears, I suggest changing app service plan sku from say P1v3 to P2v3 and see if it resolves. This will force a redeploy of the web app.
Once it is stable, try the deployment, if you still get the management.azure.com issue, suggest jumping onto the resource processor and checking external connectivity.

This morning we don't have an issue. I deployed successfully a new workspace, also guacamole and VM:
As you can see new, 3rd step for guacamole succeeded after we published and registered the firewall template with an increased porter version:
{
"operations": [
{
"id": "3a8c7893-2669-4837-9a24-f1edd453608b",
"resourceId": "35d301b1-511a-4f6d-b81a-46e627bbcaf5",
"resourcePath": "/workspaces/d91041b1-275a-4e79-9301-6df65916898e/workspace-services/35d301b1-511a-4f6d-b81a-46e627bbcaf5",
"resourceVersion": 0,
"status": "deployed",
"action": "install",
"message": "Multi step pipeline completed successfully",
"createdWhen":<PHONE_NUMBER>.141138,
"updatedWhen":<PHONE_NUMBER>.137825,
"user": {
"id": "8f110c05-78a6-49d1-9e05-703f530a73b9",
"name": "Dusan Ilic",
"email": "",
"roles": [
"WorkspaceOwner"
],
"roleAssignments": []
},
"steps": [
{
"stepId": "main",
"stepTitle": "Main step for 35d301b1-511a-4f6d-b81a-46e627bbcaf5",
"resourceId": "35d301b1-511a-4f6d-b81a-46e627bbcaf5",
"resourceTemplateName": "tre-service-guacamole",
"resourceType": "workspace-service",
"resourceAction": "install",
"status": "deployed",
"message": "35d301b1-511a-4f6d-b81a-46e627bbcaf5: install action completed successfully.",
"updatedWhen":<PHONE_NUMBER>.531701
},
{
"stepId": "12ba0dad-ea6c-4d0d-9255-d316212f5ffa",
"stepTitle": "Add Guacamole URI as AAD redirect URI",
"resourceId": "d91041b1-275a-4e79-9301-6df65916898e",
"resourceTemplateName": "tre-workspace-msl",
"resourceType": "workspace",
"resourceAction": "upgrade",
"status": "updated",
"message": "d91041b1-275a-4e79-9301-6df65916898e: upgrade action completed successfully.",
"updatedWhen":<PHONE_NUMBER>.571059
},
{
"stepId": "260421b3-7308-491f-b531-e007cdc0ff46",
"stepTitle": "Add network firewall rules for guacamole",
"resourceId": "115f3e59-bce2-4a25-89b0-4d029bd2f8ee",
"resourceTemplateName": "tre-shared-service-firewall",
"resourceType": "shared-service",
"resourceAction": "upgrade",
"status": "updated",
"message": "115f3e59-bce2-4a25-89b0-4d029bd2f8ee: upgrade action completed successfully.",
"updatedWhen":<PHONE_NUMBER>.137815
}
]
}
]
}

Great, glad its sorted.

Thank you for the help.


Add Hashmap StateMachine

Could be preliminarily merged so it can be used for the changes we talked about with the Register StateMachine as well but still needs more work!

Would it be possible to put this entirely in the examples/ directory?  Ideally we would do that for all StateMachine implementations that aren't used in unit tests.  This probably includes the register one too, I don't think we are currently using that.

@danburkert I put this here on the precidence of the register example, but I do agree. I'll move the register example and do this as well.

@danburkert These examples need some documentation and some cleaning but they're functional. I changed their interface a bit to make it easier to test via scripts.
I do agree that we should probably make state machine not return results and include some help on dealing with errors.

Removed the Result<_,_> from StateMachine. I'd like to set up snapshotting on the examples but then should be ready for r+.
Last tests have been failing due to a dependency. We'll need to re-fire the build once it's fixed.

@homu r=danburkert

:pushpin: Commit dc5f017 has been approved by danburkert

:hourglass: Testing commit dc5f017 with merge 959bd95...

:sunny: Test successful - status


mailman: 2.1.18 -> 2.1.23

Motivation for this change
https://lwn.net/Alerts/700805/
https://github.com/NixOS/nixpkgs/issues/18856
Note that mailman3  is out too, but this is still up to date.
Things done

[ ] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] OS X
[ ] Linux

[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Fits CONTRIBUTING.md.

backported to 16.09


Node process is not properly killed

It should be possible to fetch the PID of the process running by rialto.. I think there is a method for it but it is not accessible.

While writing the initial basecode of Rialto, I've added a method to get the process PID but removed it later because the library should be smart enough to manage its own Node process.
Rialto's goal is to provide a complete abstraction over the PHP/Node communication, providing a method to get the process PID would ruin this abstraction because the process could be killed (or whatever else) by the developer.
However, before rejecting the idea, maybe could you tell me why you need the PID? What will you do with it? If it's only for debugging purposes, you can already get the PID through a logger. 😉

Thank you for responding.
My php application launches a node process and I need to have the possibility to kill the node process without killing php. Maybe I'm doing something wrong but sometimes the node process keeps living.

Then we should consider this as a bug. I will need some informations to help you:

Preferably, a code reproducing the bug on all executions, but if you can't manage to create one then extract the code directly from your application (please, strip anything useless for me).
Provide the logs generated by your example, see how to get them in PuPHPeteer's guidelines.
What is your environment? OS (+ version), Node version, and PHP version.
Which package manager are you using? NPM? Yarn ? Other?
Are you using NVM? Which version?

If you have some additional context, please provide it.

Before we are sure its a bug and I deliver what you asked, is there a way to stop a process ? I do not think it should be complicated. I think I'm doing something wrong. I'm using your puppeteer wrapper and when I want to stop the process im doing the following.
    $this->browser->close();         unset($this->page);         unset($this->browser);         unset($this->puppeteer);

@sharokh1 Anything new?

Hi, sorry for not replying earlier!
When the resources are unset, the puppeteer (node) process continuous to run in the background.
To fix this, we made sure the php process is killed. This way all the relevant processes will also be killed.

So, should I consider this issue as fixed?
If you want, you can provide me with a code reproducing the bug, and I can try by myself to see what’s the cause of this.

Closing since the issue is inactive.


PUT Method not allowed (External SSL)

Config:  When I attempt to validate my app through smarthings hub I receive:
Sending HTTP request PUT https://api.smartthings.com/apps//confirm-registration?token=
And then a MethodNotAllowed exception.
The odd thing about my config is that I use Traefik/Acme/LetsEncrypt to convert https->http for my app, I'm wondering if that is the issue - any tips/updates appreciated.

Quite possible that Samsung has changed the spec since I last updated.  I have a smart app that is live without issue however, I thing the confirm-reg would only be called during install.  Will check as soon as time permits.

Samsung changed how WebHook based SmartApps are registered and is now using the CONFIRMATION api as opposed to the PING api.  I will make the updates as well as rev to .NET 5.  Will also take the opportunity to do some cleanup.  This will require a number of changes.  Stay tuned.

Should be fixed in latest release

THANKS!  Will pull it in a few days.


Export directives as top-level package exports

Is there any reason, why directives are not top-level package exports? Also tslint complains that submodules are treated as internal API and best practise is to stick only with top-level exports.
If there is no specific reason about that, I can submit PR to fix that.

Yeah, at the very least, all directives should be importable from the same file (ie directives/index.js).

Since we don't know what directives might be used, we don't want to export them all and force all their implementation code to be loaded. Directives are going to stay in their own modules so they can be imported a la carte.
I would suggest that the tslint no-submodule-imports rule is rather presumptive and definitely won't apply generally. lodash-es, for instance, publishes submodules and they are definitely prefered over the main module.


TS-1537: Fixing merge conflicts between master and release

🐻 with me on this one...
A few days ago I merged a PR to release from a different branch than master (as master had changes we didn't want to push up yet). Now I want to merge master into release, however GitHub flags a merge conflict. This PR aims to fix it, since I cannot merge release into master, as that would drag in tons of ancient stuff.

Has that PR that went from other to release been merged into master?
The link shared at the end of a diff between release and master shows that master is missing some changes that amount to 23 additions and 1 deletion.

Has that PR that went from other to release been merged into master?

Yes, it's this one.
+23 -1
I feel very silly no not to have noticed that there were a lot of commits in release vs master, but just the 3 files I was interested in anyway 🤦

that there were a lot of commits in release vs master

Most of those commits are meaningless as all they do is show that a merge happened - they don't contain any actual changes.
Any non-merge commits are the ones that are likely actually missing from master and are contributing to the file diff you see.


[VS Code] [Audio] Electron not found: /Users/onurcanavci/Library/Application Support/Code/User/globalStorage/ms-vsliveshare.vsliveshare-audio/electron-4.2.7/dist/Electron.app/Contents/MacOS/Electron

Error:
Electron not found: /Users/onurcanavci/Library/Application Support/Code/User/globalStorage/ms-vsliveshare.vsliveshare-audio/electron-4.2.7/dist/Electron.app/Contents/MacOS/Electron
Steps to Reproduce:
1.
2.

Version Data

extensionName
VSLS-Audio

extensionVersion
0.1.80

applicationName
VSCode

applicationVersion
1.43.2

platformName
MacOS

platformVersion
19.3.0

Thanks for reaching out. This seems like a dupe of #3337. Let's continue the conversation there.


Case sensitivity for the actions

Nice tool. But when you make a query with please-cli, you get these action prompts:
❗ What should I do? [use arrow keys or initials to navigate]
> [I] Invoke   [C] Copy to clipboard   [Q] Ask a question   [A] Abort

Note that the keys displayed are capital letters. It was my instinct to hit the capital I for example to invoke. But it seems that only the lower case letters (i etc.) actually work. This app should either display lower case letters for the actions or maybe more ideally be case-insensitive.
Please 0.3.0 Tested on MacOS Ventura 13.5.1, installed w/ homebrew, using zsh shell in mac terminal app

Came here to open this issue. Faced exactly this problem.


Add arithmetic error handling

This commit adds an exception type for arithmetic exceptions. The
traffic cop now catches this exception and returns an appropriate
error message.
Asin and Acos will now raise this exception when given an invalid
input.
Fixes #1098

This is the same error code used by postgress for invalid input on acos and asin
https://github.com/postgres/postgres/blob/53095b5fe650270118bc2ab77416d08e19472cd3/src/backend/utils/adt/float.c#L1768-L1776

Do we want a different exception type for classes of errors? I envisioned exception types maybe per-component. What does a separate exception type here buy us over just an ExecutionException with the properly defined code and message?

Do we want a different exception type for classes of errors? I envisioned exception types maybe per-component. What does a separate exception type here buy us over just an ExecutionException with the properly defined code and message?
It doesn't seem like we need the traffic cop to know how to handle every possible exception in the system.

I think using ExecutionException is probably fine, and I can test it out. I was just nervous because it was already thrown from a bunch of places I didn't know what implications it would have to catch it and return an error in the traffic cop.

Update: after discussing with @mbutrovich the new approach is to reuse ExecutionException. Before we can do that, we first need to add an error code to ExecutionException. Currently ExecutionException seems to only be thrown in the case of software bugs and it crashes the DB. A consequence of that change would be that ExecutionException will no longer crash the DB and instead just abort the current transaction and send an error to the user.

Pending #1146

Who ever merges this, can they please change the commit message from
"This commit adds an exception type for arithmetic exceptions. The
traffic cop now catches this exception and returns an appropriate
error message.
Asin and Acos will now raise this exception when given an invalid
input.
Fixes #1098"
to
"Asin and Acos now raise an ExecutionException when given an
invalid input, so that it is handled properly.
Fixes #1098"


Exclude image in charactername

[code]
I'm sorry, but an uncaught exception occurred.
While running game code:
File "renpy/common/00start.rpy", line 237, in script call
call _splashscreen from _call_splashscreen_1
File "game/story/ch0.rpy", line 29, in script
$ lipsync(but, "01.ogg", "“What the heck is this?”")
File "game/story/ch0.rpy", line 29, in
$ lipsync(but, "01.ogg", "“What the heck is this?”")
File "game/lip-sync-plugin/lipsync_module.rpy", line 30, in lipsync
load_lipsync_data(character_name, audio_track)  # Load lipsync data based on the audio track
File "game/lip-sync-plugin/lipsync_module.rpy", line 20, in load_lipsync_data
lipsync_text = renpy.file(file_path).read().decode("utf-8")
OSError: Couldn't find file 'lip-sync-plugin/lip-sync-data/BATTLER USHIROMIYA {image=gui/wing.png}/01.txt'.
-- Full Traceback ------------------------------------------------------------
Full traceback:
File "renpy/common/00start.rpy", line 237, in script call
call _splashscreen from _call_splashscreen_1
File "game/story/ch0.rpy", line 29, in script
$ lipsync(but, "01.ogg", "“What the heck is this?”")
File "D:\RenPy\renpy-8.0.3-sdk\renpy\ast.py", line 1138, in execute
renpy.python.py_exec_bytecode(self.code.bytecode, self.hide, store=self.store)
File "D:\RenPy\renpy-8.0.3-sdk\renpy\python.py", line 1122, in py_exec_bytecode
exec(bytecode, globals, locals)
File "game/story/ch0.rpy", line 29, in
$ lipsync(but, "01.ogg", "“What the heck is this?”")
File "game/lip-sync-plugin/lipsync_module.rpy", line 30, in lipsync
load_lipsync_data(character_name, audio_track)  # Load lipsync data based on the audio track
File "game/lip-sync-plugin/lipsync_module.rpy", line 20, in load_lipsync_data
lipsync_text = renpy.file(file_path).read().decode("utf-8")
File "D:\RenPy\renpy-8.0.3-sdk\renpy\exports.py", line 2608, in file
return open_file(fn, encoding=encoding)
File "D:\RenPy\renpy-8.0.3-sdk\renpy\exports.py", line 2590, in open_file
rv = renpy.loader.load(fn, directory=directory)
File "D:\RenPy\renpy-8.0.3-sdk\renpy\loader.py", line 833, in load
raise IOError("Couldn't find file '%s'." % name)
OSError: Couldn't find file 'lip-sync-plugin/lip-sync-data/BATTLER USHIROMIYA {image=gui/wing.png}/01.txt'.
Windows-10-10.0.19045 AMD64
Ren'Py <IP_ADDRESS>90503
Umineko: When They Cry - Witch's Promise 1.0
Wed Sep 13 18:12:50 2023
[/code]
Code should exclude/ignore {image=gui/wing.png} because you can't put a / in a folder name

can you please send me the structure of your directory?
first, voice file should be located like this before the lipsync data generation.
audio/voice/character_name/voice_file.ogg
so the generated data file would be displayed as follows:
lip-sync-data/character_name/voice_file.txt

It would be also helpful if you can send me your script.rpy file

Hello, Code(generate_lipsync_data.py, lipsync_module.rpy) is updated now. I'd be grateful if you check them out.

Looks good. Question is there support for more than one pose? If yes, how can I set this up?

Do you mean enabling to change the pose during the lipsync animation?

If you simply just need changing pose within the dialogue, not during the lipsync, you can simply use layered images. layered image is a stack of floating images. so you can define the group of base images (pose) to change different pose easily

Okay, another question is it possible to extend dialogue with extend "text" while using $ lipsync?

To be honest, it is one of my milestones for now. It is not possible to use extend in my program. Yet i'm planning to refactor my code more native by overriding classes in renpy framework. But it would take some extra time i guess.

My idea is to take out the text render function and use the normal one from renpy itself, so that $ lipsync only plays the audio and does not render the text. That way you could also use extend quite simply. I think rendering text a bit too much, the normal say screen is also sufficient.

thanks for your helpful suggestion. I will definitely consider them. I will try to make changes within few days


print WAR and JAR are not published to MVN since setup of github actions

for example, version 3.23 was not published:
https://mvnrepository.com/artifact/org.mapfish.print/print-lib

They are published here: https://github.com/orgs/mapfish/packages

not an issue then


Handle asyncio.CancelledError

Ticket
PLAT-10280
Description
We currently do not handle asyncio.CancelledError. In certain situations this could cause the datafeed to keep reading. This fixes it by catching the CancelledError exception, stopping the datafeed and rethrowing it without recreating the datafeed.

Shouldn't we catch a KeyboardInterrupt instead, like stated here: https://stackoverflow.com/questions/48562893/how-to-gracefully-terminate-an-asyncio-script-with-ctrl-c or https://stackoverflow.com/questions/37417595/graceful-shutdown-of-asyncio-coroutines ?

Shouldn't we catch a KeyboardInterrupt instead, like stated here: https://stackoverflow.com/questions/48562893/how-to-gracefully-terminate-an-asyncio-script-with-ctrl-c or https://stackoverflow.com/questions/37417595/graceful-shutdown-of-asyncio-coroutines ?

asyncio does not throw KeyboardInterrupt. I just tested and it does not work. When I do ctrl+c nothing happens. Ctrl+C again ends the program without running the logic inside the except. For example in my program that utilizes the api-client:
try:
    await datafeed
except KeyboardInterrupt as ex:
    print("Exception Keyboard Interrupt")
    logging.exception("Cancelled Exception" + str(ex))
print("after cleanup")

neither is printed nor is is logged. When I use CancelledError it works.
I was never able to reproduce the issue where the program will not stop and the exception is swallowed completely but, this is to fix the fact that we had no way to cleanly end the datafeed which should fix the exception getting swallowed up as well (hopefully)

asyncio does not throw KeyboardInterrupt. I just tested and it does not work. When I do ctrl+c nothing happens. Ctrl+C again ends the program without running the logic inside the except. For example in my program that utilizes the api-client:
try:
    await datafeed
except KeyboardInterrupt as ex:
    print("Exception Keyboard Interrupt")
    logging.exception("Cancelled Exception" + str(ex))
print("after cleanup")

neither is printed nor is is logged. When I use CancelledError it works.
I was never able to reproduce the issue where the program will not stop and the exception is swallowed completely but, this is to fix the fact that we had no way to cleanly end the datafeed which should fix the exception getting swallowed up as well (hopefully)

Here https://stackoverflow.com/questions/30765606/whats-the-correct-way-to-clean-up-after-an-interrupted-event-loop it seems to suggest that the behavior might be different with different versions of Python. Which version are you using @anthony-symphony?

Here https://stackoverflow.com/questions/30765606/whats-the-correct-way-to-clean-up-after-an-interrupted-event-loop it seems to suggest that the behavior might be different with different versions of Python. Which version are you using @anthony-symphony?

The behavior is the same it is the way we start the loop that is different. Based on the link where they are putting except KeyboardInterrupt would be similar to what is implemented here:
https://github.com/python/cpython/blob/3.8/Lib/asyncio/runners.py
Where it catches the KeyboardInterrupt and then cancels the task. But, when it cancels the task our program will still do nothing because it throws the CancelledError but, we don't catch it until the catch all where we log it and recreate the datafeed. We can verify this with the old code as we see it hit this except block.
https://github.com/SymphonyPlatformSolutions/symphony-api-client-python/blob/628cbf3aa857d3bc66a98a3ec009ce1c82295d05/sym_api_client_python/datafeed_event_service.py#L282
When I try to catch KeyboardInterrupt instead of CancelledError I end up hitting the same block as well.
This change will catch the CancelledError, stop the datafeed and reraise it so the developer can implement their cleanup.

The behavior is the same it is the way we start the loop that is different. Based on the link where they are putting except KeyboardInterrupt would be similar to what is implemented here:
https://github.com/python/cpython/blob/3.8/Lib/asyncio/runners.py
Where it catches the KeyboardInterrupt and then cancels the task. But, when it cancels the task our program will still do nothing because it throws the CancelledError but, we don't catch it until the catch all where we log it and recreate the datafeed. We can verify this with the old code as we see it hit this except block.
https://github.com/SymphonyPlatformSolutions/symphony-api-client-python/blob/628cbf3aa857d3bc66a98a3ec009ce1c82295d05/sym_api_client_python/datafeed_event_service.py#L282
When I try to catch KeyboardInterrupt instead of CancelledError I end up hitting the same block as well.
This change will catch the CancelledError, stop the datafeed and reraise it so the developer can implement their cleanup.

I ran the async loop and verified we can send a message and receive one from the bot before hitting the keyboardInterrupt (Ctrl+c).
It interrupts and stops the bot with these python versions: 3.9.0 | 3.8.0 | 3.7.9 | 3.6.12
Can you give more information about your environment so that I can reproduce the issue?

I ran the async loop and verified we can send a message and receive one from the bot before hitting the keyboardInterrupt (Ctrl+c).
It interrupts and stops the bot with these python versions: 3.9.0 | 3.8.0 | 3.7.9 | 3.6.12
Can you give more information about your environment so that I can reproduce the issue?

So using this main
`import asyncio
import logging
import os
import sys
from asyncio import CancelledError
from sym_api_client_python.auth.rsa_auth import SymBotRSAAuth
from sym_api_client_python.clients.sym_bot_client import SymBotClient
from sym_api_client_python.configure.configure import SymConfig
def configure_logging():
logging.basicConfig(
level=logging.DEBUG
)
logging.getLogger("urllib3").setLevel(logging.WARNING)
async def main():
print('Python Client runs using RSA authentication')
if sys.version_info[0] == 3 and sys.version_info[1] >= 8 and sys.platform.startswith('win'):
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
configure = SymConfig(os.path.join(os.path.dirname(file), "..", "sym_api_client_python", "resources", "rsa_config.json"))
configure.load_config()
auth = SymBotRSAAuth(configure)
auth.authenticate()
bot_client = SymBotClient(auth, configure)
datafeed_event_service = bot_client.get_async_datafeed_event_service()
print('Starting datafeed')
datafeed = datafeed_event_service.start_datafeed()
print('Started datafeed')
try:
await datafeed
except KeyboardInterrupt as ex:
print("Exception Keyboard Interrupt")
logging.exception("KeyboardInterrupt" + str(ex))
except CancelledError as ex:
print("Exception CancelledError")
logging.exception("Cancelled Exception" + str(ex))
print("after cleanup")
def main2():
print('Python Client runs using RSA authentication')
if sys.version_info[0] == 3 and sys.version_info[1] >= 8 and sys.platform.startswith('win'):
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
configure = SymConfig('./config.json')
configure.load_config()
auth = SymBotRSAAuth(configure)
auth.authenticate()
bot_client = SymBotClient(auth, configure)
print('Starting datafeed')
datafeed_event_service = bot_client.get_async_datafeed_event_service()

loop = asyncio.get_event_loop()
awaitables = datafeed_event_service.start_datafeed()
loop.run_until_complete(awaitables)
print('Started datafeed')

if name == "main":
configure_logging()
#main2()
try:
asyncio.run(main())
except KeyboardInterrupt:
print("keyboardinterrupt")
print("ending")
`
and Docker / Python 3.8, I'm seeing:
`root@1876ee2dda5a:/opt# python examples/main_async.py
DEBUG:asyncio:Using selector: EpollSelector
Python Client runs using RSA authentication
DEBUG:root:Loading config from: /opt/sym_api_client_python/resources/rsa_config.json
DEBUG:root:RSA Auth/authenticate()
DEBUG:root:RSA Auth/authenticate() --> needed to authenticate
DEBUG:root:RSA_auth/get_session_token() function started
DEBUG:root:RSA_auth/getJWT() function started
DEBUG:root:RSA/session token success
DEBUG:root:RSA_auth/get_keyauth()
DEBUG:root:RSA_auth/getJWT() function started
DEBUG:root:RSA/key manager token success
Starting datafeed
Started datafeed
DEBUG:sym_api_client_python.datafeed_event_service:AsyncDataFeedEventService/start_datafeed()---
DEBUG:sym_api_client_python.services.datafeed_id_repository:Retrieving datafeed id from file /opt/datafeed.id
DEBUG:sym_api_client_python.services.datafeed_id_repository:Retrieved datafeed id: 6af01278-b98a-4b5f-9684-f7972af4697d
DEBUG:root:DataFeedClientV1/read_datafeed_async()
DEBUG:root:bot_client/get_agent_session() - creating async agent session
DEBUG:sym_api_client_python.datafeed_event_service:Waiting on new exception, queue size: 0
^CException CancelledError
ERROR:root:Cancelled Exception
Traceback (most recent call last):
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
self.run_forever()
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
self._run_once()
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once
event_list = self._selector.select(timeout)
File "/usr/local/lib/python3.8/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "examples/main_async.py", line 42, in main
await datafeed
File "/opt/sym_api_client_python/datafeed_event_service.py", line 210, in start_datafeed
await asyncio.gather(self.read_datafeed(), self.handle_events(), self.handle_exceptions())
asyncio.exceptions.CancelledError
after cleanup
keyboardinterrupt
ending
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f9f72980940>`
Should not we have a Cancel request received. Stopping datafeed message?

So using this main
`import asyncio
import logging
import os
import sys
from asyncio import CancelledError
from sym_api_client_python.auth.rsa_auth import SymBotRSAAuth
from sym_api_client_python.clients.sym_bot_client import SymBotClient
from sym_api_client_python.configure.configure import SymConfig
def configure_logging():
logging.basicConfig(
level=logging.DEBUG
)
logging.getLogger("urllib3").setLevel(logging.WARNING)
async def main():
print('Python Client runs using RSA authentication')
if sys.version_info[0] == 3 and sys.version_info[1] >= 8 and sys.platform.startswith('win'):
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
configure = SymConfig(os.path.join(os.path.dirname(file), "..", "sym_api_client_python", "resources", "rsa_config.json"))
configure.load_config()
auth = SymBotRSAAuth(configure)
auth.authenticate()
bot_client = SymBotClient(auth, configure)
datafeed_event_service = bot_client.get_async_datafeed_event_service()
print('Starting datafeed')
datafeed = datafeed_event_service.start_datafeed()
print('Started datafeed')
try:
await datafeed
except KeyboardInterrupt as ex:
print("Exception Keyboard Interrupt")
logging.exception("KeyboardInterrupt" + str(ex))
except CancelledError as ex:
print("Exception CancelledError")
logging.exception("Cancelled Exception" + str(ex))
print("after cleanup")
def main2():
print('Python Client runs using RSA authentication')
if sys.version_info[0] == 3 and sys.version_info[1] >= 8 and sys.platform.startswith('win'):
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
configure = SymConfig('./config.json')
configure.load_config()
auth = SymBotRSAAuth(configure)
auth.authenticate()
bot_client = SymBotClient(auth, configure)
print('Starting datafeed')
datafeed_event_service = bot_client.get_async_datafeed_event_service()

loop = asyncio.get_event_loop()
awaitables = datafeed_event_service.start_datafeed()
loop.run_until_complete(awaitables)
print('Started datafeed')

if name == "main":
configure_logging()
#main2()
try:
asyncio.run(main())
except KeyboardInterrupt:
print("keyboardinterrupt")
print("ending")
`
and Docker / Python 3.8, I'm seeing:
`root@1876ee2dda5a:/opt# python examples/main_async.py
DEBUG:asyncio:Using selector: EpollSelector
Python Client runs using RSA authentication
DEBUG:root:Loading config from: /opt/sym_api_client_python/resources/rsa_config.json
DEBUG:root:RSA Auth/authenticate()
DEBUG:root:RSA Auth/authenticate() --> needed to authenticate
DEBUG:root:RSA_auth/get_session_token() function started
DEBUG:root:RSA_auth/getJWT() function started
DEBUG:root:RSA/session token success
DEBUG:root:RSA_auth/get_keyauth()
DEBUG:root:RSA_auth/getJWT() function started
DEBUG:root:RSA/key manager token success
Starting datafeed
Started datafeed
DEBUG:sym_api_client_python.datafeed_event_service:AsyncDataFeedEventService/start_datafeed()---
DEBUG:sym_api_client_python.services.datafeed_id_repository:Retrieving datafeed id from file /opt/datafeed.id
DEBUG:sym_api_client_python.services.datafeed_id_repository:Retrieved datafeed id: 6af01278-b98a-4b5f-9684-f7972af4697d
DEBUG:root:DataFeedClientV1/read_datafeed_async()
DEBUG:root:bot_client/get_agent_session() - creating async agent session
DEBUG:sym_api_client_python.datafeed_event_service:Waiting on new exception, queue size: 0
^CException CancelledError
ERROR:root:Cancelled Exception
Traceback (most recent call last):
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
self.run_forever()
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
self._run_once()
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once
event_list = self._selector.select(timeout)
File "/usr/local/lib/python3.8/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "examples/main_async.py", line 42, in main
await datafeed
File "/opt/sym_api_client_python/datafeed_event_service.py", line 210, in start_datafeed
await asyncio.gather(self.read_datafeed(), self.handle_events(), self.handle_exceptions())
asyncio.exceptions.CancelledError
after cleanup
keyboardinterrupt
ending
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f9f72980940>`
Should not we have a Cancel request received. Stopping datafeed message?

Follow-up on discussions with @anthony-symphony and @symphony-youness
We noticed a difference between 3.7 and 3.8+ but @anthony-symphony latest change fixes it.
Using Docker I was able to test it with different versions.
@symphony-elias @symphony-youness Ok for you to merge?
(@symphony-elias we should add an .circleci config it is running a build now that we configured it :))

Follow-up on discussions with @anthony-symphony and @symphony-youness
We noticed a difference between 3.7 and 3.8+ but @anthony-symphony latest change fixes it.
Using Docker I was able to test it with different versions.
@symphony-elias @symphony-youness Ok for you to merge?
(@symphony-elias we should add an .circleci config it is running a build now that we configured it :))


none of 6 egl devices matches the active cuda device

hello, I build your code with guide, and complete pip install -e . .
but when i test python tests/square_test.py, I got error:

2019-12-07 12:25:07.943853: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-07 12:25:08.133682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:03:00.0
totalMemory: 31.75GiB freeMemory: 28.20GiB
2019-12-07 12:25:08.486567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 1 with properties:
name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:82:00.0
totalMemory: 31.75GiB freeMemory: 13.53GiB
2019-12-07 12:25:08.919288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 2 with properties:
name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:83:00.0
totalMemory: 31.75GiB freeMemory: 29.09GiB
2019-12-07 12:25:09.085312: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 3 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:02:00.0
totalMemory: 22.38GiB freeMemory: 22.23GiB
2019-12-07 12:25:09.170704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 4 with properties:
name: GeForce GTX 750 Ti major: 5 minor: 0 memoryClockRate(GHz): 1.2545
pciBusID: 0000:01:00.0
totalMemory: 1.95GiB freeMemory: 1.90GiB
2019-12-07 12:25:09.175134: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1469] Ignoring visible gpu device (device: 4, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0) with Cuda multiprocessor count: 5. The minimum required count is 8.  I will be appreciated for your reply as soon as possible, thank you very much!

Could you try running square_test with CUDA_VISIBLE_DEVICES set to select a single GPU -- try with each V100, with the P40, and with the GTX 750, and let me know if it works for any of them.

CUDA_VISIBLE_DEVICES=0 python tests/square_test.py
2019-12-10 12:01:08.061047: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-10 12:01:08.481524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:03:00.0
totalMemory: 31.75GiB freeMemory: 6.81GiB
2019-12-10 12:01:08.481575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2019-12-10 12:01:09.742074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-10 12:01:09.742156: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0
2019-12-10 12:01:09.742166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N
2019-12-10 12:01:09.742380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6544 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-32GB, pci bus id: 0000:03:00.0, compute capability: 7.0)
2019-12-10 12:01:10.120317: F /home/ajl/code/neur/dirt/csrc/gl_common.h:65] none of 6 egl devices matches the active cuda device
[1]    40363 abort (core dumped)  CUDA_VISIBLE_DEVICES=0 python tests/square_test.py

CUDA_VISIBLE_DEVICES=1 python tests/square_test.py
2019-12-10 12:01:39.792989: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-10 12:01:40.478529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:82:00.0
totalMemory: 31.75GiB freeMemory: 13.53GiB
2019-12-10 12:01:40.478581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2019-12-10 12:01:41.718759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-10 12:01:41.718819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0
2019-12-10 12:01:41.718831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N
2019-12-10 12:01:41.719019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 13076 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-32GB, pci bus id: 0000:82:00.0, compute capability: 7.0)
2019-12-10 12:01:42.185089: F /home/ajl/code/neur/dirt/csrc/gl_common.h:65] none of 6 egl devices matches the active cuda device
[1]    40979 abort (core dumped)  CUDA_VISIBLE_DEVICES=1 python tests/square_test.py

CUDA_VISIBLE_DEVICES=2 python tests/square_test.py
2019-12-10 12:01:47.382988: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-10 12:01:47.737285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:83:00.0
totalMemory: 31.75GiB freeMemory: 9.55GiB
2019-12-10 12:01:47.737355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2019-12-10 12:01:48.499910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-10 12:01:48.499973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0
2019-12-10 12:01:48.499985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N
2019-12-10 12:01:48.500327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9212 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-32GB, pci bus id: 0000:83:00.0, compute capability: 7.0)
2019-12-10 12:01:48.785660: F /home/ajl/code/neur/dirt/csrc/gl_common.h:65] none of 6 egl devices matches the active cuda device
[1]    41332 abort (core dumped)  CUDA_VISIBLE_DEVICES=2 python tests/square_test.py

CUDA_VISIBLE_DEVICES=3 python tests/square_test.py
2019-12-10 12:20:52.437749: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-10 12:20:52.710276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:02:00.0
totalMemory: 22.38GiB freeMemory: 22.23GiB
2019-12-10 12:20:52.710325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2019-12-10 12:20:53.484790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-10 12:20:53.484839: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0
2019-12-10 12:20:53.484849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N
2019-12-10 12:20:53.494248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21565 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:02:00.0, compute capability: 6.1)
2019-12-10 12:20:54.133291: F /home/ajl/code/neur/dirt/csrc/gl_common.h:65] none of 6 egl devices matches the active cuda device
[1]    7264 abort (core dumped)  CUDA_VISIBLE_DEVICES=3 python tests/square_test.py

That's all output

This is strange. Could you run nvidia-smi -q | grep 'GPU Operation Mode', and check they're not set to COMPUTE? I don't think it is possible for these GPU models, but worth checking. What version of the nvidia driver are you using?
You could also check what eglQueryDeviceAttribEXT is returning. At csrc/gl_common.h line 60, add std::cout <<  to the start of the line, to have it print the return code.

@pmh47 Thanks for your reply! I have installed successfully with your dockfile. maybe it's my env cause the issue. I will close this issue soon.

@RanFeng Hi, I got exactlly the same problems as u have met. After I sucessfully  built the code, I test
python tests/square_test.py, I got error:

2020-01-09 07:49:36.951439: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-01-09 07:49:37.201253: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55f27ce89c60 executing computations on platform CUDA. Devices:
2020-01-09 07:49:37.201314: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-01-09 07:49:37.205853: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2020-01-09 07:49:37.211061: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55f27d54ddf0 executing computations on platform Host. Devices:
2020-01-09 07:49:37.211100: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): ,
2020-01-09 07:49:37.211475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:02:00.0
totalMemory: 10.92GiB freeMemory: 10.77GiB
2020-01-09 07:49:37.211512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2020-01-09 07:49:37.214396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-09 07:49:37.214428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0
2020-01-09 07:49:37.214440: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N
2020-01-09 07:49:37.214674: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10481 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2020-01-09 07:49:37.528888: F /tmp/pip-req-build-e92skiyt/csrc/gl_common.h:65] none of 1 egl devices matches the active cuda device
Aborted (core dumped)

U said u had installed successfully with the dockfile, but I wonder how to fix the problem rather than use the dockfile(because i am not familiar with docker....). Besides, I run the test in the remote ssh server, I don't know if is this the problem that caused?

I am getting the exact same error message as @Qingcsai -
`I have no name!@2bcd5592e454:~/dirt$ python tests/square_test.py
WARNING:tensorflow:From tests/square_test.py:41: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2020-03-07 23:08:54.864182: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-03-07 23:08:57.912261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:21:00.0
2020-03-07 23:08:57.912894: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-03-07 23:08:57.915643: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-03-07 23:08:57.918103: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-03-07 23:08:57.918696: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-03-07 23:08:57.921856: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-03-07 23:08:57.924259: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-03-07 23:08:57.930714: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-03-07 23:08:57.938051: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-03-07 23:08:57.939114: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-03-07 23:08:57.959327: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2020-03-07 23:08:57.964453: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55605835f450 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-03-07 23:08:57.964538: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-03-07 23:08:58.078474: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556058bc90d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-03-07 23:08:58.078525: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2020-03-07 23:08:58.080621: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:21:00.0
2020-03-07 23:08:58.080679: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-03-07 23:08:58.080727: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-03-07 23:08:58.080768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-03-07 23:08:58.080801: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-03-07 23:08:58.080848: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-03-07 23:08:58.080869: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-03-07 23:08:58.080924: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-03-07 23:08:58.084960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-03-07 23:08:58.085039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-03-07 23:08:58.087362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-07 23:08:58.087391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0
2020-03-07 23:08:58.087447: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N
2020-03-07 23:08:58.091489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30555 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-32GB, pci bus id: 0000:21:00.0, compute capability: 7.0)
2020-03-07 23:08:59.132281: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-03-07 23:08:59.136092: F /tmp/pip-req-build-rzlvw14z/csrc/gl_common.h:65] none of 1 egl devices matches the active cuda device
Aborted (core dumped)
`
We have tried several different combinations of CUDA, CUDNN, Ubuntu, and TensorFlow versions for creating DIRT Docker images and this is the combination that has brought us thus far:
export CUDA_BASE_VERSION=10.0
export CUDNN_VERSION=<IP_ADDRESS>
export UBUNTU_VERSION=18.04
export TENSORFLOW_VERSION=1.15.0
We recently also updated Docker on our server to 19.03.
We have spent several weeks trying to solve this one problem and are kinda desperate at this point because we feel that for our cause, DIRT is the best option out there yet. Any help will be deeply appreciated.


Refactor more specs to use expect_offense

This change focuses on replacing the specs matching the following:

expect(cop.messages).to eq(Array<String>)
expect(cop.highlights).to eq(Array<String>)
expect(cop.offenses.size).to eq(Integer)

as well as patterns I enumerated in #4357

Before submitting the PR make sure the following are checked:

[x] Wrote good commit messages.
[x] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Used the same coding conventions as the rest of the project.
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Added an entry to the Changelog if the new code introduces user-observable changes. See changelog entry format.
[x] All tests are passing.
[x] The new code doesn't generate RuboCop offenses.
[x] The PR relates to only one subject with a clear title
and description in grammatically correct, complete sentences.
[x] Updated cop documentation with rake generate_cops_documentation (required only when you've added a new cop or changed the configuration/documentation of an existing cop).

There are a few long message lines I want to tweak later. Otherwise I think this is semantically correct


Integrate income_statement_ferc1 table

Overview
Closes #1813
This tbl is weird because it has two dbf tabels flowing into one pudl table.

bespoke extract step the concatenates the two dbf tables together
pudl metadata resource and field definitions
row maps for the dbf row numbers (which is 348 of the line changes!)
both the dbf and xbrl table need a reshape (dbf table has columns for each utility_type while the xbrl has multiple columns for each income_type
I enabled align_row_numbers_dbf to take a list of dbf_table_names instead of a single dbf_table_name. This felt a little silly because this many be only one table that ever needs this but it was very easy to implement and felt simpler than having two version of align_row_numbers_dbf
I had to make an overridden version of source_table_id because the Ferc1AbstractTableTransformer assumes there is only one source table. The new version uses the table name that was added during the extract step. We cooooould do this for all of the DBF tables, but we would need to always add the XBRL table name directly from the extract step. That all felt too.. much for one table. The main quirk here is that source_table_id in the main assign_record_id method takes a source_ferc1 and a df. for all of the other tables the df does nada, but the income statement table needs it. The default method as kwargs so this works okay but feels a little weird.

PR Checklist
Before requesting a review of your pull request, please make sure you've done the
following:

[x] Merge the most recent version of dev (or the appropriate upstream branch) into
your branch and resolved any merge conflicts. You may need to do this several
times over the course of a PR as dev changes frequently.
[ ] Verify that all of the CI checks on your PR are passing. See
Running Tests with Tox
for details on how to run the full test suite locally if you need to debug a
particular failure.
[x] Ensure that the docstrings for any new modules, classes, functions, or methods are
descriptive enough for developers and users to understand your code.
[ ] If you expanded data coverage or changed the outputs, ensure that the full
data validation tests
pass locally on a fresh DB.
[x] If you've added new functions or classes, ensure that they have at least basic
unit tests.
[ ] If you've added new analyses, make sure they include defensive sanity checks that
will catch unexpected data issues.
[ ] Update the
release notes
to reflect your changes. Make sure to reference the PR and any related issues.
[x] Do your own review of the PR. Add comments highlighting areas where you have
questions you'd like reviewers to answer, known issues, solutions you're
unsatisfied with, or other things that deserve special attention from the
reviewer.

I guess we never figured out why you were having weird Numpy failures in the CI.  If the builds pass would you bump the max Numpy version back up to <1.25?


Waffle io badge

Coverage remained the same at 85.0% when pulling bf57ef9d0ed182805c55e7a0424e1fd6850d0ade on waffle-io-badge into 6a6d1e347fc6bd5282a0c2224b7d333baa3f330a on master.


how to find files under a directory with helm-browse-project?

This is a question
https://emacs.stackexchange.com/questions/56161/how-to-find-files-under-a-directory-with-helm-browse-project

(helm-browse-project ARG)
Preconfigured helm to browse projects.  Browse files and see status
of project with its vcs.  Only HG and GIT are supported for now.  Fall
back to ‘helm-browse-project-find-files’  if current directory is not
under control of one of those vcs.  With a prefix ARG browse files
recursively, with two prefix ARG  rebuild the cache.

So I tried c-u and helm-browse-project but it finds files for the root project.
Is there a way to find files under a current directory where buffer is in?

Wrong project - you’re looking for emacs-helm. This is the ticket queue for helm, the kubernetes package manager.


[timeseries] Update StatsForecast to v1.6.0

The new version of StatsForecast contains new models and improvements to existing models, so it would be desirable to upgrade.
Currently the upgrade is blocked by a dependency conflict:

omegaconf pins the dependency antlr4-python3-runtime==4.9.*
statsforecast==1.5.0 depends on fugue, which in turn depends on antlr4-python3-runtime>=4.11

omegaconf is required for pytorch_lightnng, so it's a hard requirement for AutoGluon. We need to wait until omegaconf updates its dependencies.
Relevant PR: https://github.com/omry/omegaconf/pull/1114

Completed in #4194


Multiple argument support for DISTINCTCOUNTHLL/DISTINCTCOUNTTHETASKETCH

Currently pinot only supports a single argument for distinct count hll and thetasketch functions. For some use cases like ours where we needed distinct count by two columns and id and a date it is very useful.
https://docs.pinot.apache.org/users/user-guide-query/how-to-handle-unique-counting

One work-around is to concat the two columns into a single string, e.g. select distinctcounthll(concat(id, date, ',')) from myTable. The performance could be sub-optimal because the dictionary/type-based optimization cannot be applied.
To optimize the performance, one solution is to combine multiple values from different columns into a byte[] then feed into the hll/thetasketch. This should give better performance than representing numbers using strings


crypto/tls: add GODEBUG to control max RSA key size

tlsmaxrsasize, See http://go.dev/cl/517495.

@gopherbot please open backport issues.

CL 517495 is submitted. Is there more to do for Go1.22?

I don't believe so, I opened this after the fact in order to create backport issues, so it didn't get closed when the CL was submitted (because it was submitted before the issue existed 😅). Can probably be closed.

Let's close it to help keep issues that are open actionable, then. Thanks.


Bump WordPress and PHP minimums

Bump WP & PHP Minimums to the following

[ ] WordPress to 5.7
[ ] PHP to 7.4
[ ] Remove any no-longer-needed conditional code for older version support.
[ ] Take a look at adjusting our testing matrices, if applicable.
[ ] Bump the minimums in plugin documentation and header fields.

Closed by https://github.com/10up/Open-Source-Best-Practices/pull/91


JSON and vars issues

Great extension. However, I ran into a few issues with it.

I can't use JSON.parse to parse json data. I get an error that JSON is not found.
Cell vars are local? I get an error for the csvString var in my test notebook, even tho I can see csv data ouput:

see this ticket in my data table renderer repo for more info: https://github.com/RandomFractals/vscode-data-table/issues/27

btw, I was able to put together a decent example with this extension: https://github.com/RandomFractals/vscode-data-table#typescript-notebook-example
Would be nice if JSON.parse worked to take it further.

Hmm, i think this is a bug in the jupyter extension, let me check.

Please could you check what version of tslab is installed.
tslab --version.
I am unable to repro the issue

I ran it with the latest tslab v1.0.15

I am unable to repro the issue

I just tried with the latest insiders v. Same results:

Version: 1.59.0-insider (user setup)
Commit: 9520f0a515e7e24802c92f885a0efb242fca3831
Date: 2021-07-14T07:15:41.024Z
Electron: 13.1.6
Chrome: 91.0.4472.124
Node.js: 14.16.0
V8: 9.1.269.36-electron.0

the same results in the latest vscode stable Version: 1.58.1 (user setup)
Commit: 2aeda6b18e13c4f4f9edf6667158a6b8d408874b
Date: 2021-07-13T06:58:28.115Z
Electron: 12.0.13
Chrome: 89.0.4389.128
Node.js: 14.16.0
V8: <IP_ADDRESS>-electron.0
Note that your TS cell does recognize JSON.parse as a valid method:

Ok, I can see similar issues at my end when dealing with arrays.
I think this is a problem with tslab (kernel).

Closing this issue, as this is caused by the tslab kernel. Please report this issue here https://github.com/yunabe/tslab


After a participant has logged in, the only tab that should be active is the readiness checklist. Other tabs may become active after approval from the administrators.

It aint logical to deactivate a participant all menus cause that means the participant can not view previous reports at all and any configurations on his/her side.
However he can only be added to participation while his/her readiness form has been approved


Fix display of variants in Gene Mutation filter box

Description
To reproduce:
Go to Search, choose Gene expression, type gene name (e.g. KRAS), click on Variant.
Observed:

Expected:
a list of variants to choose from or Pick all option
Acceptance criteria

[x] drop-down window displaying variants is  fixed

will check dev for any features, if not will make fix in beta and merge back to dev, and do release to prod

This issue is related to the upgraded Thymeleaf engine. Thymeleaf 3 doesn't allow the usage of th events anymore, these should be defined as attributes.

This is now fixed on dev but the fix cannot be applied to beta/prod until system solves the current VM issue.

Pushed to prod. We have some permission issues with the failover server, so the update didn't make it there.

The issue with permissions on the failover still exists. Contacted support.

closing this, will make a new ticket for the deployment of the failover server


expose the redis instance

expose the redis instance so that we can reuse it for custom data storage.

done

btw, there shouldn't be performance issues associated with using spawning new clients for storage… I mostly exposed them for error handling


Migration failed to apply cleanly to shadow database

Bug description
We're running into an issue with npx prisma migrate dev where it's failing to apply the first migration in our migration history cleanly to the shadow db and failing with the error code
Migration<PHONE_NUMBER>1922_initial_migration failed to apply cleanly to the shadow database.
Error code: P1001
Error:
Can't reach database server at [dev-unwrap-service-db-dbcluster15af587f-nghplpmzwp98.cluster-cjcc6iyom5aw.us-east-2.rds.amazonaws.com](http://dev-unwrap-service-db-dbcluster15af587f-nghplpmzwp98.cluster-cjcc6iyom5aw.us-east-2.rds.amazonaws.com/):3306
Please make sure your database server is running at [dev-unwrap-service-db-dbcluster15af587f-nghplpmzwp98.cluster-cjcc6iyom5aw.us-east-2.rds.amazonaws.com](http://dev-unwrap-service-db-dbcluster15af587f-nghplpmzwp98.cluster-cjcc6iyom5aw.us-east-2.rds.amazonaws.com/):3306.

However it is very clearly able to connect to the db because before this error is thrown it created the db unwrap_v2 and applies all the migrations to the db. I can also go into the DB and see all the tables created. Is there a way in the short term to turn off the shadow db check?
See the full stack trace below.

Error code: P1001
Error:
Can't reach database server at [dev-unwrap-service-db-dbcluster15af587f-nghplpmzwp98.cluster-cjcc6iyom5aw.us-east-2.rds.amazonaws.com](http://dev-unwrap-service-db-dbcluster15af587f-nghplpmzwp98.cluster-cjcc6iyom5aw.us-east-2.rds.amazonaws.com/):3306
Please make sure your database server is running at [dev-unwrap-service-db-dbcluster15af587f-nghplpmzwp98.cluster-cjcc6iyom5aw.us-east-2.rds.amazonaws.com](http://dev-unwrap-service-db-dbcluster15af587f-nghplpmzwp98.cluster-cjcc6iyom5aw.us-east-2.rds.amazonaws.com/):3306.

How to reproduce
I'm not actually sure how to reproduce. I can only get this to happen on two of our dev db's. This is verified on a fresh db however, we dropped the database unwrap_v2 as well as all shadow databases. It happens reproducibly on two of our dbs.
Expected behavior
It should just create and apply the migration file
Prisma information

// Add your schema.prisma

// Add your code using Prisma Client

Environment & setup

OS: macOs
Database: MySQL
Node.js version: 20.12.1

Prisma Version
5.0.0

Is there a way in the short term to turn off the shadow db check?

No, it is not only a "shadow database check" but actually how Prisma figures out which SQL has to be executed to bring your database into the desired state. You can of course avoid that by not using migrate dev and creating the migration file by hand - but I assume that would defeat the purpose.
The real question here is why this is failing on some of your database servers. Any idea what could be configured differently there to cause a migration to fail? Can you maybe enable query logging on the database level, and see what is being executed under the hood - and where it fails with which error? Seems Can't reach database server at ... is a misleading error message on our side which somehow is triggered when something else goes wrong.

Unfortunately not for migrations, sorry.

Ok so did a little investigation here as this is becoming a bigger problem. I see that what's happening in the error log is I'm getting dropped connections.
error log
2024-06-11T17:09:45.784337Z 7118 [Note] [MY-010914] [Server] Aborted connection 7118 to db: 'unwrap_v2' user: 'admin' host: '<IP_ADDRESS>' (Got an error reading communication packets). (sql_connect.cc:828)
2024-06-11T17:11:54.424896Z 7120 [Note] [MY-010914] [Server] Aborted connection 7120 to db: 'prisma_migrate_shadow_db_35e045c5-4582-4547-aac0-11546b2486f4' user: 'admin' host: '<IP_ADDRESS>' (Got an error reading communication packets). (sql_connect.cc:828)
2024-06-11T17:12:10.307386Z 7119 [Note] [MY-010914] [Server] Aborted connection 7119 to db: 'unwrap_v2' user: 'admin' host: '<IP_ADDRESS>' (Got an error reading communication packets). (sql_connect.cc:828)
2024-06-11T17:14:37.623637Z 7146 [Note] [MY-010914] [Server] Aborted connection 7146 to db: 'unwrap_v2' user: 'admin' host: '<IP_ADDRESS>' (Got an error reading communication packets). (sql_connect.cc:828)
2024-06-11T17:16:42.727038Z 7148 [Note] [MY-010914] [Server] Aborted connection 7148 to db: 'prisma_migrate_shadow_db_3e1ed3f6-dca7-46df-b202-5d0ba119847a' user: 'admin' host: '<IP_ADDRESS>' (Got an error reading communication packets). (sql_connect.cc:828)
2024-06-11T17:16:56.823406Z 7147 [Note] [MY-010914] [Server] Aborted connection 7147 to db: 'unwrap_v2' user: 'admin' host: '<IP_ADDRESS>' (Got an error reading communication packets). (sql_connect.cc:828)

I look at RDS performance insights, our dev db's run in AWS RDS on very small machines, and I see that the migrate dev operation is smoking the CPU with operations that look like the shadow db migration (see screenshot).

We have 190 migrations in our migrations directory, is this too many? Anecdotally it seems like this problem has grown with our DB schema. migrate dev used to run very quickly and now takes forever 1-2 min to run and intermittently fails with this error mentioned above.

Interesting, it seems that this database is not up to the task to apply these 190 migrations in the shadow database regularly. That is the first time this has been reported to us. Can you test out if a bigger database has less trouble applying them?
If you do not need the migrations any more, it might be worth squashing them all together: https://www.prisma.io/docs/orm/prisma-migrate/workflows/squashing-migrations#how-to-create-a-clean-history-in-a-production-environment


1.21 Support?

See https://github.com/DuncanRuns/Chunkumulator/issues/2

i tried launching it on my server but it crashed, is there something I'm doing wrong?
 A potential solution has been determined, this may resolve your problem:          - Replace mod 'Chunkumulator' (chunkumulator) 1.0.3+MC1.20.1 with any version that is compatible with:                  - minecraft 1.21 More details:          - Mod 'Chunkumulator' (chunkumulator) 1.0.3+MC1.20.1 requires any version between 1.20.1 (inclusive) and 1.21- (exclusive) of 'Minecraft' (minecraft), but only the wrong version is present: 1.21!         at net.fabricmc.loader.impl.FormattedException.ofLocalized(FormattedException.java:51) ~[fabric-loader-0.15.11.jar:?]         at net.fabricmc.loader.impl.FabricLoaderImpl.load(FabricLoaderImpl.java:195) ~[fabric-loader-0.15.11.jar:?]         at net.fabricmc.loader.impl.launch.knot.Knot.init(Knot.java:146) ~[fabric-loader-0.15.11.jar:?]         at net.fabricmc.loader.impl.launch.knot.Knot.launch(Knot.java:68) [fabric-loader-0.15.11.jar:?]         at net.fabricmc.loader.impl.launch.knot.KnotServer.main(KnotServer.java:23) [fabric-loader-0.15.11.jar:?]         at net.fabricmc.loader.impl.launch.server.FabricServerLauncher.main(FabricServerLauncher.java:69) [fabric-loader-0.15.11.jar:?]         at net.fabricmc.installer.ServerLauncher.main(ServerLauncher.java:69) [server.jar:1.0.1]

Ok
Remove the mod
It is not needed past 1.20.1

alr


Create KFServing Python SDK

What this PR does / why we need it:
The PR is to create KFServing Python SDK, according to instructions from @yuzisun in the #160, and you can see the practice in the hack/sdk-gen/README.md in the PR.
Which issue(s) this PR fixes :
Fixes #160
Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

Created KFServing Python SDK

This change is

/assign @yuzisun
/retest

Hey this is pretty cool. Do you mind separating the CLs from generated code and from the code that generates it. It will make this easier to review.

@ellis-bigelow the openapi-gen (already in vendor) and swagger-codegen is ignored before checkin, in fact all commits are generated files. And do you think (or mean) to separate openapi model (openapi-gen generated) and  and python sdk (swagger-codegen generated)? If so, we may can do that like TFJob done in update-codegen.sh.

/hold

Added some input-dirs for openapi-gen, otherwise, there will be import error during tests. Thanks.

Awesome work!  Looks the core kubernetes python object model are already here  https://github.com/kubernetes-client/python/tree/master/kubernetes/client/models, I guess we can just add this dependency to our sdk?

Hey this is pretty cool. Do you mind separating the CLs from generated code and from the code that generates it. It will make this easier to review.

@ellis-bigelow the openapi-gen (already in vendor) and swagger-codegen is ignored before checkin, in fact all commits are generated files. And do you think (or mean) to separate openapi model (openapi-gen generated) and and python sdk (swagger-codegen generated)? If so, we may can do that like TFJob done in update-codegen.sh.

@ellis-bigelow I am quite familiar with all the generations, it is ok to leave as is to review this time :).

Awesome work! Looks the core kubernetes python object model are already here https://github.com/kubernetes-client/python/tree/master/kubernetes/client/models, I guess we can just add this dependency to our sdk?

@yuzisun It's really appreciated for your great comments!  For the dependency, yes, kubernetes python clients is exists , but not sure we can depends on that, but will have more investigation for this. Thanks.

@yuzisun Thanks for your mentoring. I updated the code changed, removed the k8s existing stuff, and used the "importMappings"  to set dependency from k8s in swagger_config.json file.
Here it's worth mentioning that, for io_k8s_apimachinery_pkg_apis_meta_v1_time.py that's called by knative, I did not get that from official k8s python client repo, so added in this PR.
Sample demo for the SDK here.
(python36) [root@jinchi1 sdk]# cat mytest.py
from kubernetes import client

from kfserving.models.kfserving_v1alpha1_model_spec import KfservingV1alpha1ModelSpec as V1alpha1ModelSpec
from kfserving.models.kfserving_v1alpha1_tensorflow_spec import KfservingV1alpha1TensorflowSpec as V1alpha1TensorflowSpec
from kfserving.models.kfserving_v1alpha1_kf_service import KfservingV1alpha1KFService as V1alpha1KFService
from kfserving.models.kfserving_v1alpha1_kf_service_spec import KfservingV1alpha1KFServiceSpec as V1alpha1KFServiceSpec

model_spec =  V1alpha1ModelSpec(tensorflow=V1alpha1TensorflowSpec(model_uri='gs://flower'))
kfsvc = V1alpha1KFService(metadata=client.V1ObjectMeta(name='flower'),spec= V1alpha1KFServiceSpec(default=model_spec))

print(kfsvc)

Executing the sample python script:
(python36) [root@jinchi1 sdk]# python mytest.py
{'api_version': None,
 'kind': None,
 'metadata': {'annotations': None,
              'cluster_name': None,
              'creation_timestamp': None,
              'deletion_grace_period_seconds': None,
              'deletion_timestamp': None,
              'finalizers': None,
              'generate_name': None,
              'generation': None,
              'initializers': None,
              'labels': None,
              'name': 'flower',
              'namespace': None,
              'owner_references': None,
              'resource_version': None,
              'self_link': None,
              'uid': None},
 'spec': {'canary': None,
          'canary_traffic_percent': None,
          'default': {'custom': None,
                      'max_replicas': None,
                      'min_replicas': None,
                      'pytorch': None,
                      'service_account_name': None,
                      'sklearn': None,
                      'tensorflow': {'model_uri': 'gs://flower',
                                     'resources': None,
                                     'runtime_version': None},
                      'tensorrt': None,
                      'xgboost': None}},
 'status': None}
(python36) [root@jinchi1 sdk]#

/hold cancel

Seems the kubeflow-kfserving-presubmit testing env has problem, will retest once the env ready. Thanks.

Great work! This looks pretty good to me now, a few final comments.

@ellis-bigelow Do you have more comments on this?

I'm good if you are.

/lgtm
/approve

/retest


React Select MUI - not updating css in part of text in options

Duplicates

[X] I have searched the existing issues

Latest version

[X] I have tested the latest version

Steps to reproduce 🕹
Link to live example: https://react-apkfhd.stackblitz.io
Link to my code :- https://stackblitz.com/edit/react-apkfhd?file=Demo.tsx
Current behavior 😯
i have added css for span under options element like below

              {name}   // this is name
              plus // this is tag which i want to apply bold

no matter what approach i follow or how i inject css its not applying to span
for more context i have provided link to my demo code can follow there
Expected behavior 🤔
"plus" which is text in span element should apply all required css from customSpan
Context 🔦
so i have list of options with tags like Oliver Hansen plus
here Oliver Hansen is name and plus is tag for that.

  Output from `npx @mui/envinfo` goes here.

@faizansayyed In your example, you are using @mui/styles which is originally for Material UI v4, and is not compatible with React 18 above (our docs about this)
Check out the docs here on how to customize Material UI (v5) components


Missing --line-length in the command line API

At the command line flake8 exposes two line length settings:
$ flake8 --help | grep length
  --max-line-length n   Maximum allowed line length for the entirety of this
  --max-doc-length n    Maximum allowed doc line length for the entirety of

These control E501 (already in ruff) and W505 (also from pycodestyle but off by default in flake8 - see https://pycodestyle.pycqa.org/en/latest/intro.html#error-codes - not yet in ruff) respectively.
See also https://github.com/PyCQA/flake8-bugbear/issues/285 for a discussion about what flake8-bugbear B950 ought to obey.
Currently ruff has nothing like this at the command line:
$ ruff --help | grep length

However ruff does have a line-length configuration setting:
https://github.com/charliermarsh/ruff#configuration
I would like you to add this to the ruff command line, e.g. as --line-length or if you want to match flake8 and/or add W505 support later, --max-line-length

Good call! I’ll add this today. I actually thought it already existed but clearly not!


Istio release 0.2.6

Updated Istio chart to support release 0.2.6
The PR is all but done but need an assist on the following:

Upstream fix for the initializer installation

Is apiextensions beta in 1.7.x?

@viglesiasce
$ kubectl api-versions | grep apiextensions
apiextensions.k8s.io/v1beta1
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c", GitTreeState:"clean", BuildDate:"2017-09-15T08:51:09Z", GoVersion:"go1.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T08:56:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

@gyliu513 I took a look through the initializer source and it appears to ignore anything with the following annotation set so we should be covered. Feel free to test and let me know
sidecar.istio.io/inject: "false"

/retest

@lachie83 What I concerned is sometimes, you may found that you cannot deploy applications and all applications will be pending for about 30s and then timeout. For such kind of issue, it is usually caused by the initializerConfig if the sidecar initializer does not work. Please check out one of my blog post here for detail https://github.com/gyliu513/jay-work/blob/master/k8s/example/kube-initializer-tutorial/medium-article.md#troubleshooting
No worry, I do hope that we can merge your PR first and I can follow up with other issues ;-)

@lachie83 0.2.7 is released https://github.com/istio/istio/releases/tag/0.2.7 :-) Hope we can have 0.2.6 soon!

@gyliu513 I've confirmed that 0.2.7 is just an image bump for now...


Access pretrained weights and test model with different image size

Hi, how we van access pretrained weights. Is this possible to use image with different sizes for test the model?
Do you prepare any google colab demo for project?
@Shuweis

Hi, thank you for your attention. We will release the MANIQA weights several days later. For different image size, you can crop 224*224 with multiple times and average the score to get the last result. We will prepare the google colab demo in the future.

Hi, how we van access pretrained weights. Is this possible to use image with different sizes for test the model? Do you prepare any google colab demo for project? @Shuweis

Hi，Our code includes the timm library. When you train your code, the timm library automatically loads VIT's pretrained models.

Hi, how we van access pretrained weights. Is this possible to use image with different sizes for test the model? Do you prepare any google colab demo for project? @Shuweis

Hi，Our code includes the timm library. When you train your code, the timm library automatically loads VIT's pretrained models.

I want to use the model for inference without training. Could you please share the weight of the model?

Hi, how we van access pretrained weights. Is this possible to use image with different sizes for test the model? Do you prepare any google colab demo for project? @Shuweis

Hi，Our code includes the timm library. When you train your code, the timm library automatically loads VIT's pretrained models.

I want to use the model for inference without training. Could you please share the weight of the model?

Hi, we have already released the checkpoint. You can use the model for inference PIPAL22 Validation.
Link: https://github.com/IIGROUP/MANIQA/releases/tag/PIPAL22-VALID-CKPT

I can't use this files you mentioned above for run inference file. How I can run inference file without training with pretrained weight?
@TianheWu


Implement ODATA

Implement ODATA v4 standard to query.

Hi,
Great library!
Just wondering if there are any plans to implement this in the near future?
Thanks!


Add documentation in README

Resolves #6

@parisk

Should we keep the Glossary section in the README?
Do you think that we should keep each function signature?
Should we also include our generated documentation? e.g. descriptions for parameters etc