Compilation of light example fails with "ModuleNotFoundError: No module named 'lark'" (CON-252)

Hi,
I've issues during compilation and linking of the examples!
Following steps to reproduce:

fresh checked out repos esp-matterand esp-idf(v4.4.3)
./install.sh in both repos
activatation of both environments (first esp-idf, second esp-matter)

cd esp-matter/examples/ligth
idf.py set-target esp32c3
idy.py build

fails with following message trace:
[46/729] ACTION //third_party/connectedhomeip/src/controller/data_model:data_model_codegen_codegen(//third_party/connectedhomeip/config/esp32/toolchain:esp32)
FAILED: gen/third_party/connectedhomeip/src/controller/data_model/app/PluginApplicationCallbacks.h gen/third_party/connectedhomeip/src/controller/data_model/app/callback-stub.cpp
python ../../../../../connectedhomeip/connectedhomeip/config/esp32/third_party/connectedhomeip/third_party/pigweed/repo/pw_build/py/pw_build/python_runner.py --gn-root ../../../../../connectedhomeip/connectedhomeip/config/esp32/ --current-path ../../../../../connectedhomeip/connectedhomeip/config/esp32/third_party/connectedhomeip/src/controller/data_model --default-toolchain=//third_party/connectedhomeip/config/esp32/toolchain:esp32 --current-toolchain=//third_party/connectedhomeip/config/esp32/toolchain:esp32 --capture-output --python-dep-list-files gen/third_party/connectedhomeip/src/controller/data_model/data_model_codegen_codegen_metadata_path_list.txt -- ../../../../../connectedhomeip/connectedhomeip/config/esp32/third_party/connectedhomeip/scripts/codegen.py --generator cpp-app --output-dir gen/third_party/connectedhomeip/src/controller/data_model --expected-outputs gen/third_party/connectedhomeip/src/controller/data_model/data_model_codegen.expected.outputs ../../../../../connectedhomeip/connectedhomeip/config/esp32/third_party/connectedhomeip/src/controller/data_model/controller-clusters.matter
Traceback (most recent call last):
  File "/media/psf/Home/VM_Workspaces/Ubuntu_Matter_ESP/esp/esp-matter/examples/light/build/esp-idf/chip/../../../../../connectedhomeip/connectedhomeip/config/esp32/third_party/connectedhomeip/scripts/codegen.py", line 28, in <module>
    from idl.matter_idl_parser import CreateParser
  File "/media/psf/Home/VM_Workspaces/Ubuntu_Matter_ESP/esp/esp-matter/connectedhomeip/connectedhomeip/scripts/idl/matter_idl_parser.py", line 7, in <module>
    from lark import Lark
ModuleNotFoundError: No module named 'lark'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/media/psf/Home/VM_Workspaces/Ubuntu_Matter_ESP/esp/esp-matter/examples/light/build/esp-idf/chip/../../../../../connectedhomeip/connectedhomeip/config/esp32/third_party/connectedhomeip/scripts/codegen.py", line 32, in <module>
    from idl.matter_idl_parser import CreateParser
  File "/media/psf/Home/VM_Workspaces/Ubuntu_Matter_ESP/esp/esp-matter/connectedhomeip/connectedhomeip/scripts/idl/matter_idl_parser.py", line 7, in <module>
    from lark import Lark
ModuleNotFoundError: No module named 'lark'

Full log is attached here full_log_light_example.log
Anybody an idea what happens?

Please following these steps to set up the python environment.

install and export IDF python environment by cd esp-idf; ./install.sh; . ./export.sh
install and export esp_matter requirement by cd esp-matter; ./install.sh; . ./export.sh

Then the error should be fixed.

Please following these steps to set up the python environment.
1. install and export IDF python environment by `cd esp-idf; ./install.sh; . ./export.sh`

2. install and export esp_matter requirement by `cd esp-matter; ./install.sh; . ./export.sh`

Then the error should be fixed.

Sorry that's not the solution for my issue. Of course I activate the environments after installation process.

Could you please capture the log of running ./install.sh in esp-matter directory? This will help understand why the lark package didn't get installed.
(I got a similar issue recently, and in my case it was caused by the failure on ./install.sh due to a missing prerequisite. I haven't noticed that the script has failed and proceeded to the next step.)

Applications are built in idf python environment, so we have to run source esp-idf/export.sh before executing esp-matter/install.sh. This installs all the python dependencies in idf python environment.

Documentation does mention it but in getting esp-if section. It should have been into getting esp-matter section.

Applications are built in idf python environment, so we have to run source esp-idf/export.sh before executing esp-matter/install.sh. This installs all the python dependencies in idf python environment.

Okay, that could be my problem. I follow the instructions on the esp documentation site (you've linked it already).
And also as you said, the installation steps are not correct.

I had the same error while compiling yesterday, i fixed the error by installing Lark manually.
pip3 install lark

Applications are built in idf python environment, so we have to run source esp-idf/export.sh before executing esp-matter/install.sh. This installs all the python dependencies in idf python environment.

That was definitely the solution for my issue!


Revert "Bump org.apache.derby:derby from <IP_ADDRESS> to <IP_ADDRESS> in /hadoop-project"

Reverts apache/hadoop#6816
We have founed some HDFS RBF errors.  We can refer to this issue https://issues.apache.org/jira/browse/HDFS-17533.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m  0s

Docker mode activated.

-1 :x:
patch
0m 19s

https://github.com/apache/hadoop/pull/6841 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/6841

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6841/1/console

versions
git=2.34.1

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


Crossgen is silently failing in CI

While testing crossgen component on Alpine, I found there was one error which is in CI as well as in Alpine:
/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/Tools/csc.dll -> /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/Tools/csc.ni.dll
Error compiling /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/Tools/Microsoft.DotNet.Build.Tasks.dll: Could not load file or assembly 'Microsoft.Build.Framework, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
 (Exception from HRESULT: 0x80070002)
Error: file "/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/Tools/Microsoft.DotNet.Build.Tasks.dll" or one of its dependencies was not found
Unable to successfully compile /Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/Tools/Microsoft.DotNet.Build.Tasks.dll

Is this something that can fix in this repo or would we the tools repo need to be updated?

This is coming from buildtools. I do not think you can fix this in this repo.

(Also, this failure can be ignored. It just means that the msbuild will do a bit more JITing.)

Thanks, I have opened https://github.com/dotnet/buildtools/issues/2073 to silence this error.


Update the incomplete steps to add a custom header column

Source:
https://devtoolstips.org/tips/en/custom-headers-in-network-table/
AB#45387340

Thank you @pankajparashar for submitting this PR. I'll make sure we review and merge this soon.

offline, i requested merging this PR

All merged and published to the live branch. Should appear on the live site any time now. Thank you!


Fix the sourceType parameter

As a side note, it fixes also the issue mentioned in https://github.com/danielstjules/jsinspect/issues/51.

Thanks! :) Will publish a patch release.


Revert "[RP-1135] Temp Disable Publisher Worker on Staging"

Prepared draft PR to revert this once testing is completed.
Reverts alphagov/govuk-helm-charts#982

Going to roll this back for the weekend.


Cannot read property 'COLUMN_NAME' of undefined

When calling columnInfo with a table and column name, if the search query returns undefined the function throws Cannot read property 'COLUMN_NAME' of undefined when mapping the response to Column.
This can be easily fixed, but it feels like we should consider what return type we want in this scenario. Personally I think columnInfo should return Column | undefined in this instance, but this would be a breaking change.

Should we return null/undefined, or throw an error that the table/column name is invalid? In the last case, it wouldn't be a breaking change

I personally feel like this function should return undefined, which lines up more with how other knex functions operate. But I also don't think it's worth doing a major increment just to make things a bit nicer! So probably best to just throw an error for now.


Enable plugin menu contributions with app-model (v2)

Description
This is an alternative to #4844 ... it implements adding plugin menu items to our menus.
The difference here is that rather than using custom "PluginAware" registries that keep plugin and napari contributions fundamentally separate, this one just directly adds/removes plugin items from the app model registries as those plugins are
enabled/disabled.
I think  I prefer this version to #4844...
As with all the other app model PRs, this includes #4784.  here's the diff for just this PR (~130 lines)
https://github.com/tlambert03/napari/compare/use-app-model...tlambert03:napari:appmodel-plugin-menu2
Type of change

[ ] Bug-fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

References

How has this been tested?

[ ] example: the test suite for my feature covers cases x, y, and z
[ ] example: all tests pass with my change
[ ] example: I check if my changes works with both PySide and PyQt backends
as there are small differences between the two Qt bindings.

Final checklist:

[ ] My PR is the minimum possible work for the desired functionality
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] If I included new strings, I have used trans. to make them localizable.
For more information see our translations guide.

This feels like a nice, simple set of changes - I don't feel like I have enough context to have an opinion on whether napari should know whether a contribution came from a plugin. Am I correct in thinking that with this implementation we cannot differentiate?
Is there any existing discussion on this issue specifically? I would like to better understand if this might be a limitation

I don't feel like I have enough context to have an opinion on whether napari should know whether a contribution came from a plugin.

my feeling is that yes, absolutely we need to know.

Am I correct in thinking that with this implementation we cannot differentiate?

yes and no.  since all plugins must start their command ids with their package name, we kinda know already.  However, another more explicit approach that I think I prefer is to add something like a source attribute to each of the "registered" objects in app-model.  That is each of the menu, command, and keybinding registries would add a source attribute to their corresponding RegisteredX() objects, which would be populated at registration time.

i should add here that this PR requires this merged (but yet to be released) PR from npe2: https://github.com/napari/npe2/pull/210

However, another more explicit approach that I think I prefer is to add something like a source attribute to each of the "registered" objects in app-model. That is each of the menu, command, and keybinding registries would add a source attribute to their corresponding RegisteredX() objects, which would be populated at registration time.

Love the sound of this! 👍🏼

The difference here is that rather than using custom "PluginAware" registries that keep plugin and napari contributions fundamentally separate, this one just directly adds/removes plugin items from the app model registries as those plugins are
enabled/disabled.

I also prefer this to #4844 as it seems simpler


Upgrading proxy to new contract no longer seems to work

It appears that upgrading a proxy contract's implementation to a different contract no longer works as before since beta.53. Previously, you would be able to:
// Deploy a proxied `SomeContract`
await deploy("SomeContract, { proxy: true, /* ... */ });
// Upgrade the proxied implementation to the new `SomeContractV2`
await deploy("SomeContract", { contract: "SomeContractV2", proxy: true, /* ... */ });

However, this no longer seems to work. In looking through the code it appears to be caused by the fact that, for the second deployment, there already is a "oldDeployment", which causes the following branch to get executed after deploying the new implementation:
https://github.com/wighawag/hardhat-deploy/blob/b4a8037c6d687cb736b960e0f96148839142fae7/src/helpers.ts#L1132-L1148
This branch just updates and saves the deployment so the implementation points to the newly deployed implementation address, but doesn't seem to execute any transactions for actually setting the proxy's implementation on-chain (which only happens if there is oldDeployment is falsy, so unset).
This also leads me to believe that it may not work for contracts with existing proxy deployments (regardless of the different contract name), but I may be wrong and did not test this hypothesis.

Thanks for the issue, I indeed introduce the bug recently.
the issue is at this line : https://github.com/wighawag/hardhat-deploy/blame/b4a8037c6d687cb736b960e0f96148839142fae7/src/helpers.ts#L984
sorry for the inconvenience, going to fix that and report back

sorry for the inconvenience

No need to apologize! Thanks for the awesome plugin :smile:.

should be fixed in 0.7.0-beta.55
Feel free to reopen if there are still issues

@wighawag Sorry, I think there might still be an issue :see_no_evil: (I can't reopen the issue for some reason, I hope the ping is enough).
I think this:
https://github.com/wighawag/hardhat-deploy/blob/4c08e03410067eaa576c43bd025e72f546823059/src/helpers.ts#L1085-L1090
should be changed to
            executeReceipt = await execute(
              proxyName,
              {...options, from: currentOwner},
              'upgradeTo',
              implementation.address
            );

As the proxy function is called upgradeTo (upgrade is the proxy admin function, which isn't being used in this branch):
https://github.com/wighawag/hardhat-deploy/blob/4c08e03410067eaa576c43bd025e72f546823059/solc_0.7/proxy/EIP173Proxy.sol#L61-L63

You are right, thanks! going to fix that.

should be fixed in 0.7.0-beta.57

Works like a charm! :rocket: Thanks!

@nlordell @wighawag (thanks for this great plugin) I read all of the issues and the readme many times, everything works great but I just cannot understand how to do an upgrade, can you please post example scripts of how to deploy a proxy then upgrade it? I am going crazy here. Thank you

@Gutyn with hardhat-deploy upgrades are handled automaticaly
You just declare the new state you want to be in with the same deploy call you used originally, hardhat-deploy will detect the change of bytecode and make the upgrade


authentication timeout after successful login - ignoring appCookie.ExpireTimeSpan property

Apologies if this is the incorrect process for logging issues but I've asked this question on SO: http://stackoverflow.com/questions/35695975/asp-net-mvc-core-and-identity-3 and had no answers
Everything appears setup ok but after a certain amount of time I'm logged out.
I have the cookie timeout set to 8 hours and SlidingExpiration is set to true but it logs me out way before 8 hours, its something like 20-30mins after it kicks me out
appCookie.SlidingExpiration = true;
appCookie.ExpireTimeSpan = TimeSpan.FromHours(8);
Does it have something to do with session at all?

Make sure you are configuring the cookie via identityOptions and not any other way, can you include the startup code that you are using to set the cookie options?

Please reopen if this is still a problem and can attach the relevant startup...

I do not know if this is still an issue but I noticed when I used
ApplicationCookie.ExpireTimeSpan = TimeSpan.FromMinutes(5)
And cookie has never been created (or shown in browser) and I also get not "logged in"
As I have UTC + 2 (Berlin) timezone, I tried DateTime.Now.Subtract(DateTime.UtcNow).Add( TimeSpan.FromMinutes(5.0) ); which makes the TimeSpan to 02h05m. But in Chrome browser using "EditCookie" it shows 5 minutes difference only and login worked

One note: it seems that ExpireTimeSpan and other options work well only if you set RememberMe parameter to true during SignIn... call.


Missing "Created From" property in Get-SPOSite

Hello,
I realized in SharePoint Admin Center there is a field called "Created From" and i am unable to find an equivalent command that will help me output how the site was created. Is that feature not available in the SharePoint PowerShell Module?

Hello @potluri96
Currently Get-SPOSite does not return this information.
Thanks

@get-itips Thank you very much for the contribution and sharing this explanation. @potluri96 Hope this comment is helpful for you. If you see a documentation update is required, please feel free to open an issue for the same. We proceed here to close it. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


Can WAS not be installed on arm64?

It would be preferable for me to install WAS on a little Ubuntu Mate Pi4 server I have. When I run the docker command it barks that the architecture doesn't match amd64, and the container exits. Is there any way I can get this on here, or do I absolutely need it running on an amd64 machine?

Can you try the command below and let us know if it works for you?
docker run --detach --name=willow-application-server --pull=always --network=host --restart=unless-stopped --volume=was-storage:/app/storage ghcr.io/toverainc/willow-application-server:improvements

@kristiankielhofner discovered this is still not enough to run on Raspberry Pi OS with 64 bit kernel and 32 bit userspace. Unfortunately it is non-trivial to build WAS for 32 bit ARM userspace, as there don't seem to be any numpy wheels available for that. We're sure it can be done, but right now we have other priorities, so if you run in the following error we suggest to switch to either build yourself or switch to a full 64 bit OS:
docker: no matching manifest for linux/arm/v8 in the manifest list entries.

Can you try the command below and let us know if it works for you?
docker run --detach --name=willow-application-server --pull=always --network=host --restart=unless-stopped --volume=was-storage:/app/storage ghcr.io/toverainc/willow-application-server:improvements

I didn't realize this got close so quickly. I haven't had a chance to log back in. My OS is 64-bit, so you think it would work? I'm going to try it this evening.

Yes, arm64 works (tested on Raspberry Pi 4 and Nvidia Jetson).

Yes, arm64 works (tested on Raspberry Pi 4 and Nvidia Jetson) as long as you're using a 64 bit OS.

It's working. 👍


Use this library with meshes

Hi there,
I have around 20 meshes with img maps as a texture. All of them are a plane.
Could I use this library to detect intersections of the mouse as it moves, with all these meshes?
I made it work, but there is no way for me to find the Object that is "under" the mouse when an intersection happens.
I opened a question on Stackoverflow, and I come here asking for kind help because I have been stuck for days.
Thanks a lot,
https://stackoverflow.com/questions/48691642/three-js-raycaster-find-intersections-as-mouse-moves/48693593?noredirect=1#comment84413010_48693593

hi @Richacinas can you show me what the intersect = gpuPicker.pick(mouse, raycaster); returns? also I haven't updated the library with latest THREEjs so it may not work if you are using the latest version.

Hi again,
Yes, I was receiving data on the property object, however, it wasn't correct many times as it was giving me a different object... It looks like the positions are not right so maybe the problem was about that.
All my elements are inside a sphere where I'm rendering a texture.. it is like a 360 tour inside a famous building.
I don't think I can put a demo online because it would take me a long time.. If I can do it, I'll post the url here...

I created a demo:
https://jsfiddle.net/ggL3uemh/13/
On this demo, I'm not using your library. This is the way we are doing it right now, so hopefully you can take a look and find out how we could integrate with your library.
There is some code that is aimed to render a video texture on a sphere geometry (tex2, renderVideo, addVideoTexture...) so don't pay attention to that because I am unable to render the video texture from another host (cross origin).
Any help would be greatly appreciated,
Thanks a lot,
Ricardo

have you tried to turn on the debug view like the example? it will show you the encoded rendering

I have updated the library for THREEjs r92, hopefully it works for you now.

Thanks a lot. Eventually I will try. I couldn't solve this issue but I'm sure something was missing. Ended up using raycast + octrees and it improved performance and worked


add fields json.RawMessage in SearchHit

Description
Add fields fields in SearchHit for support mapping response with fields key
Issues Resolved
For support retrieve specific fields in the search response
Request
{
  "query": {
    "match": {
      "username": "thekogo"
    }
  },
  "fields": [
    "username",
    "email"
  ],
  "_source": false
}

Response (add mapping for support this response)
{
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "index-001",
        "_id" : "0",
        "_score" : 1.0,
        "fields" : {
          "username" : [
            "thekogo"
          ],
          "email" : [
<EMAIL_ADDRESS>          ]
        }
      }
    ]
  }
}

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Can you please signoff your commits? You can amend your commits using `git commit --amend -s" to sign them off. https://github.com/opensearch-project/opensearch-go/blob/main/CONTRIBUTING.md#developer-certificate-of-origin

Sorry I'm not sure why Lint check / Prettify (pull_request) error with
Run npm i -g npm@8 && npm i -g prettier
npm ERR! code EACCES
npm ERR! syscall mkdir
npm ERR! path /usr/local/share/man/man5
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, mkdir '/usr/local/share/man/man5'
npm ERR!  [Error: EACCES: permission denied, mkdir '/usr/local/share/man/man5'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'mkdir',
npm ERR!   path: '/usr/local/share/man/man5'
npm ERR! }
npm ERR!
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR!
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.

npm ERR! A complete log of this run can be found in: /home/runner/.npm/_logs/2024-04-04T18_42_12_[7](https://github.com/opensearch-project/opensearch-go/actions/runs/8559713019/job/23457055244?pr=508#step:3:8)53Z-debug-0.log
Error: Process completed with exit code 243.


Web: adding support for computeMode to App Service plans

This checklist is used to make sure that common issues in a pull request are addressed. This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
PR information

[x] The title of the PR is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
[x] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
[ ] If applicable, the PR references the bug/issue that it fixes.
[ ] Swagger files are correctly named (e.g. the api-version in the path should match the api-version in the spec).

Quality of Swagger

[x] I have read the contribution guidelines.
[x] My spec meets the review criteria:

[x] The spec conforms to the Swagger 2.0 specification.
[x] The spec follows the guidelines described in the Swagger checklist document.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.

I've based this on this ARM Template: https://github.com/Azure/azure-quickstart-templates/blob/master/101-function-app-create-dynamic/azuredeploy.json#L48 - which appears that it's possible to create a Dynamic App Service Plan without the App Service Environment. This is currently blocking https://github.com/terraform-providers/terraform-provider-azurerm/issues/960

Automation for azure-libraries-for-java

Automation for azure-sdk-for-go

Automation for azure-sdk-for-python

Hey Tom!
Nice to see you here again! we'll tag someone from the Web team to review and Ack and we can move forward!

@naveedaz  PTAL since you may have some context regarding this spec.
@tombuildsstuff I will try and debug the issues with CIs once we get a review from the service team
Thanks!

@naveedaz replying out of the in-line thread to be able to scroll to the bottom and see the latest reply - hope you don't mind :)
AFAIK in order to create a Function App in a Consumption plan this is required, as can be seen in this ARM Template - if that's not the case is there any documentation we can reference about this? All of the examples I've seen for creating a Consumption plan referencing the ARM Templates which have this field present, which is why we believe it's required, but maybe we're wrong 😄
At the moment customers are unable to deploy Function Apps into a Consumption Plan with Terraform since this field is missing - hence the disclaimer here: https://www.terraform.io/docs/providers/azurerm/r/function_app.html
Thanks!

@tombuildsstuff
Not sure if it is documented publicly. The template at https://github.com/Azure/azure-quickstart-templates/blob/master/101-function-app-create-dynamic/azuredeploy.json#L48 is fairly old and uses the old serverfarm API.
The template excerpt below will achieve what you are looking for:
{
"type": "Microsoft.Web/serverfarms",
"apiVersion": "2016-09-01",
"name": "[variables('hostingPlanName')]",
"kind": "functionapp",
"sku": {
"name": "Y1",
"tier": "Dynamic",
"size": "Y1"
},
"location": "[resourceGroup().location]",
"properties": {
}
}

@naveedaz I've taken a look and this appears correct, as such I've opened https://github.com/terraform-providers/terraform-provider-azurerm/pull/981. Given this field is available in the API (and ARM Templates) - should it be exposed in the Swagger either way?

@tombuildsstuff
It is already defined in the swagger: https://github.com/Azure/azure-rest-api-specs/blob/master/specification/web/resource-manager/Microsoft.Web/stable/2016-03-01/CommonDefinitions.json#L2211
and is referenced in the App Service Plan model: https://github.com/Azure/azure-rest-api-specs/blob/master/specification/web/resource-manager/Microsoft.Web/stable/2016-03-01/CommonDefinitions.json#L379

@naveedaz right - however this value isn't configurable using the current Swagger (the value is only retrievable) - hence this PR :)

@tombuildsstuff
The property sku is settable from swagger. The property sku is at the same level as kind, type and location not within the "properties" property
{
             "type": "Microsoft.Web/serverfarms",
             "kind": "functionapp",
             "sku": {
                          "name": "Y1",
                          "tier": "Dynamic",
                          "size": "Y1"
             },
             "location": "",
             "properties": {
             }
}

@naveedaz the field computeMode is available in the API within the SKU block - but isn't visible/setable in any of the SDK's: https://github.com/Azure/azure-sdk-for-net/blob/psSdkJson6/src/SDKs/WebSites/Management.Websites/Generated/Models/SkuDescription.cs#L61
Whilst admittedly the field lives within the SKU block, which is available in the SDK - the computeMode field isn't exposed in either the C#/.net SDK or the Go SDK (or any of the other SDK's) since they're missing from the Swagger
Hope that helps clarify what I mean :)
Thanks!

@tombuildsstuff
You should NOT be using the ComputeMode  field at all. Please only use the sku block to set the App Service Plan. ComputeMode will eventually be removed. The supported way of creating a consumption plan is to use the sku property.

@naveedaz ok cool, thanks for confirming that - I'll close this PR in the meantime :)
I guess there's two outstanding issues which need to be fixed as a result of this PR:

Document this field has been removed/deprecated, so folks don't continue using it
Fix the outdated ARM Templates in the Azure/azure-quickstart-templates repository to use the newer SKUs without the computeMode field

#2 is particularly important in our case since many folks are either using/looking at ARM Templates in order to determine which configuration of fields needs to be set in other tooling (such as Terraform) in order to achieve certain configurations.
@salameer @naveedaz where would be the best place to file bugs for these two issues? :)
Thanks!

Automation for azure-sdk-for-node

For #2 Please create an issue on https://github.com/Azure/azure-quickstart-templates against the template: https://github.com/Azure/azure-quickstart-templates/blob/master/101-function-app-create-dynamic/azuredeploy.json.
I am not aware of a good place for documenting api-version differences. We will need to figure that out.

@naveedaz opened https://github.com/Azure/azure-quickstart-templates/issues/4352 to cover No. 2
@salameer would you be able to comment on the best way forward to fix No. 1? :)
Thanks!


updated maven repository to use SSL

The install instructions
mvn clean install -DskipTests
didn't work because of a wrong repository address.
[ERROR] Failed to execute goal on project kafka-connect-hana: Could not resolve dependencies for project com.sap:kafka-connect-hana:jar:1.0-SNAPSHOT: Failed to collect dependencies at org.scala-lang:scala-library:jar:2.12.11: Failed to read artifact descriptor for org.scala-lang:scala-library:jar:2.12.11: Could not transfer artifact org.scala-lang:scala-library:pom:2.12.11 from/to central (http://central.maven.org/maven2/): Transfer failed for http://central.maven.org/maven2/org/scala-lang/scala-library/2.12.11/scala-library-2.12.11.pom: Unknown host central.maven.org: nodename nor servname provided, or not known -> [Help 1]

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Michael Bodnar seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Forward signals via HostCommandSignal

Signals received from the host-spawn side should get sent to the host command using HostCommandSignal: https://docs.flatpak.org/en/latest/libflatpak-api-reference.html#gdbus-method-org-freedesktop-Flatpak-Development.HostCommandSignal
Right now if I spawn a process on the host and send signals, the host side never receives them. I believe this is the issue but not 100% sure.

Seems like a legit request, but I don't have time to work on it. PR is welcome.

I have a WIP version in the handle-signals branch that forwards signals. I still need a little time to test it before merging


Update weaver-cats, weaver-discipline, ... to 0.8.0

Updates

com.disneystreaming:weaver-cats
com.disneystreaming:weaver-discipline
com.disneystreaming:weaver-framework
com.disneystreaming:weaver-scalacheck

from 0.7.15 to 0.8.0.
GitHub Release Notes
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.disneystreaming" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "@monthly" },
  dependency = { groupId = "com.disneystreaming" }
}]

labels: library-update, early-semver-major, semver-spec-minor, commit-count:1

Superseded by #145.


Add GravityTurn from Kerbal Stuff

This pull request was automatically generated by Kerbal Stuff on behalf of Overengineer1, to add GravityTurn to CKAN.
Please direct questions about this pull request to Overengineer1.

Done in #2920


MxNet input dataset

Hello,
Does mxnet take the same data file as Caffe (lmdb)? Thank you. If yes, please elaborate.

As I know, Mxnet doesn't take lmdb(or leveldb) as data backend which is used by caffe.
But it uses a unique data IO module to tackle the bottle-neck of training big models, the database files usually end with '.rec', which can be easily generated by mxnet_root/bin/im2rec.

Thanks for the quick reply. But wouldn't be nice if they are using the same dataset format. ImageNet are pretty large dataset.

Hello,
After I have successfully compiled it with CMake, there is no such a file named mxnet_root/bin/im2rec.
Could you please provide more info? Thanks.

Use the python version tools/im2rec.py

Thank you for the assist. Is there an example to build a dataset with im2rec.py? Thank you.

@linnanwang you can generate the image list use awk


[media/CircleGraph] Set visq attributes into node

This will revise to set visq attributes into node properties.
ONE-vscode-DCO-1.0-Signed-off-by: SaeHie Park<EMAIL_ADDRESS>

for https://github.com/Samsung/ONE-vscode/issues/1386#issuecomment-1283180153 , from  #1387


Split entities table into one table per subgraph

This PR makes it so that newly deployed subgraphs will be stored in their own entities (and entity_histroy) table. Deploying a new subgraph will create a schema in Postgres, and store the data for the subgraph in that schema.
The PR does not migrate existing data; instead, queries (and changes) against existing data are still done against the existing public.entities table. Migrating data will be the subject of another PR.
(Opening this as draft as I haven't tested this other than running unit tests; will remove draft status once I've done that. The code is ready for review, though)

After the small fixes I made, this works for me now, both with existing entities, a manually migrated subgraph, and a freshly deployed one. As always, somebody else checking is hugely appreciated.

If I may ask a naive question, did you consider the approach of creating a view of the split tables that looks like our current entities table and letting Diesel treat that as single table? I've read that not all views are updatable, presumably such a view would have that issue?

If I may ask a naive question, did you consider the approach of creating a view of the split tables that looks like our current entities table and letting Diesel treat that as single table? I've read that not all views are updatable, presumably such a view would have that issue?

Yes, with a view we would have to put a trigger on the view that shuffles the data into the right tables; that would force us into a situation where every CRUD operation on entities is inherently a single-object operation and preclude optimizations I'd like to do further down the line like using bulk inserts (insert with many entries in the VALUES clause or an outright COPY statement)
What makes this awkward with Diesel is that Diesel's design is fundamentally based on the idea that you know the entire schema at compile time, which does not apply to our situation.

Addressed all comments (I believe)


Add a "how to deploy and install" instruction

Could you please add a small section on how to deploy this application onto a standard OS like Ubuntu? This way I and other can help to test and use the application.

Hiya @tjvandam, could you explain a bit more about the kind of information you're looking for? As mentioned in the README, installation uses the environment.yml which specifies all the dependencies that conda can install for you. So it's a pretty standard python library installation process.

Get mamba from https://mamba.readthedocs.io/en/latest/installation.html (micromamba should work) or conda from https://docs.conda.io/projects/conda/user-guide/install/linux.html (miniconda should work).
Clone this repo.
Open a terminal in the root of the cloned repo.
If you chose miniconda/conda, run

conda env create --name environment_name_you_want_to_use --file environment.yml

If you chose mamba/micromamba, run
mamba env create --name environment_name_you_want_to_use --file environment.yml

Now you can activate the environment

For conda:
conda activate environment_name_you_want_to_use

For mamba:
mamba activate environment_name_you_want_to_use

Glancing over the conda docs might be helpful in case you want to manage your environments.
Once you've activated the environment, you can run ecoscope from the command-line. Example Colab notebooks can be found on https://ecoscope.io/en/latest/notebooks.html but you could just repurpose the python snippets to run them from the command-line yourself.

@tjvandam the instructions I provided above are more oriented towards developers (who want to contribute to the source code, for example).
For users with a less of a technical background, we would suggest just using Google Colab, which is free to use.
There are example notebooks at https://ecoscope.io/en/latest/notebooks.html that you can use as a reference and run your analyses in Colab on the cloud. This is a convenient zero-configuration option for you as it saves you from having to set up and manage the environment yourself on your local machine - Google handles all of that for you on the cloud.


Pad 2518 migrate facebook marketing v1 for vfp to bq v2

JIRA Ticket: https://techallies.atlassian.net/browse/PAD-2518?atlOrigin=eyJpIjoiNTQxZjNhYWI3MWQ4NDdkYTkzNzFiYjZiYWZhZjZmMWUiLCJwIjoiaiJ9
Airbyte PR: https://github.com/community-tech-alliance/airbyte-configs/pull/221
airflow-dags PR: https://github.com/community-tech-alliance/airflow-dags/pull/840

could be ready for review so that airflow dag in prod will deploy this code, but also can run this locally to make changes if needed


[Sprint: 47] [1.1.x] XD-2923 Spark streaming module pubsub consumer fix

Fix the spark streaming MessageBusReceiver to bind
to the pubsub consumer if the underlying input channel is of pub/sub.

Minor fixes to SparkStreamingTests class

LGTM; merging

merged as https://github.com/spring-projects/spring-xd/commit/a81f1439ac259e64f6992cf44944ac63b9629e5e


CEO-188 Add back end changes for user assignment

Purpose
Add back end changes for user assignment
Related Issues
Closes CEO-188
Submission Checklist

[x] Commits include the JIRA issue and the #review hashtag (e.g. CEO-### #review <comment>)
[x] Code passes linter rules (npm run eslint/clj-kondo --lint src)

Testing
No testing yet since the UI is not created.  Can manually check if users are assigned by uncommenting 329

Looking good 👍
Do we have a way to handle if a user adds or removes a user from the assigned? Or if the percentages of a users change?

Two questions:

Do we have a way to handle if a user adds or removes a user from the assigned? Or if the percentages of a users change?
Once users are assigned, they should remain fixed.  However if the user updates the projects, and changes the plot layout, the users would get reassigned.  I think its valid to check for just assignment changes and update them. https://sig-gis.atlassian.net/jira/software/projects/CEO/boards/10/backlog?selectedIssue=CEO-215

How would you like to handle the camel->kebab for the designSettings properties?
I will do the integration of the UI for each of these as the UI piece gets merged.


Add new collection - 'ctf-cybersec-resources'

Thank you for contributing! Please confirm this pull request meets the following requirements:

[x] I followed the contributing guidelines: https://github.com/github/explore/blob/master/CONTRIBUTING.md
[x] I have no affiliation with the project I am suggesting (as a maintainer, creator, contractor, or employee).

Which change are you proposing?

[ ] Suggesting edits to an existing topic or collection
[x] Curating a new topic or collection
[ ] Something that does not neatly fit into the binary options above

Editing an existing topic or collection
I'm suggesting these edits to an existing topic or collection:

[ ] Image (and my file is *.png, square, dimensions 288x288)
[ ] Content (and my changes are in index.md)

Please replace this line with an explanation of why you think these changes should be made.

Curating a new topic or collection

[x] I've formatted my changes as a new folder directory, named for the topic or collection as it appears in the URL on GitHub (e.g. https://github.com/topics/[NAME] or https://github.com/collections/[NAME])
[x] My folder contains a *.png image (if applicable) and index.md
[x] All required fields in my index.md conform to the Style Guide and API docs: https://github.com/github/explore/tree/master/docs

There's no collection, as of now, that showcases the resources that CTF participants and cybersecurity enthusiasts find useful.

Something that does not neatly fit into the binary options above

[ ] My suggested edits are not about an existing topic or collection, or at least not a single one
[ ] My suggested edits are not about curating a new topic or collection, or at least not a single one
[ ] My suggested edits conform to the Style Guide and API docs: https://github.com/github/explore/tree/master/docs

Please replace this line with an explanation of your proposed changes.

Please note: we will close your PR without comment if you do not check the boxes above and provide ALL requested information.

Thank you @gallexi for helping me figure out why the build was failing on #1795 repeatedly.
I was using the GitHub website to upload the files on Windows, so every line was automatically terminated with CRLF.
I solved it by creating it again, using the Git CLI this time.

hey, thanks for fixing the build @Mr-Skully! Jekyll frontmatter uses YAML syntax, and it looks like generally YAML doesn't play well with \r as a line breaker, so I'm inclined to leave the test as it is. We can however update the error message to have a clearer suggestion, such as "Please try editing this file in the web UI" which should eliminate the introduction of carriage returns, regardless of OS! I've left some in-line suggestions as well.


Crash when node-ffi 2.0 is required inside electron

Hi,
We are having a complicated issue, related to both node-ffi and electron. Let me explain the full details of the problem and I'll be glad if you can advise us how to proceed in this situation.
We are using node-ffi in one of our libraries. Now we are working on integration of this library inside electron application. Electron itself is using a fork of iojs and they apply some fixes in order to keep it working in the same way.
After some hard time of debugging, we've found that in latest electron versions, when we call require(ffi) the electron crashes. The problem is caused by one commit in their fork of iojs. I've logged a new issue in electron, explaining the whole situation, and they explained why they need this commit:
We do need this commit for embedded blink allocator in node's buffer. Because iojs reimplements node buffer in v3.0, and it starts to manage the memory of buffer, which conflicts with Blink's behavior.
It will cause electron to crash if we remove this commit.

You can check the whole discussion in the electron issue and the callstack of the error is:
0   ffi_bindings.node               0x0000000110491fba ffi_prep_cif_core + 61
1   ffi_bindings.node               0x000000011049211a ffi_prep_cif + 23
2   ffi_bindings.node               0x000000011048ff93 FFI::FFIPrepCif(Nan::FunctionCallbackInfo<v8::Value> const&) + 469
3   ffi_bindings.node               0x0000000110490a18 Nan::imp::FunctionCallbackWrapper(v8::FunctionCallbackInfo<v8::Value> const&) + 131
4   libnode.dylib                   0x000000010b15b0bf v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) + 159
5   libnode.dylib                   0x000000010b17fa85 0x10ade2000 + 3791493
...

So the guys from electron explained that currently it is not possible to remove the commit, which breaks node-ffi. Maybe you can suggest a workaround that we can apply in our node-ffi fork in order to be able to use your library inside electron.
You can use this application as a sample reproduction case. Just execute the follow the steps below:

npm i -g electron-prebuilt - make sure to check the installed version as you'll need it later.
git clone https://github.com/rosen-vladimirov/myElectronApp
cd myElectronApp
./installAndStart.sh 0.31.2 x64 - here you can replace 0.31.2 with the version of electron-prebuilt that you have installed.
This script will install the required dependencies, rebuild them with correct headers (electron requires special way of rebuilding) and start the app. You will not see anything first time you start it, as electron will crash. After that, just remove ffi = require("ffi")  from main.js file and execute electron . You'll see that the app will start.

I know that your latest version is working fine with iojs 3.x, but we need it to work with electron as well. We'll be really happy in case you have time to take a look at this issue and suggest any possible solutions/workarounds.
Thanks in advance for your help and understanding!

We're aware of this issue. If you try node-ffi with io.js 3.0 or with 3.1 then it will crash too. That's because of an alignment issue of that commit above. We cannot do anything about it, it lies in that buffer implementation not in ffi side. Issue history and the fix is there: https://github.com/nodejs/node/issues/2484#issuecomment-133563531
Electron guys should integrate that mentioned commit too.

Hi @unbornchikken,
Thank you for the information and please excuse me for the late reply. Electron guys have found a way to fix this issue. However we are trying to use node-ffi with iojs 3.3.0 on our side and we have a lot of troubles with the garbage collection. I'll open separate issue for it.
Thank you, your help is really appreciated!


Extend recipes documentation with multi browser examples

Describe the bug
We have a rather complicated end to end test case scenario, but nothing not manageable with an e2e test framework (in my opinion).
Let me explain:

we have 2 frontends:

frontendA is coded with UI5 (for managing business partners)
frontendB is coded with angular (a webshop where business partners can login and order items)

we have an admin user userA that will login into frontendA (by using the chrome browser for example) and create a new business partner by typing first name, last name, email address and click save button
in the backend the magic starts happening and the new business partner is persisted in the DB with a default initial login password
userB will be able to login into frontendB (by using the firefox browser for example) by using the email address and the default initial password
userB will place an order in frontendB webshop
a live statistic notification with number of orders increased +1 will be visible in frontendA

To be able to implement this test case scenario we wanted to use the capabilities configuration option from webdriverIO https://webdriver.io/docs/options/#capabilities-1 and have multiremote 2 browser instances.

For one browser instance we want to skipInjectUI5OnStart and after a successful login (the login page from the authentication provider service does not contain UI5 controls) we want to inject wdi5 service.
For the other browser instance we don't even need wdi5, since it's written in angular.
Expected behavior
I would have wished for a better documentation regarding the whole multi browser configuration for wdi5.
Please think about extending the documentation here, since this kind of recipe might be also helpful for developers trying to test chatbot applications.
I also see that "login for e2e real test cases" is on the road map https://github.com/ui5-community/wdi5/issues/207 with the first intention to add support for standard BTP IdP login (w/ username + password).
Please also take into consideration extending the documentation how it's supposed to work with multi remote configuration when one browser instance would use the BTP IdP login and the other browser instance won't.

the scenario/use-case you describe is spot-on for the multi-remote feature of wdi5 👍
and I take it you've gained quite some experience with coding multi-remote tests in wdi5!
given that it's pretty easy to contribute to the documentation, would you mind giving this a shot?
as you stated, many would benefit from that.
thanks in advance!


[#4705] Ensure ChannelHandler.handlerAdded(...) is always called as f…

…irst method of the handler
Motivation:
If a user adds a ChannelHandler from outside the EventLoop it is possible to get into the situation that handlerAdded(...) is scheduled on the EventLoop and so called after another methods of the ChannelHandler as the EventLoop may already be executing on this point in time.
Modification:

Ensure we always check if the handlerAdded(...) method was called already and if not add the currently needed call to the EventLoop so it will be picked up after handlerAdded(...) was called. This works as if the handler is added to the ChannelPipeline from outside the EventLoop the actual handlerAdded(...) operation is scheduled on the EventLoop.
Some cleanup in the DefaultChannelPipeline

Result:
Correctly order of method executions of ChannelHandler.

@johnou good catch! Can you just address the one nit :)

@normanmaurer nit addressed, PTAL.

Cherry-picked into 4.0 as ba381f1a27c3251fa13725f09dd0cb771326f113 (after fixing the check style error)

@normanmaurer sorry about that, I must have had the wrong profile loaded in my IDE.


Add support for io.js #385.

Add support for io.js.
All tests are passing on all io.js versions (1.0.0 - 1.0.4).

https://github.com/mapbox/node-sqlite3/pull/399 also fix same problem

@fengmk2 Yes, I saw, but too late.
Did you manage to update Travis-CI and Appveyor? Both of them should support io.js now.


Updating the code

creating a simple code
What type of PR is this? (check all applicable)

[ ] 🚀 Added Name
[x] ✨ Feature
[ ] 🌟 stared the repo
[ ] 🐛 Grammatical Error
[ ] 📝 Documentation Update
[ ] 🚩 Other

Description
Add Link of GitHub Profile

i just make simple code from complex code


Clara update request

This is relate to this fixed issue https://github.com/philsquared/Clara/pull/5
It would be helpful if Catch's copy of Clara is updated to the latest version <IP_ADDRESS>, at least.
Thanks!

Bump
Sorry for annoyance, but is there any chance for the update soon'ish?


Make Node versions consistent with nvm

nvm uses node and stable to denote the latest version, though the latter is deprecated.
(Note that I was unable to build nvm-windows on OSX due to issues finding settings.txt, so this PR has not been actually tested.)

Closing this since a) I fell way behind and b) the goal is to focus on being as close to node as possible (i.e. current/lts vs node/latest)


alt+click not working

Alt+Click to insert cursor is not working. It simply won't allow me to choose multiple cursors.

VSCode Version: Code 1.9.1 (f9d0c687ff2ea7aabd85fb9a43129117c0ecf519, 2017-02-08T23:44:55.542Z)
OS Version: Linux x64 4.4.0-62-generic
Extensions:

Extension
Author
Version

html-snippets
abusaidm
0.1.0

trumpsum
jerky676
1.1.2

html5-boilerplate
sidthesloth
1.0.1

open-in-browser
techer
0.0.3

Steps to Reproduce:

Duplicate https://github.com/Microsoft/vscode/issues/3091


Fix metrics collection for privileged process on Windows

What does this PR do?
It improves metric collection on Windows hosts so we can collect some metrics from privileged process like Elastic Endpoint or Elastic-Agent. In order to achieve that metrics collection for Windows are grouped by the access level required to collect metrics. First the metrics that can be collected with PROCESS_QUERY_LIMITED_INFORMATION are collected, then the others are collected. In case the second batch fails, we still report the partial metrics.
Why is it important?
It allows us to collect CPU and memory metrics from Endpoint and Elastic-Agent, which improves our monitoring dashboards.
Checklist

[x] My code follows the style guidelines of this project
[x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added tests that prove my fix is effective or that my feature works
[x] I have added an entry in CHANGELOG.md

Author's Checklist

[ ] Make sure nothing broke on:

[ ] Linux
[x] MacOS

Related issues

Closes https://github.com/elastic/beats/issues/17314

It's ready for review.

@faec I addressed all of your comments on 9ef7b7172


Sun azimuth at current time

Hello,
thank you for your awesome work.
As the sun moves from sunrise to sunset in a given location you provide the azimuth of the sun on sunrise and sunset.
I have a question: is it possible to have the data of the current position of the sun. For example if it is 10h30 to see the position of the sun on the map between sunrise and sunset, and if the time evolves you see the sun on a map moving?
is this possible?

Is it possible to provide an example ? It would be appreciated

Oh course, you can find it in the README file.

Thank you very much for your help.


Error in the conference.py file

When I run the inference.py file, I always report an error: KeyError: 'vd'
Traceback (most recent call last):
File "E:\daima\brain-diffuser-main\versatile_diffusion\inference.py", line 566, in
vd_wrapper = vd_inference(pth=pth, fp16=args.fp16, device=device)
File "E:\daima\brain-diffuser-main\versatile_diffusion\inference.py", line 35, in init
net = get_model()(cfgm)
File "E:\daima\brain-diffuser-main\versatile_diffusion\lib\model_zoo\common\get_model.py", line 87, in call
net = self.modelt
KeyError: 'vd'
Thank you very much for your help

Including running reconstruct_ Images. py and reconstruct_ txt2im.py file.The above error may also occur


The performance of toBuffer will drop after overlayWith a small png file.

I did a test about the toBuffer method of sharp.
First, I test the toBuffer without overlayWith. The test result is:
2017-11-16T04:42:32.026Z - info: ended_to_buffer_image:  383
2017-11-16T04:42:32.037Z - info: ended_to_buffer_image:  392
2017-11-16T04:42:32.071Z - info: ended_to_buffer_image:  117
2017-11-16T04:42:32.105Z - info: ended_to_buffer_image:  150
2017-11-16T04:42:32.116Z - info: ended_to_buffer_image:  161
2017-11-16T04:42:32.129Z - info: ended_to_buffer_image:  142
2017-11-16T04:42:32.153Z - info: ended_to_buffer_image:  166
2017-11-16T04:42:32.167Z - info: ended_to_buffer_image:  179
2017-11-16T04:42:32.184Z - info: ended_to_buffer_image:  192
2017-11-16T04:42:32.222Z - info: ended_to_buffer_image:  228
2017-11-16T04:42:32.225Z - info: ended_to_buffer_image:  235
2017-11-16T04:42:32.248Z - info: ended_to_buffer_image:  252
2017-11-16T04:42:32.254Z - info: ended_to_buffer_image:  257
Then, I test the toBuffer with overlayWith another image(It's a png image, size is 3k). I cached the overlay image buffer. Each time of:
rawImage.overlayWith(buffer, {
              tile: filterRequest.overlayTile(),
              gravity: gravity,
              top: position.top,
              left: position.left
            });

it will use the same overlay image buffer. I test the time used of overlayWith is 0 or 1ms.
Here's the time used of toBuffer method:
2017-11-15T08:17:47.862Z - info: ended_to_buffer_image:  564
2017-11-15T08:17:47.928Z - info: ended_to_buffer_image:  741
2017-11-15T08:17:48.083Z - info: ended_to_buffer_image:  764
2017-11-15T08:17:48.182Z - info: ended_to_buffer_image:  518
2017-11-15T08:17:48.344Z - info: ended_to_buffer_image:  675
2017-11-15T08:17:48.415Z - info: ended_to_buffer_image:  744
2017-11-15T08:17:48.543Z - info: ended_to_buffer_image:  850
2017-11-15T08:17:48.709Z - info: ended_to_buffer_image:  1015
2017-11-15T08:17:48.821Z - info: ended_to_buffer_image:  1126
2017-11-15T08:17:48.860Z - info: ended_to_buffer_image:  1163
2017-11-15T08:17:49.025Z - info: ended_to_buffer_image:  1326
2017-11-15T08:17:49.175Z - info: ended_to_buffer_image:  1472
2017-11-15T08:17:49.263Z - info: ended_to_buffer_image:  1556
2017-11-15T08:17:49.302Z - info: ended_to_buffer_image:  1589
2017-11-15T08:17:49.517Z - info: ended_to_buffer_image:  1796
2017-11-15T08:17:49.635Z - info: ended_to_buffer_image:  1910
2017-11-15T08:17:49.703Z - info: ended_to_buffer_image:  1975
2017-11-15T08:17:49.830Z - info: ended_to_buffer_image:  2094
2017-11-15T08:17:49.948Z - info: ended_to_buffer_image:  2207
2017-11-15T08:17:50.091Z - info: ended_to_buffer_image:  2343
As you can see, the performance of toBuffer drop.
VIPS version 8.5.5.
Node version 8.9.1.
Sharp version 0.18.4.

"I test the time used of overlayWith is 0 or 1ms."

This function sets the values to be used later when toBuffer is called so this is expected.
The timings suggest you may be initiating all the toBuffer calls at once rather than (a)waiting for each one to return before the next call to toBuffer.
Did you see http://sharp.dimens.io/en/stable/api-utility/#concurrency ? By default 4 images are processed in parallel.
As an aside the overlayWith code will be undergoing a fairly major refactor as part of #728 and will provide the repeated overlay scenario you're testing.

@lovell Thanks for your reply.
I use ec2 c4.large to run the test. So the default concurrency count should be 2.
What confuses me is that the size of raw image info will increase 6 times after use overlayWith. I guess this makes the overlayWith slow.

HI @lovell, Do you know why the size of raw image info will increase maybe 6 times after use overlayWith?

Please can you provide a more complete, standalone code sample with the time measurements in place for me to be able to help further.

@lovell Thank you. I guess I know why overlayWith become slow. Becuase the overlay image that I use is a png file, it contains an alpha channel.
This comes from the document:
If the overlay image contains an alpha channel then composition with premultiplication will occur.
If I use a jpg overlay image, then toBuffer will be very quick. But my problem is I need the overlay image has transparent space. I don't know if there are some better ways to implement it.

Please subscribe to #728 for updates about improvements to overlayWith, which I think will help you, but without a more complete code sample it's quite hard to tell what you're trying to achieve.

Hi @lovell ,  My scenario is that I have a web application it depends on "sharp" and it can resize and overlay a watermark of the requested image then return to the client side. There are lots of image request from the client, So I need the performance of this web application to be very high. Previously, We didn't have the requirement of the watermark and our web application just has the resize functionality, the performance is very high. For now, we have the requirement to dynamically add a watermark. The problem is that the watermark is png image and it contains an alpha channel, so it will cause:
If the overlay image contains an alpha channel then composition with premultiplication will occur.
Watermarking reduces performance maybe 6x after overlay with a png watermark. I know this is also related to the complexity of watermarking pictures.  My question is if there are some better ways to overlay with a watermark image which contains an alpha channel?
The code is like this:
const processImage = (rawImage, cb) => {
  rawImage
    .resize(
      400,
      400,
      {interpolator: sharp.interpolator.bicubic}
    );

  sharp('/overlay/logo.png').toBuffer((err, overlayImage, metadata) => {
    if (err) {
      return cb(err);
    }

    rawImage.overlayWith(buffer, {
      gravity: 'southeast',
      top: xxx,
      left: xxx
    })
    .toFormat('jpeg');

    return cb(null, rawImage);
  });
}

...
processImage(img, (err, newImg) => {
  if (err) {
    return cb(err);
  }

  const start_to_buffer_image = +new Date();
  newImg.toBuffer((err, output, infoResized) => {
    if (err) {
      return cb(err);
    }

    const ended_to_buffer_image = +new Date();
    logger.log.info("ended_to_buffer_image: ", (ended_to_buffer_image - start_to_buffer_image));

    cb(err, output);
  });

});
...

HI @lovell , Have you done a performance test of overlay an image which contains an alpha channel before?

Thank you for providing more complete example code..
Try providing the 'overlay/logo.png' path directly to overlayWith rather than via
sharp('/overlay/logo.png').toBuffer(... as you're incurring a costly and unnecessary PNG decode/encode roundtrip.
-sharp('/overlay/logo.png').toBuffer((err, overlayImage, metadata) => {
-    if (err) {
-      return cb(err);
-    }
-
-    rawImage.overlayWith(overlayImage, {
+    rawImage.overlayWith('/overlay/logo.png', {

@lovell Thank you, but after use path directly the time used of toBuffer is the same:
  const start_to_buffer_image = +new Date();
  newImg.toBuffer((err, output, infoResized) => {
    if (err) {
      return cb(err);
    }

    const ended_to_buffer_image = +new Date();
    logger.log.info("ended_to_buffer_image: ", (ended_to_buffer_image - start_to_buffer_image));

    cb(err, output);
  });

Without overlay toBuffer just need nearly 15ms, but with overlayWith png file then toBuffer, the time used will go to 90ms or more.

I'd expect overlayWith to involve more CPU time as it's doing more work.
This module uses uv_queue_work to access a shared worker thread pool. If your code is processing more than 4 images at once then the timings captured by this sample code will also include the time queued waiting for a libuv thread to service the toBuffer call.
If you've not already done so, please read http://sharp.dimens.io/en/stable/api-utility/#concurrency
See also this Stack Overflow answer https://stackoverflow.com/a/29472222/4625124

I hope this answered your questions, please re-open with more details if not.


How to pass FileActivatedEventArgs

I've been unable to find samples, docs or articles regarding file activation using T10. I tried this:
public override async Task OnStartAsync(StartKind startKind, IActivatedEventArgs args)
{
var fileArgs = args as FileActivatedEventArgs;
        if (fileArgs != null)
        {
                    NavigationService.Navigate(typeof(Views.MainPage), fileArgs);
        }

// ...
}
But I'm getting this error when Navigate() is called:
"Error getting value from 'Folder' on Windows.Storage.Search.StorageFileQueryResults."
I think the serialization service is having problems with this data type...

Please ask questions like this on StackOverflow with Template10 as a tag.


HACS and Sonoff LAN fails with "Could not use repository with ID xxxxxxx ('NoneType' object has no attribute 'attributes')"

Hi,
I've completely wiped out my install just to start from fresh, but still nothing.
Installed HA: core-2021.2.1 running on ubuntu.
Installed HACS 1.11.0
Added a new repo: Lightwave RF (just to test, and it works).
Added another repo: Sonoff LAN
Restarted HA, cleared cache, stopped HA.
Started HA:

So, it's using it.
But nothing shows on the device list. Even after setting the user/pass in configuration.yaml
And if I go HACS:

Selecting, "REPOSITORY" send me to github (OK), but selecting "Information", "Update information" or "Uninstall" don't work.
Information:

Update Information:
Doesn't show anything.

Uninstall:
Doesn't show anything.

Any help?

Well... it's just working now... :|


Generate name before validation

What type of PR is this?

/kind bug
What this PR does / why we need it:
generate name for the resource before validation if it has no name but generatedName is set.
Which issue(s) this PR fixes:

Fixes #121
Special notes for your reviewer:
Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/easycla


Default implementation of sender adaptors are underconstrained

 Issue by whatishappened
Sunday Nov 14, 2021 at 10:02 GMT
Originally opened as https://github.com/NVIDIA/stdexec/issues/258

There are lots of sender adaptors that has to store the arguments in it, such as then, upon_, let_, bulk, etc.
Sender concept requires the type to be move constructible, so I believe any sensible implementation of those adaptors will require the arguments to be move constructible as well, however none of the specification of the default implementation of those adaptors constrains the arguments to be move constructible.

In P2300R10 the algorithms let_value/error/stopped, then/upon_error/upon_stopped, bulk and just/just_error now all require their captured values/function-objects to satisfy movable-value, which implies move-constructibility.
So I think this issue is addressed now.


2.6.0 Adding ability to cache requests made with methods beyond GET is a breaking change

Hello 👋
Today we had the below unit test fail, due to the change introduced in 2.6.0 that now automatically caches requests made with HTTP verbs outside of GET (i.e. POST, PUT etc.). I think the behaviour should be for consumers to opt-in to this enhancement through a config option, and the default behaviour should remain. Otherwise it is a breaking change and should result in a major version bump.
it('should not cache the request', async () => {
      nock(baseUrl, { reqheaders }).post(path, body).twice().reply(200, validResponse);
      const firstResponse = await service.post(`${baseUrl}${path}`, body, reqheaders, operation);
      const secondResponse = await service.post(`${baseUrl}${path}`, body, reqheaders, operation);
      expect(firstResponse.data).to.deep.equal(validResponse);
      expect(secondResponse.data).to.deep.equal(validResponse);
      expect(firstResponse.request.fromCache).to.be.undefined;
      expect(secondResponse.request.fromCache).to.be.undefined; // Test fails on this line with true != undefined
    });

I agree with the above. Codebases using the adapter make the assumption that only GET requests get cached and now need to go through a migration process to update all calls with other methods to control caching. I may be wrong, just sharing my thought 😄

Hello @JCass45 really sorry for introducing this breaking change. Will work on restoring previous default behavior of caching only GET requests and introduce a specific option to extend it to other methods.

@RasCarlito 🙏🏻 Appreciate it!

Should be fixed in latest release https://github.com/RasCarlito/axios-cache-adapter/releases/tag/2.7.1

Yep confirmed 👍


Not Able to send proactive message in React Component because of directlineJS re-initialization

I am trying to send a pro-active message using the React Webchat component. Following are the steps:
import React, { Component } from 'react'
import { Chat } from 'botframework-webchat';
import { DirectLine } from 'botframework-directlinejs'

const DIRECT_LINE_SECRET = "****"

class Webchat extends  Component {
  constructor(props) {
    super(props)
    this.directLine = new DirectLine({
        secret: DIRECT_LINE_SECRET
    })
  }
  componentDidMount() {
    this.directLine.postActivity({
        from: { id: 'id', name: "name"}, // required (from.name is optional)
        type: 'event',
        text: `Hey!`
    }).subscribe(
        id => console.log("Posted activity, assigned ID ", id),
        error => console.log("Error posting activity", error)
    );
  }
  render() {
    return (

      <Chat directLine={this.directLine} user={{ id: ""  , name: profileObj.name , google_token_id : tokenId }} />

    )
  }
}

I sent a backchannel message using the directline object to inform the chatbot server about the address of the webchat in order to send a pro-active message ( can be found in the componentDidMount method) .
But the conversation id changes for the directline property on the  component, because it is reinitialized in the Chat.tsx  file.
 const botConnection = this.props.directLine
            ? (this.botConnection = new DirectLine(this.props.directLine))
            : this.props.botConnection
            ;

And now because of this , the chatbot server is unable to reach the newly initialized WebChat Component.

I overlooked the botConnection property. My bad.
This is the right way to do it I guess:
render() {
    return (

      <Chat botConnection={this.directLine} user={{ id: ""  , name: profileObj.name , google_token_id : tokenId }} />

    )
  }

Closing the issue.

Even that simpler code still doesn't seem right. I think maybe you're trying to have your web page send a backchannel message to the bot before the user has typed a message, so that the bot can then send a proactive message to the right address. Is that correct? If so, that is a very common pattern.

Yes. That's absolutely correct.

Great, then that code is correct. I want to make sure you know that you don't need to do class Webchat extends Component unless you're doing so for other reasons.


Mutating webhook is not idempotent

Describe the bug:
We intend to use Velero to perform cluster backup including restic integration for volume snapshot, however Velero makes a backup of all resources including pods and does a restore of those as well. If we exclude pods, restic can not gain access to the mounted volumes to perform a snapshot.
The problem is when the pod is restored, then it already contains the copy-vault-env container, but the mutating webhook ignores that and appends another container and volume anyway. Kubernetes then fails with a validation error:
Pod "kiali-oauth2-proxy-76cdff896c-55p26" is invalid: [spec.volumes[3].name: Duplicate value: "vault-env", spec.containers[0].volumeMounts[3].mountPath: Invalid value: "/vault/": must be unique, spec.initContainers[1].name: Duplicate value: "copy-vault-env"]

Expected behaviour:
According to the documentation https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#idempotence then the webhook must be idempotent:

Mutating webhooks must be idempotent, able to successfully process an object they have already admitted and potentially modified. This is true for all mutating admission webhooks, since any change they can make in an object could already exist in the user-provided object, but it is essential for webhooks that opt into reinvocation

Steps to reproduce the bug:

Pick a pod which has already been mutated kubectl get po <some-name> -o yaml > vault-test-pod.yaml
Delete the pod kubectl delete po <some-name>
Re-create the original pod: kubectl apply -f vault-test-pod.yaml
Observe the validation error

The Pod "<some-name>" is invalid:
* spec.volumes[3].name: Duplicate value: "vault-env"
* spec.containers[0].volumeMounts[3].mountPath: Invalid value: "/vault/": must be unique
* spec.initContainers[1].name: Duplicate value: "copy-vault-env"

Additional context:
Code shows that container and volume is appended if criterias match, but does not take into account if it has already been appended:
https://github.com/banzaicloud/bank-vaults/blob/master/pkg/webhook/pod.go#L49
Environment details:

Kubernetes version (e.g. v1.10.2): 1.19.7
Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): k3d
bank-vaults version (e.g. 0.4.17): 1.10.1
Install method (e.g. helm or static manifests): helm
Logs from the misbehaving component (and any other relevant logs): see reproduce steps
Resource definition (possibly in YAML format) that caused the issue, without sensitive data: any - see reproduce steps

/kind bug

@jasmdk Hi, I am running into same issue. By any chance do you fix for this issue?

Totally valid I will push a fix for this shortly.

https://github.com/banzaicloud/bank-vaults/pull/1419 is incoming.


Add Test Directory for VScraper

Idea
Add a test directory to fjv4.com filled with the different file types that VScraper should be able to scrape.

Update
Added links to files. Next thing to do is actually add the files I'm linking to

Update
Added code files

Update
Adding <img> files

Forgot about this


Empty array when visualizing some of OakInk-Shape data

I am trying to run viz_oakink_shape.py. But for some categories such as apple with intents [use, hold, liftup, handover] and split train, I get the following error: What are some possible reasons for this? Maybe there is no hand present for grasp?
  File "viz_oakink_shape.py", line 53, in <module>
    main(arg)
  File "viz_oakink_shape.py", line 28, in main
    oi_shape = OakInkShape(category=category, intent_mode=intent, data_split=split)
  File "/home/snarasimhaswamy/OakInk/oikit/oi_shape/oi_shape.py", line 126, in __init__
    batch_hand_shape = torch.from_numpy(np.stack(batch_hand_shape))
  File "<__array_function__ internals>", line 180, in stack
  File "/opt/conda/envs/oakink/lib/python3.8/site-packages/numpy/core/shape_base.py", line 422, in stack
    raise ValueError('need at least one array to stack')
ValueError: need at least one array to stack

Currently, we randomly select 80% of objects for training, 10% for validation, and the rest 10% for testing. The apple category has only one CAD model, the C90001. And unfortunately, this object is randomly assigned to the test set. Therefore, you can not find any grasps of an apple in the train split.
You can use OakInkShape(category="apple", data_split="test") or just OakInkShape(category="apple") to visualize the grasps of the apple.

Thank you for the quick response!

I'm trying to run viz_oakink_shape.py too.
But to me, this error occurs in all categories. :(

Sorry to hear about the issue.
This typically shouldn’t happen.
Please ensure OAKINK_DIR is set correctly and the dataset is extracted and organized as per the documentation.


Bump JS samples' dependencies in samples-work-in-progress branch to 4.4.0

Describe the bug
~As part of the initial work, we bumped the version to 4.3.0, which isn't a listed version on NPM, as such we should go to 4.3.4.~
We want to go to 4.4.0, because we'll be releasing the updated samples at  //BUILD.
[bug]

pushed changes to samples-work-in-progress


Row wise sum in 2D array

Hello, I would like to contribute to this project by adding the code for the row-wise sum of the 2D array in java language. Please assign it to me.

Please assign this to me in C++

Please do assign this to me.


[vm] Implement push, pop, borrow for native vector.

Motivation
Implement push_back, pop, and borrow for native vectors. This will give us the MVP for collection functionality needed by the validator set contracts. I added one error code to represent errors in the Native Functions, which was treated as abort previously, which thus changed the tests for the signature checking primitive tests.
Have you read the Contributing Guidelines on pull requests?
Yes
Test Plan
Added tests for all runtime errors for vector.
Related PRs
#869

:umbrella: The latest upstream changes (presumably 2f613a945fd3ed7b8776740227707bf712d05b1a) made this pull request unmergeable. Please resolve the merge conflicts.

:umbrella: The latest upstream changes (presumably e47969f8eb98c6f87ec45e211e58a62bb83aa95d) made this pull request unmergeable. Please resolve the merge conflicts.

@bors-libra r=sblackshear

:pushpin: Commit 9505dd4 has been approved by sblackshear

:hourglass: Testing commit 9505dd43aeab567936dbccedf980a28d7bd9dbd4 with merge 1aa769ce9b9ba6691350b0d1bc99c5ff41816ed7...

:sunny: Test successful - checks-circle_commit_workflow
Approved by: sblackshear
Pushing 1aa769ce9b9ba6691350b0d1bc99c5ff41816ed7 to master...


Add rendered examples to ResponseDefinition#describe

Signed-off-by: Dane Jensen<EMAIL_ADDRESS>

LGTM


Error in scheduler.ts

Check if clearIdleCallback is defined before calling it
see https://github.com/lineupjs/lineupjs/blob/93e18c26a61797342b4953a2d6b6a9558ce4a177/src/internal/scheduler.ts#L232-L233
This causes the error

https://github.com/lineupjs/lineupjs/pull/521 will remove this code chunk


Find a way to update the version number in code

Currently version numbers are managed by semantic release, however, I'd like to have it in the code too so that I can log client version information

This is addressed in a branch on my fork, using rollup's replace plugin:
https://github.com/chriswilty/applause-button/compare/feature/a11y...chriswilty:applause-button:chore/package-version


Finder failed to analy

Hi Sagnik
I'm trying to run finder on the provided example data. However, it's just failing and trowing the following error.
Any help is appreciated.
finder -no_cleanup -mf test_metadata.csv -n 30 -gdir_star $PWD/star_index_without_transcriptome -out_dir $PWD/FINDER_test_ARATH -g $PWD/Arabidopsis_thaliana.TAIR10.dna_sm.toplevel.fa -p $PWD/uniprot_ARATH.fasta -gdir_olego olego_index -preserve
[E::hts_open_format] Failed to open file /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_olego_round5.sorted.bam
samtools merge: fail to open "/data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_olego_round5.sorted.bam": No such file or directory
[E::hts_open_format] Failed to open file /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_olego_round5.sorted.bam
samtools merge: fail to open "/data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_olego_round5.sorted.bam": No such file or directory
[E::hts_open_format] Failed to open file /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam
samtools index: failed to open "/data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam": No such file or directory
[E::hts_open_format] Failed to open file /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam
samtools index: failed to open "/data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam": No such file or directory
[E::hts_open_format] Failed to open file /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam
samtools index: failed to open "/data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam": No such file or directory
[E::hts_open_format] Failed to open file /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam
samtools index: failed to open "/data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam": No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
sh: line 1: 148093 Segmentation fault      (core dumped) /data/apps/finder/dep/psiclass_terminal_exon_length_modified//subexon-info /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_introns --noStats > /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_exons
sh: line 1: 148092 Segmentation fault      (core dumped) /data/apps/finder/dep/psiclass_terminal_exon_length_modified//subexon-info /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_introns --noStats > /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_exons
[E::hts_open_format] Failed to open file /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam
samtools view: failed to open "/data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam" for reading: No such file or directory
[E::hts_open_format] Failed to open file /data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam
samtools view: failed to open "/data/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam" for reading: No such file or directory
mv: cannot stat ‘/data/apps/finder/example/FINDER_test_ARATH/assemblies_psiclass_modified/combined/psiclass_output_sample_0.gtf’: No such file or directory
mv: cannot stat ‘/data/apps/finder/example/FINDER_test_ARATH/assemblies_psiclass_modified/combined/psiclass_output_sample_1.gtf’: No such file or directory
mv: cannot stat ‘/data/apps/finder/example/FINDER_test_ARATH/assemblies_psiclass_modified/combined/psiclass_output_vote.gtf’: No such file or directory
Traceback (most recent call last):
File "/data/apps/finder/finder", line 675, in
main()
File "/data/apps/finder/finder", line 636, in main
orchestrateGeneModelPrediction(options,logger_proxy,logging_mutex)
File "/data/apps/finder/finder", line 449, in orchestrateGeneModelPrediction
findTranscriptsInEachSampleNotReportedInCombinedAnnotations(options,logger_proxy,logging_mutex)
File "/data/apps/finder/scripts/findTranscriptsInEachSampleNotReportedInCombinedAnnotations.py", line 17, in findTranscriptsInEachSampleNotReportedInCombinedAnnotations
combined_transcript_info=readAllTranscriptsFromGTFFileInParallel([combined_gtf_filename,"combined","combined"])[0]
File "/data/apps/finder/scripts/fileReadWriteOperations.py", line 279, in readAllTranscriptsFromGTFFileInParallel
fhr=open(gtf_filename,"r")
FileNotFoundError: [Errno 2] No such file or directory: '/data/apps/finder/example/FINDER_test_ARATH/assemblies_psiclass_modified/combined/combined.gtf'
Kind regards

Hello @jb018,
Thank you so much for your interest in finder. The issue seems to be around the Olego aligner. Could you please check that Olego is properly installed within the conda environment? Please issue the command olegoindex and let me know the output.
Thank you.

Thanks for your prompt response @sagnikbanerjee15
olegoindex doesn't show any error. correct me if I'm wrong. Pasting the output as shown below for your reference.
../dep/olego/olegoindex -p olego_index Arabidopsis_thaliana.TAIR10.dna_sm.toplevel.fa
[bwa_index] Pack FASTA... 0.73 sec
[bwa_index] Reverse the packed sequence... 0.28 sec
[bwa_index] Construct BWT for the packed sequence...
[BWTIncConstructFromPacked] 10 iterations done. 29216662 characters processed.
[BWTIncConstructFromPacked] 20 iterations done. 51116486 characters processed.
[BWTIncConstructFromPacked] 30 iterations done. 68388582 characters processed.
[BWTIncConstructFromPacked] 40 iterations done. 82010006 characters processed.
[BWTIncConstructFromPacked] 50 iterations done. 92751542 characters processed.
[BWTIncConstructFromPacked] 60 iterations done. 101221302 characters processed.
[BWTIncConstructFromPacked] 70 iterations done. 107898902 characters processed.
[BWTIncConstructFromPacked] 80 iterations done. 113162758 characters processed.
[BWTIncConstructFromPacked] 90 iterations done. 117311366 characters processed.
[bwt_gen] Finished constructing BWT in 97 iterations.
[bwa_index] 40.41 seconds elapse.
[bwa_index] Construct BWT for the reverse packed sequence...
[BWTIncConstructFromPacked] 10 iterations done. 29216662 characters processed.
[BWTIncConstructFromPacked] 20 iterations done. 51116486 characters processed.
[BWTIncConstructFromPacked] 30 iterations done. 68388582 characters processed.
[BWTIncConstructFromPacked] 40 iterations done. 82010006 characters processed.
[BWTIncConstructFromPacked] 50 iterations done. 92751542 characters processed.
[BWTIncConstructFromPacked] 60 iterations done. 101221302 characters processed.
[BWTIncConstructFromPacked] 70 iterations done. 107898902 characters processed.
[BWTIncConstructFromPacked] 80 iterations done. 113162758 characters processed.
[BWTIncConstructFromPacked] 90 iterations done. 117311366 characters processed.
[bwt_gen] Finished constructing BWT in 97 iterations.
[bwa_index] 40.56 seconds elapse.
[bwa_index] Update BWT... 0.28 sec
[bwa_index] Update reverse BWT... 0.31 sec
[bwa_index] Construct SA from BWT and Occ... 6.26 sec
[bwa_index] Construct SA from reverse BWT and Occ... 6.29 sec
====================================
Just checking the file size of olego index otputs (seems to be fine).
ls olego_index.* -Ssh
43M olego_index.bwt   43M olego_index.rbwt   29M olego_index.pac   29M olego_index.rpac   15M olego_index.rsa   15M olego_index.sa  7.5K olego_index.amb  1.0K olego_index.ann
Thanks

Hmm. Looks like the installation was successful. Could you send me a screenshot of the ls -lhrt for the bam files in the output directory.
Thank you.

Please find as shown below
ls FINDER_test_ARATH/alignments/ -lhrt
total 174K
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:02 dummy_data1_round1.error
-rw-rw-r-- 1 jb018 jb018  772 Nov 22 00:03 dummy_data1_round1_SJ.out.tab
-rw-rw-r-- 1 jb018 jb018  30K Nov 22 00:03 dummy_data1_round1_Aligned.sortedByCoord.out.bam
-rw-rw-r-- 1 jb018 jb018 2.0K Nov 22 00:03 dummy_data1_round1_Log.final.out
-rw-rw-r-- 1 jb018 jb018  15K Nov 22 00:03 dummy_data1_round1_Unmapped.out.mate1
-rw-rw-r-- 1 jb018 jb018  239 Nov 22 00:03 dummy_data1_round1.output
drwx------ 3 jb018 jb018    0 Nov 22 00:03 dummy_data1_round1__STARtmp
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_round1.error
-rw-rw-r-- 1 jb018 jb018  772 Nov 22 00:03 dummy_data2_round1_SJ.out.tab
-rw-rw-r-- 1 jb018 jb018  30K Nov 22 00:03 dummy_data2_round1_Aligned.sortedByCoord.out.bam
-rw-rw-r-- 1 jb018 jb018 2.0K Nov 22 00:03 dummy_data2_round1_Log.final.out
-rw-rw-r-- 1 jb018 jb018  15K Nov 22 00:03 dummy_data2_round1_Unmapped.out.mate1
-rw-rw-r-- 1 jb018 jb018  239 Nov 22 00:03 dummy_data2_round1.output
drwx------ 3 jb018 jb018    0 Nov 22 00:03 dummy_data2_round1__STARtmp
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 leaf_round1_SJ.out.tab
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data1_round2.error
drwx------ 2 jb018 jb018    0 Nov 22 00:03 dummy_data1_round2__STARgenome
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data1_round2_SJ.out.tab
-rw-rw-r-- 1 jb018 jb018  932 Nov 22 00:03 dummy_data1_round2_Aligned.sortedByCoord.out.bam
-rw-rw-r-- 1 jb018 jb018 2.0K Nov 22 00:03 dummy_data1_round2_Log.final.out
-rw-rw-r-- 1 jb018 jb018  15K Nov 22 00:03 dummy_data1_round2_Unmapped.out.mate1
-rw-rw-r-- 1 jb018 jb018  239 Nov 22 00:03 dummy_data1_round2.output
drwx------ 3 jb018 jb018    0 Nov 22 00:03 dummy_data1_round2__STARtmp
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_round2.error
drwx------ 2 jb018 jb018    0 Nov 22 00:03 dummy_data2_round2__STARgenome
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_round2_SJ.out.tab
-rw-rw-r-- 1 jb018 jb018  932 Nov 22 00:03 dummy_data2_round2_Aligned.sortedByCoord.out.bam
-rw-rw-r-- 1 jb018 jb018 2.0K Nov 22 00:03 dummy_data2_round2_Log.final.out
-rw-rw-r-- 1 jb018 jb018  15K Nov 22 00:03 dummy_data2_round2_Unmapped.out.mate1
-rw-rw-r-- 1 jb018 jb018  239 Nov 22 00:03 dummy_data2_round2.output
drwx------ 3 jb018 jb018    0 Nov 22 00:03 dummy_data2_round2__STARtmp
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 leaf_round2_SJ.out.tab
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 leaf_round1_and_round2_SJ.out.tab
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data1_round3.error
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data1_round3_SJ.out.tab
-rw-rw-r-- 1 jb018 jb018  886 Nov 22 00:03 dummy_data1_round3_Aligned.sortedByCoord.out.bam
-rw-rw-r-- 1 jb018 jb018 2.0K Nov 22 00:03 dummy_data1_round3_Log.final.out
-rw-rw-r-- 1 jb018 jb018  15K Nov 22 00:03 dummy_data1_round3_Unmapped.out.mate1
-rw-rw-r-- 1 jb018 jb018  239 Nov 22 00:03 dummy_data1_round3.output
drwx------ 3 jb018 jb018    0 Nov 22 00:03 dummy_data1_round3__STARtmp
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_round3.error
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_round3_SJ.out.tab
-rw-rw-r-- 1 jb018 jb018  886 Nov 22 00:03 dummy_data2_round3_Aligned.sortedByCoord.out.bam
-rw-rw-r-- 1 jb018 jb018 2.0K Nov 22 00:03 dummy_data2_round3_Log.final.out
-rw-rw-r-- 1 jb018 jb018  15K Nov 22 00:03 dummy_data2_round3_Unmapped.out.mate1
-rw-rw-r-- 1 jb018 jb018  239 Nov 22 00:03 dummy_data2_round3.output
drwx------ 3 jb018 jb018    0 Nov 22 00:03 dummy_data2_round3__STARtmp
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 leaf_round3_SJ.out.tab
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 leaf_round1_and_round2_and_round3_SJ.out.tab
-rw-rw-r-- 1 jb018 jb018  273 Nov 22 00:03 dummy_data1_round5.error
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data1_olego_round5.sorted.bam2
-rw-rw-r-- 1 jb018 jb018  273 Nov 22 00:03 dummy_data2_round5.error
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_olego_round5.sorted.bam2
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_introns
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data1_introns
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data1_introns.bed
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_introns.bed
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_exons
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data1_exons
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data1_exons.bed
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_exons.bed
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_num_exons_in_intron
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data1_num_exons_in_intron
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data1_final.sortedByCoord.out.sam
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_final.sortedByCoord.out.sam
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data2_for_psiclass.sam
-rw-rw-r-- 1 jb018 jb018    0 Nov 22 00:03 dummy_data1_for_psiclass.sam
-rw-rw-r-- 1 jb018 jb018   71 Nov 22 00:03 dummy_data1_for_psiclass.bam
-rw-rw-r-- 1 jb018 jb018   71 Nov 22 00:03 dummy_data2_for_psiclass.bam
-rw-rw-r-- 1 jb018 jb018  345 Nov 22 00:03 mapping_stats.csv
===============================================
cat  FINDER_test_ARATH/alignments/mapping_stats.csv
Bioproject,Run,condition,Total reads,Round1_umr,Round1_mmr,Round1_tmr,Round2_umr,Round2_mmr,Round2_tmr,Round3_umr,Round3_mmr,Round3_tmr,round4_umr,round4_mmr,round4_tmr
Dummy1,dummy_data1,leaf,250,0.76,0.04,0.81,0.76,0.04,0.81,0.76,0.04,0.81,0.76,0.04,0.81
Dummy2,dummy_data2,leaf,250,0.76,0.04,0.81,0.76,0.04,0.81,0.76,0.04,0.81,0.76,0.04,0.81

Hello @jb018,
Thank you for posting the output. I noticed that the bam file output dummy_data2_olego_round5.sorted.bam2 has a 2 in the end which is leading to the failure. Could you do a git pull and rerun it? I fixed this issue about a week back.
Please let me know if you encounter any more issues. Just to let you know, I am working to release a new version as a docker image. Hopefully, that will address most of these issues.
Thank you.

Hi @sagnikbanerjee15
Yes, I took a notice on the recent release and updated it too. However, still it fails. I did git pull, it also confirm its up to date.
git pull
Already up-to-date.git pull
Many thanks

Hi @jb018,
Thanks for your reply. Could you please remove the folder and do a fresh download?
Thanks.

Hi @sagnikbanerjee15
I re-installed from scratch and run it again, but I get the same error
example]$ ../finder -no_cleanup -mf Arabidopsis_thaliana_metadata.csv -n $CPU -gdir_star $PWD/star_index_without_transcriptome -out_dir $PWD/FINDER_test_ARATH -g $PWD/Arabidopsis_thaliana.TAIR10.dna_sm.toplevel.fa -p $PWD/uniprot_ARATH.fasta -gdir_olego olego_index -preserve
[E::hts_open_format] Failed to open file /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_olego_round5.sorted.bam
samtools merge: fail to open "/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_olego_round5.sorted.bam": No such file or directory
[E::hts_open_format] Failed to open file /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_olego_round5.sorted.bam
samtools merge: fail to open "/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_olego_round5.sorted.bam": No such file or directory
[E::hts_open_format] Failed to open file /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam
samtools index: failed to open "/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam": No such file or directory
[E::hts_open_format] Failed to open file /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam
samtools index: failed to open "/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam": No such file or directory
[E::hts_open_format] Failed to open file /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam
samtools index: failed to open "/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam": No such file or directory
[E::hts_open_format] Failed to open file /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam
samtools index: failed to open "/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam": No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
sh: line 1: 52497 Segmentation fault      (core dumped) /apps/finder/dep/psiclass_terminal_exon_length_modified//subexon-info /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_introns --noStats > /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_exons
sh: line 1: 52496 Segmentation fault      (core dumped) /apps/finder/dep/psiclass_terminal_exon_length_modified//subexon-info /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_introns --noStats > /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_exons
[E::hts_open_format] Failed to open file /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam
samtools view: [E::hts_open_format] failed to open "/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data1_final.sortedByCoord.out.bam" for reading: No such file or directory
Failed to open file /apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam
samtools view: failed to open "/apps/finder/example/FINDER_test_ARATH/alignments/dummy_data2_final.sortedByCoord.out.bam" for reading: No such file or directory
mv: cannot stat ‘/apps/finder/example/FINDER_test_ARATH/assemblies_psiclass_modified/combined/psiclass_output_sample_0.gtf’: No such file or directory
mv: cannot stat ‘/apps/finder/example/FINDER_test_ARATH/assemblies_psiclass_modified/combined/psiclass_output_sample_1.gtf’: No such file or directory
mv: cannot stat ‘/apps/finder/example/FINDER_test_ARATH/assemblies_psiclass_modified/combined/psiclass_output_vote.gtf’: No such file or directory
Traceback (most recent call last):
File "../finder", line 675, in
main()
File "../finder", line 636, in main
orchestrateGeneModelPrediction(options,logger_proxy,logging_mutex)
File "../finder", line 449, in orchestrateGeneModelPrediction
findTranscriptsInEachSampleNotReportedInCombinedAnnotations(options,logger_proxy,logging_mutex)
File "/apps/finder/scripts/findTranscriptsInEachSampleNotReportedInCombinedAnnotations.py", line 17, in findTranscriptsInEachSampleNotReportedInCombinedAnnotations
combined_transcript_info=readAllTranscriptsFromGTFFileInParallel([combined_gtf_filename,"combined","combined"])[0]
File "/apps/finder/scripts/fileReadWriteOperations.py", line 279, in readAllTranscriptsFromGTFFileInParallel
fhr=open(gtf_filename,"r")
FileNotFoundError: [Errno 2] No such file or directory: '/apps/finder/example/FINDER_test_ARATH/assemblies_psiclass_modified/combined/combined.gtf'
Any help is appreciated !!
Thanks

Hello @jb018,
Well, this is unexpected. Could you please send me the output folder and the folder you downloaded? You can email me those to<EMAIL_ADDRESS>I would have created a docker image to eliminate these problems but the current version of finder depends on a few software that are not permitted to be redistributed.
Thank you.

Thanks @sagnikbanerjee15
Just sent you an email.
Many thanks


runtime: interface is nil, not *testing.T

Random tests have started failing with "interface conversion: interface {} is nil, not *testing.T" or "interface conversion: interface is nil, not *testing.T". The first was http://build.golang.org/log/d1c8715084bdae5d80214470404aa422ac875dd4.
This flake happens enough that it's quite likely it happened within a few commits of the first occurrence and the commit where it first happened seems like a likely suspect: 8e496f1 runtime: simplify buffered channels (@randall77).
First observed: 2015-11-05T15:41:05 8e496f1
Last observed:  2015-11-05T21:29:17 b4447a1
Failure probability: 44% (8 of 17 commits)
Chance failure is still happening: 1 in 6
50% chance failure started within 1 commit of first failure:
8e496f1 runtime: simplify buffered channels.
7bb2a7d cmd/dist: remove vestigial -s flag
95% chance failure started within 5 commits of first failure:
61ca7e5 crypto/x509: add /etc/ssl/certs to certificate directories
9496815 cmd/go: put all generate variables in the environment
b16c699 internal/syscall/unix: eliminate non-trivial randomTrap initializer
dcd9e5b runtime: make putfull start mark workers
No past failures

I should add that the line that fails is out := (<-t.signal).(*T).

Finally got a decent repro case for this.  Looks like a bad combination of channel ops and stack copy (shrink) of the receiver.  Somehow the receiver stack gets copied away just as the sender is trying to send to it.