aws-sdk-cpp: 1.4.70 -> 1.4.78

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools.
This update was made based on information from https://repology.org/metapackage/aws-sdk-cpp/versions.
These checks were done:

built on NixOS

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 1.4.78 with grep in /nix/store/5jxaryh2w692r3a20nijn5q7zc2xd43v-aws-sdk-cpp-1.4.78

directory tree listing: https://gist.github.com/9e9bda76c45da3aee968472e112d51dc

du listing: https://gist.github.com/890b235c67a7057ba2a51cd58db3c0fa

meta.description for aws-sdk-cpp is: '"A C++ interface for Amazon Web Services"'.
cc @edolstra for testing.
This build is cached with Cachix. To use the cache follow these instructions:
One time setup in nixpkgs Git checkout:
cachix use r-ryantm
git remote add r-ryantm https://github.com/r-ryantm/nixpkgs.git

Test this build:
git fetch r-ryantm && git checkout 6d2bc746f3f55a810735acdc44dff3b7799b1782 && nix-shell --pure -I nixpkgs=. -p aws-sdk-cpp

Failure on aarch64-linux (full log)
Attempted: aws-sdk-cpp
Partial log (click to expand)
[ 60%] Building CXX object aws-cpp-sdk-ec2/CMakeFiles/aws-cpp-sdk-ec2.dir/source/model/VpnConnectionOptions.cpp.o
[ 60%] Building CXX object aws-cpp-sdk-ec2/CMakeFiles/aws-cpp-sdk-ec2.dir/source/model/VpnConnectionOptionsSpecification.cpp.o
[ 60%] Building CXX object aws-cpp-sdk-ec2/CMakeFiles/aws-cpp-sdk-ec2.dir/source/model/VpnConnection.cpp.o
[ 60%] Building CXX object aws-cpp-sdk-ec2/CMakeFiles/aws-cpp-sdk-ec2.dir/source/model/VpcTenancy.cpp.o
[ 60%] Building CXX object aws-cpp-sdk-ec2/CMakeFiles/aws-cpp-sdk-ec2.dir/source/model/VpcState.cpp.o
[ 60%] Linking CXX shared library libaws-cpp-sdk-ec2.so
[ 60%] Built target aws-cpp-sdk-ec2
make: *** [Makefile:130: all] Error 2
builder for '/nix/store/r61yh89pyj8yri4gqrdvyyxmfnfxvbap-aws-sdk-cpp-1.4.78.drv' failed with exit code 2
error: build of '/nix/store/r61yh89pyj8yri4gqrdvyyxmfnfxvbap-aws-sdk-cpp-1.4.78.drv' failed

Failure on x86_64-darwin (full log)
Attempted: aws-sdk-cpp
Partial log (click to expand)
[  9%] Building CXX object aws-cpp-sdk-cloudformation/CMakeFiles/aws-cpp-sdk-cloudformation.dir/source/model/AccountGateStatus.cpp.o
[  9%] Building CXX object aws-cpp-sdk-cloudformation/CMakeFiles/aws-cpp-sdk-cloudformation.dir/source/model/AccountLimit.cpp.o
[  9%] Building CXX object aws-cpp-sdk-cloudformation/CMakeFiles/aws-cpp-sdk-cloudformation.dir/source/model/CancelUpdateStackRequest.cpp.o
[  9%] Building CXX object aws-cpp-sdk-cloudformation/CMakeFiles/aws-cpp-sdk-cloudformation.dir/source/model/Capability.cpp.o
[  9%] Building CXX object aws-cpp-sdk-cloudformation/CMakeFiles/aws-cpp-sdk-cloudformation.dir/source/model/Change.cpp.o
[  9%] Building CXX object aws-cpp-sdk-cloudformation/CMakeFiles/aws-cpp-sdk-cloudformation.dir/source/model/ChangeAction.cpp.o
[  9%] Building CXX object aws-cpp-sdk-cloudformation/CMakeFiles/aws-cpp-sdk-cloudformation.dir/source/model/ChangeSetStatus.cpp.o
[  9%] Building CXX object aws-cpp-sdk-cloudformation/CMakeFiles/aws-cpp-sdk-cloudformation.dir/source/model/ChangeSetSummary.cpp.o
building of '/nix/store/2plvlsl9s2v8vcq5mfsi1cg83a75qly6-aws-sdk-cpp-1.4.78.drv' timed out after 1800 seconds
error: build of '/nix/store/2plvlsl9s2v8vcq5mfsi1cg83a75qly6-aws-sdk-cpp-1.4.78.drv' failed

Failure on x86_64-linux (full log)
Attempted: aws-sdk-cpp
Partial log (click to expand)
[  6%] Building CXX object aws-cpp-sdk-batch/CMakeFiles/aws-cpp-sdk-batch.dir/source/model/ContainerOverrides.cpp.o
[  6%] Building CXX object aws-cpp-sdk-batch/CMakeFiles/aws-cpp-sdk-batch.dir/source/model/ContainerProperties.cpp.o
[  6%] Building CXX object aws-cpp-sdk-batch/CMakeFiles/aws-cpp-sdk-batch.dir/source/model/ContainerSummary.cpp.o
[  6%] Building CXX object aws-cpp-sdk-batch/CMakeFiles/aws-cpp-sdk-batch.dir/source/model/CreateComputeEnvironmentRequest.cpp.o
[  6%] Building CXX object aws-cpp-sdk-batch/CMakeFiles/aws-cpp-sdk-batch.dir/source/model/CreateComputeEnvironmentResult.cpp.o
[  6%] Building CXX object aws-cpp-sdk-batch/CMakeFiles/aws-cpp-sdk-batch.dir/source/model/CreateJobQueueRequest.cpp.o
[  6%] Building CXX object aws-cpp-sdk-batch/CMakeFiles/aws-cpp-sdk-batch.dir/source/model/CreateJobQueueResult.cpp.o
[  6%] Building CXX object aws-cpp-sdk-batch/CMakeFiles/aws-cpp-sdk-batch.dir/source/model/DeleteComputeEnvironmentRequest.cpp.o
building of '/nix/store/f9p0zvnbncyra3rm005ri37xfl09pn7d-aws-sdk-cpp-1.4.78.drv' timed out after 3600 seconds
error: build of '/nix/store/f9p0zvnbncyra3rm005ri37xfl09pn7d-aws-sdk-cpp-1.4.78.drv' failed

The only 2 derivations which directly depend on this one are Nix itself and osquery, both of which compiled successfully on my machine.

fww it was a timeout: building of '/nix/store/f9p0zvnbncyra3rm005ri37xfl09pn7d-aws-sdk-cpp-1.4.78.drv' timed out after 3600 seconds
error: build of '/nix/store/f9p0zvnbncyra3rm005ri37xfl09pn7d-aws-sdk-cpp-1.4.78.drv' failed


WIP: send to GEX

Description
Changes to update send-to-gex test utility.
Reviewer Notes
Still needs to be fixed up to use the pkg/cli/ setup that we use for other CLI tools.
Setup
Add a file named helloworld.json with the contents:
{
    "text": "hello world"
}

Then run this command:
go run cmd/send_to_gex/main.go --edi helloworld.json --transaction-name abc --gex-url 'https://gexweba.daas.dla.mil:443/msg_data/submit?channel=TRANSCOM-DPS-MILMOVE-GHG-IN-IGC-RCOM'

Code Review Verification Steps

[ ] Code follows the guidelines for Logging
[ ] The requirements listed in
Querying the Database Safely
have been satisfied.
Any new migrations/schema changes:

[ ] Follow our guidelines for zero-downtime deploys (see Zero-Downtime Deploys)
[ ] Have been communicated to #g-database
[ ] Secure migrations have been tested following the instructions in our docs

[ ] There are no aXe warnings for UI.
[ ] This works in Supported Browsers and their phone views (Chrome, Firefox, IE, Edge).
[ ] Tested in the Experimental environment (for changes to containers, app startup, or connection to data stores)
[ ] User facing changes have been reviewed by design.
[ ] Request review from a member of a different team.
[ ] Have the Jira acceptance criteria been met for this change?

References

Jira story for this change
this article explains more about the approach used.

Screenshots
If this PR makes visible UI changes, an image of the finished UI can help reviewers and casual
observers understand the context of the changes. A before image is optional and
can be included at the submitter's discretion.
Consider using an animated image to show an entire workflow instead of using multiple images. You may want to use GIPHY CAPTURE for this! 📸
Please frame screenshots to show enough useful context but also highlight the affected regions.

Warnings

:warning:
Please add the JIRA issue key to the PR title (e.g. MB-123)

  Generated by :no_entry_sign: dangerJS against 93f5f501a0057c47db43a16709eeb7452e68aeaf

@lynzt - I pushed the changes and updated the PR description for you.

@lynzt - I also updated the GEX URL checking to make it a lot easier to use in the future.


Dependencies strategy for .NET Core switch

Currently AutoFixture follows a strategy when all the dependencies were included to the project in the Packages folder. It works well for the Full Framework because we can use the platform assemblies "for free". However, .NET Core (and .NET Standard) follows a bit different strategy and all the API you use should be installed via NuGet. As result, you have a lot of platform dependencies.
I played with migration of the AutoFixture project to .NET Core and noticed that after the NuGet restore the Packages folder started to take much more size.
Currently: 94MB
After switch to .NET Core: 320MB.
The delta is 226MB. The new size is 3.4 times larger.
Most of the libraries are not specific to our project somehow and, rather, are related to the netstandard1.5 API we are going to target to.
Therefore, it's a good time to revisit our strategy and decide whether we are going to continue the approach we had before. The worst thing is that after new versions of .NET Standard might be released, they might increase all the dependency versions and our repository will blow up again.
Another thing is that after we migrate our projects to the .NET Core build system, they will target both .NET Framework and .NET Standard simultaneously. It will be impossible to keep local cache for .NET Framework only - build will if any dependencies are missing. Therefore, our solution should cover both targets simultaneously.
Personally, I see the following ways to follow:
1. Don't maintain the repository state more
The main reason why we maintained the repository cache is to be able to work without Internet immediately after checkout. We can continue to support work without Internet, however you will need to trigger a build before, so that the required packages are cached in local NuGet cache.
Of course, this solution is not perfect and not all the cases are covered. For instance, if you have a custom branch with different dependencies, you should prefetch packages for it as well.
The benefit is that we will not blow up repository and that will simplify maintenance - given than number of packages now is much larger (about 65 system packages), it might be hard to keep track all them.
2. Try to change the strategy to store nupkg files only
We could try to investigate whether it would be possible to store .nupkg files only. After that when you trigger a NuGet restore, it should unpack files locally. The packed size is much smaller and takes 34.2MB compared to 320MB. However, we should still realize that if we update packages for a couple of time, that will add 100MB to the repository size and next git clone will take much longer.
I'm also not sure that NuGet restore will work fine without Internet even if all the necessary packages are present - that should be investigated.
3. Keep as is
The option is to keep everything as it is. This approach might not work well because .NET Core build system now creates special files in the obj directory during the NuGet restore. These files are not committed, so after you checkout the repository, build will fail even if all the required assemblies are present on file system.
Build will pass only after you trigger the NuGet restore. There is a risk that NuGet might fail to restore packages if Internet connection is missing, even if unpacked packages are present in the Packages folder.

If I'm missing something - add your ways to solve the issue.
Personally, I don't see problems with the first option - it's a common situation today that you should compile after the checkout.
@adamchester, @moodmosaic, @ecampidoglio looking forward to your thoughts.

Option 3 is not possible to do reliably anymore.
I have not even attempted Option 2 anywhere for fear that there are so many packages, however I believe it's a scenario that both Paket and nuget can support (store nupkg files in source control and restore from that local cache). There are so few examples of this "out in the wild"for NuGet and dotnet core that I can't recommend it.
Option 1 is the only course we can take that I'm sure will work. It has my vote for that reason alone.

Yes, from the above options, I think the 1st one might be the easiest. Voting for that one too... 😕

Thanks guys! :wave:
Ok, so the majority voted to follow with the first option - don't maintain repository cache. @ecampidoglio Do you also agree to follow with Option 1, so I can proceed with .NET Core migration?

At this point, nuget restore doesn't seem to work without an Internet connection (see NuGet/Home#2623) so I think we should just go with times. Option 1 gets my vote. 👍


does't work in browser

```Anonymous AMD module used, but it was not included as a dependency``

Hello, and thanks for your feedback.
Are you loading sprintf with requirejs(['sprintf']) or are you simply including the script in your page? I'm not a AMD user myself, but I think you should be doing the former.

i'm not using requirejs.

Ok, let's try something else: what other libraries are you using and how are you using (loading) them?

jquery is the only other one. I'm just using a script tag to load the sprintf.js library.

I think the problem stems from inconsistency around naming. Is it "sprintfjs" or "sprintf-js"? The file itself is called "sprintf.js", which suggests the module name (in AMD terms) should simply be "sprint".
I suggest having a bit of a consolidation of the name and things will all sort themselves out. Also the TypeScript.DefinitelyTyped package is wrongly using a module name of "script-js". It doesn't matter what is chosen, as long as everything is consistent. It would mean the file being "sprintfjs.js", in a "sprintfjs" directory, with the module called "sprintfjs" and the corresponding TypeScript definition being "sprintfjs".

If I'm not mistaken, the current sprintf module is currently wrapping an old version of this library: https://www.npmjs.com/package/sprintf -> https://github.com/maritz/node-sprintf/blob/master/lib/sprintf.js
@maritz can you release the name so that this module can use sprintf uniformly?

As far as I know, not without possibly breaking these https://www.npmjs.com/browse/depended/sprintf
Do you know of any way to make sure a transition would be easy and not break anything?
Sadly people keep using my package despite the disclaimer at the top of the readme. :-(

@maritz as far as I can tell, you started from https://raw.githubusercontent.com/alexei/sprintf.js/554285c46b4ac59e3a88ba83c19324812e9ecd8d/src/sprintf.js and merely added the export lines in your first commit: https://github.com/maritz/node-sprintf/blob/e6cdecfe21c388d0622939cef2158182a96943b9/lib/sprintf.js .  Some of the later commits, like cloning argv, also appear in this repo git log.  So functionality wise, had you continued updating like this repo, you would probably have something similar to the sprintf-js module.  The main feature in sprintf missing from sprintf-js is the object printing.  sprintf has %O which deviates from the sprintf-js %j for functions:
> require('sprintf-js').sprintf("%j", console.log)

''
> require('sprintf').sprintf("%O", console.log)
'[Function: bound ]'

The relevant code can be found here: https://github.com/maritz/node-sprintf/blob/master/lib/sprintf.js#L80 . That functionality has to be added to this module, as well as exposing the object_stringify method.
Assuming that we can put together a PR and @alexei approves, the next step could be:
A) just make sprintf a wrapper that depends on sprintf-js
OR
B) Adjust the sprintf-js api to align with the sprintf API and push the module to sprintf, making sprintf-js the wrapper.
The API difference is pretty minimal: sprintf directly exports the sprintf function whereas sprintf-js exports an object.   For example, in the (A) approach, the sprintf module would be:
var old_sprintf = require('sprintf-js');
var sprintf = old_sprintf.sprintf;
sprintf.sprintf = sprintf;
sprintf.vsprintf = old_sprintf.vsprintf;
module.exports = sprintf;

That functionality has to be added to this module
What are the use cases for it?

This module will be split into two so that it doesn't export an object anymore. That might pave the way to rename it if @maritz agrees to drop the name. But even though I'm generally reckless enough to make a package unavailable for a couple of minutes, I admit I don't know how to do it right.

Just to make this clear: I really don't care about the name, my fork of your project on github or my package on npm.
But I very strongly care about not breaking any other packages/projects.
I don't really understand what the problem is right now or what the proposed solutions are. Sadly I currently have very little capacity to do any work on this.

@maritz the underlying problem is not with your implementation or the current module, it is with some third party bundlers and AMD module loaders that get confused because the main npm module name is sprintf-js but the source file is sprintf.js.  If the two modules were spiritually and functionally irreconcilable we would have to find another workaround, but since https://www.npmjs.com/package/sprintf hasn't been touched in more than 2 years and attempts to redirect people to this module, it makes sense to explore whether the two modules can be merged.
@alexei when I said that it "has" to be added, it is the only functionality present in sprintf that is not in sprintf-js, so adding that functionality is a prerequisite for a unification.
If both of you are in agreement that we should try to reduce the overall confusion surrounding both sprintf and sprintf-js modules, I think the most painless process is:

merge the relevant features from the sprintf module to the sprintf-js module.  I looked at the diff and the main functionality that is in sprintf but not sprintf-js is the %O for rendering objects.  The code block starts here https://github.com/maritz/node-sprintf/blob/master/lib/sprintf.js#L80 .  To maintain 100% backward compatibility with the existing sprintf module, the function would have to be exported as object_stringify as well

fix the modules.  There are 2 ways to do this:

A) change the sprintf module by making it a simple wrapper around sprintf-js.  Based on my cursory understanding of the git changes in the last few years, there were no major breaking changes that would disrupt a user.  The main difference is sprintf exports the sprintf function directly (like how 0.7 version of the library did it) and sprintf-js does not.  This is an easy fix.
B) change the sprintf-js library to align with the API of sprintf and then publish to sprintf directly.  Changing the source here is straightforward: export the sprintf function rather than the object that houses both, then attach sprintf and vsprintf as keys of the sprintf function.
@maritz if all of this seems too much and you are afraid of making changes, could you at least deprecate the module using npm deprecate? https://docs.npmjs.com/cli/deprecate

I got the point on aligning functionality, yet I'm reluctant about adding the object_stringify function which seems like an opinionated way of representing objects. JSON is near & dear to any JS developer and should suffice for most if not all cases. Imho moving it into it's own project would suit it better

@alexei right now sprintf-js does not use the %O conversion:
> require("sprintf-js").sprintf("%O",[1,2,3])
SyntaxError: [sprintf] unexpected placeholder

So this is not breaking existing functionality.  This can coexist with the JSON %j rendering.  And on a somewhat related note, the WHATWG console spec mentions %O and at first blush the object_stringify function satisfies the requirement:  https://console.spec.whatwg.org/#formatter
To answer the question about JSON deficiencies, the biggest problem is with circular structures:
> var x = {}; x.x = x; x;
{ x: [Circular] } // <-- node knows it's a circular reference
> require("sprintf-js").sprintf("%j", x)
TypeError: Converting circular structure to JSON
    at JSON.stringify (<anonymous>)
    at sprintf.format (/usr/local/lib/node_modules/sprintf-js/src/sprintf.js:73:36)
    at Object.sprintf (/usr/local/lib/node_modules/sprintf-js/src/sprintf.js:21:31)
    at repl:1:23
    at realRunInThisContextScript (vm.js:22:35)
    at sigintHandlersWrap (vm.js:98:12)
    at ContextifyScript.Script.runInThisContext (vm.js:24:12)
    at REPLServer.defaultEval (repl.js:313:29)
    at bound (domain.js:280:14)
    at REPLServer.runBound [as eval] (domain.js:293:12)
> require("sprintf").sprintf("%O", x)
'{x: {x: {x: {x: {x: [Object]}}}}}'

Whether object_stringify is doing the right thing here is up for debate, but it is definitely not blowing up with an error in the same way that JSON does.  Again, not saying one is better than the other, but both can coexist since %O is not currently being used


Consolidate Amundsen Microservices

The idea is to (eventually) deprecate the split of amundsenmetadatalibrary, amundsenfrontendlibrary, amundsensearchlibrary and amundsencommon repositories, and merge them all into one mono repository. The split will instead be based on the backend and frontend.
Following will be the repositories (and packages) of Amundsen after the change.

amundsen (frontend, metadata, search, and common repositories)
amundsendatabuilder (Same as today's databuilder repository)

There will not be a separate amundsenmetadatalibrary, amundsenfrontendlibrary, amundsensearchlibrary or amundsencommon packages.
Signed-off-by: verdan<EMAIL_ADDRESS>

Would we still be able to access the code as packages? I am concerned that for anyone running a custom implementation of Amundsen this would be impossible to transition into.

Would we still be able to access the code as packages? I am concerned that for anyone running a custom implementation of Amundsen this would be impossible to transition into.

@allisonsuarez I believe this is a pretty common use case, and a few companies (including ING) does some kind of customizations this way. IMO the way to do such customizations will be the same i.e., you clone one repo as a submodule, and overwrite the files/classes the way you are doing today. (via a bash script etc.).
Instead of maintaining three different submodules and packages, you will be dealing with just one package to have your desired configuration. Of course, the whole migration is backward compatible in a way that there is no change in the data, you can still inject your custom endpoints, you will still be able to replace/overwrite the files and configs. But yes, there will be some manual work to do to move your code from multiple places to a single place, and rework the deployment.
I also believe that this will provide us the opportunity to think of better ways of doing such customizations.
I can think of a few ways to achieve a high level of customizations:

Replace/Overwrite files, the same way we are doing for the frontend today. This is how we at Stemma doing this kind of customizations. https://github.com/stemma-ai/amundsen-custom
Inject/override custom endpoints, the way you want them.
Override the Proxy Class, using the configuration.  Or even write your own proxy.

# private/custom_proxy.py (placed somewhere in a private repo/directory)
from amundsen.metadata.proxies.neo4j import Neo4jProxy
class CustomNeo4jProxy(Neo4jProxy)
	# override the functions based on your needs
	pass

# Set env variable `METADATA_PROXY_PATH` with your custom implementation
$ export METADATA_PROXY_PATH = "priavate.custom_proxy.CustomNeo4jProxy"

# private/upstream/configs/config.py (upstream == OSS amundsen as a submodule)
METADATA_PROXY = os.environ.get('METADATA_PROXY_PATH', 'default_metadata_proxy')

There are a few scenarios for deployment:

different private services use different OSS libraries as dependency. Those deploy service directly EC2 instead of using docker image.
different private services build different packages based on existing docker image, oss repo with overlap files. They deploy on k8s.
besides the two cases, quite a few users build its own proxy classes with private overrided method to support different heuristic.
private API.  the metadata / search service support service  to service authentication which is not accessible before by users.

Could you clarify how to upgrade / migrate without interruption for the above cases?

Sorry, 1 and 2 are not clear to me. Do you mean a company's internal services are using amundsenmetada as dependency and deploying amundsenmetadata directly to EC2
OR
do you mean some companies are deploying amundsenmetadata with some extra dependencies like flaskoidc etc., on EC2 directly?
For 3: https://github.com/amundsen-io/rfcs/pull/30#issuecomment-806243839
For 4: The security model will not change much. If a user is using a proxy or auth system like flaskoidc, then no endpoint will be exposed directly, as the whole system will be behind that. This will be same as how frontend behaves today.

There are a few scenarios for deployment:

different private services use different OSS libraries as dependency. Those deploy service directly EC2 instead of using docker image.
different private services build different packages based on existing docker image, oss repo with overlap files. They deploy on k8s.
besides the two cases, quite a few users build its own proxy classes with private overrided method to support different heuristic.
private API.  the metadata / search service support service  to service authentication which is not accessible before by users.

Could you clarify how to upgrade / migrate without interruption for the above cases?

Sorry, 1 and 2 are not clear to me. Do you mean a company's internal services are using amundsenmetada as dependency and deploying amundsenmetadata directly to EC2
OR
do you mean some companies are deploying amundsenmetadata with some extra dependencies like flaskoidc etc., on EC2 directly?
For 3: #30 (comment)
For 4: The security model will not change much. If a user is using a proxy or auth system like flaskoidc, then no endpoint will be exposed directly, as the whole system will be behind that. This will be same as how frontend behaves today.

For 4, how does it not change? One of the typical use case is that the metadata service is only accessible from frontend or a whitelist service but not by the user (e.g write badge API, call gdpr asset). Now the same api will be accessible the same user if the could access the frontend.

Would we still be able to access the code as packages? I am concerned that for anyone running a custom implementation of Amundsen this would be impossible to transition into.

@allisonsuarez I believe this is a pretty common use case, and a few companies (including ING) does some kind of customizations this way. IMO the way to do such customizations will be the same i.e., you clone one repo as a submodule, and overwrite the files/classes the way you are doing today. (via a bash script etc.).
Instead of maintaining three different submodules and packages, you will be dealing with just one package to have your desired configuration. Of course, the whole migration is backward compatible in a way that there is no change in the data, you can still inject your custom endpoints, you will still be able to replace/overwrite the files and configs. But yes, there will be some manual work to do to move your code from multiple places to a single place, and rework the deployment.
I also believe that this will provide us the opportunity to think of better ways of doing such customizations.
I can think of a few ways to achieve a high level of customizations:

Replace/Overwrite files, the same way we are doing for the frontend today. This is how we at Stemma doing this kind of customizations. https://github.com/stemma-ai/amundsen-custom
Inject/override custom endpoints, the way you want them.
Override the Proxy Class, using the configuration.  Or even write your own proxy.

# private/custom_proxy.py (placed somewhere in a private repo/directory)
from amundsen.metadata.proxies.neo4j import Neo4jProxy
class CustomNeo4jProxy(Neo4jProxy)
	# override the functions based on your needs
	pass

# Set env variable `METADATA_PROXY_PATH` with your custom implementation
$ export METADATA_PROXY_PATH = "priavate.custom_proxy.CustomNeo4jProxy"

# private/upstream/configs/config.py (upstream == OSS amundsen as a submodule)
METADATA_PROXY = os.environ.get('METADATA_PROXY_PATH', 'default_metadata_proxy')

I don't want to keep arguing one package vs multiple packages. I am talking about no downtime service migration. I won't accept if we go from multiple packages to one package that will require messy / huge service migrations that has downtime. This basically put others stop upgrade.
For 1, 2 I know companies do these type of deployment.

To me, putting FE/metadata/search into single package deployment (one pypi, or one docker image) is not followed the long term vision (data discovery + Metadata engine/ platform). In the end, you will want the same metadata service which could be enhanced with push model  (e.g https://github.com/amundsen-io/amundsendatabuilder/pull/431) to serve different Apps with different network rules.

Thanks @verdan for taking a swing at our development experience and architecture issues in this RFC!
In my opinion, although I like the idea of simplifying the architecture, I think this RFC proposes too many changes all at once. Specifically, I think that just moving us into a monorepo with the same package configuration we have right now will fix the issues #1, #2 and #4 as you describe them, while making the migration much less daunting and disruptive.
I am not saying that there is no benefits on evolving our architecture and simplifying deploys, but I think all this will be easier to do step by step after we fix the issues with the development experience we have right now.

Just read this, somewhat of a newb engineer 😅 , but here's some comments w/ our implementation hopefully of use.
On dependencies, we are mainly affected w/ databuilder (we use w/ Airflow so we have it in our reqs.txt). The other services are mainly done w/ Docker outside or CICD and monorepo.
On development, a single service might make it easier (dependencies, share code-base), but admittedly not so much an issue for us, as most of the things in the immediate future don't require tandem implementation across all services (yet though).
We're mostly focused on front-end (if Flask is removed, will there be an OIDC implementation)? Although we run the services behind VPN, we do like having Okta to get usernames, we're looking to expand a single RBAC story across tooling (amundsen/snowflake/looker/airflow).
Probably the next thing we might look at would be developing the metadata service (but nothing heavy, maybe just some endpoints to update metadata directly vs. databuilder jobs, i.e. from Airflow).
I realize I probably don't have a super strong opinion either way. One of the reqs we fulfill though is keeping optionality (i.e. we have kind of hacky thing w/ databuilder to use neo4j 4.x, this was done so we could have the option on switch to SAAS provide like Aura if our Infra team can't manage it in the future. We use AWS for ES because I think it's pretty cheap :) vs eng overhead).

Would we still be able to access the code as packages? I am concerned that for anyone running a custom implementation of Amundsen this would be impossible to transition into.

@allisonsuarez I believe this is a pretty common use case, and a few companies (including ING) does some kind of customizations this way. IMO the way to do such customizations will be the same i.e., you clone one repo as a submodule, and overwrite the files/classes the way you are doing today. (via a bash script etc.).
Instead of maintaining three different submodules and packages, you will be dealing with just one package to have your desired configuration. Of course, the whole migration is backward compatible in a way that there is no change in the data, you can still inject your custom endpoints, you will still be able to replace/overwrite the files and configs. But yes, there will be some manual work to do to move your code from multiple places to a single place, and rework the deployment.
I also believe that this will provide us the opportunity to think of better ways of doing such customizations.
I can think of a few ways to achieve a high level of customizations:

Replace/Overwrite files, the same way we are doing for the frontend today. This is how we at Stemma doing this kind of customizations. https://github.com/stemma-ai/amundsen-custom
Inject/override custom endpoints, the way you want them.
Override the Proxy Class, using the configuration.  Or even write your own proxy.

# private/custom_proxy.py (placed somewhere in a private repo/directory)
from amundsen.metadata.proxies.neo4j import Neo4jProxy
class CustomNeo4jProxy(Neo4jProxy)
	# override the functions based on your needs
	pass

# Set env variable `METADATA_PROXY_PATH` with your custom implementation
$ export METADATA_PROXY_PATH = "priavate.custom_proxy.CustomNeo4jProxy"

# private/upstream/configs/config.py (upstream == OSS amundsen as a submodule)
METADATA_PROXY = os.environ.get('METADATA_PROXY_PATH', 'default_metadata_proxy')

This is how we're doing customizations today as well, so going from three packages to one sounds like it would make life easier. 👍

Override the Proxy Class, using the configuration.  Or even write your own proxy.

I have found this general approach to be the most flexible and powerful, and use it extensively in other apps like Airflow.  Having a configuration system that lets me override classes with my own custom versions without patching or making a copy of the repo just to overwrite some files is really nice.

PTAL at https://github.com/amundsen-io/rfcs/pull/31 -- I've extracted the monorepo portions of this RFC into a single RFC.

Closing this one for now in favor of #31, and will have a separate RFC for the refactoring.


Mypy warns about missing orm_mode when custom from_orm is used

Initial Checks

[X] I have searched GitHub for a duplicate issue and I'm sure this is something new
[X] I have searched Google & StackOverflow for a solution and couldn't find anything
[X] I have read and followed the docs and still think this is a bug
[X] I am confident that the issue is with pydantic  (not my code, or another library in the ecosystem like FastAPI or mypy)

Description
I have a model with a custom implementation of from_orm. Pydantic's mypy plugin is too strict, and warns about a missing orm_mode = True even though it's not required.
The docs don't say anything about defining custom from_orms so maybe this is bad practice that you wish to discourage.
Still, I think it should not give that warning if the model has its own from_orm. (However, this might be tricky to check, since there might be a super().from_orm call inside the custom method.
Example Code
from pydantic import BaseModel

class User(BaseModel):
    id: int

    def from_orm(self):
        return {}

u = User(id=1)
u.from_orm()

# mypy.ini:
[mypy]
plugins = pydantic.mypy

# mypy output:
# a.py:10: error: "User" does not have orm_mode=True

Python, Pydantic & OS Version
pydantic version: 1.9.2 (also tested with current master)
python version: 3.10.4
mypy version: 0.971
platform: Linux
optional deps. installed: ['typing-extensions']

Affected Components

[ ] Compatibility between releases
[ ] Data validation/parsing
[ ] Data serialization - .dict() and .json()
[ ] JSON Schema
[ ] Dataclasses
[ ] Model Config
[ ] Field Types - adding or changing a particular data type
[ ] Function validation decorator
[ ] Generic Models
[ ] Other Model behaviour - construct(), pickling, private attributes, ORM mode
[ ] Settings Management
[X] Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

I think we wait for V2 where this whole thing will be refactored.


Add option for custom endpoint

Could you possibly add an option for custom endpoints so we could use other s3 compatible provider (eg: backblaze,wasabi). The S3Client Constructor has an endpoint field so this should be straightforward.

Hi @jinyus,
can you take a look at the latest release's changes? Let me know if it works as expected 😉

Thanks for implementing this, but I no longer use this app because I wanted something that works on mobile, but I looked at the code and it should work now.


feat: style and fontStyle added in ActionSheet

#1231
style and fontStyle added in ActionSheet for Android.
<Button
              onPress={() =>
                ActionSheet.show(
                  {
                    options: BUTTONS,
                    cancelButtonIndex: CANCEL_INDEX,
                    destructiveButtonIndex: DESTRUCTIVE_INDEX,
                    title: 'Testing ActionSheet',
                    style: {backgroundColor: 'yellow'},
                    fontStyle: {color: 'red'},
                  },
                  (buttonIndex) => {
                    this.setState({clicked: BUTTONS[buttonIndex]});
                  },
                )
              }>
              <Text>Actionsheet</Text>
</Button>

Actually, it'd be really nice if it was possible to set a custom item component. I have a bug where the Icon gets cut in some devices, and can't get it fixed.

It does not work on vue native


Adsk Contrib - Add bit-depth support to apply methods

The pull request implements an important part of #609 .

OCIO currently only accepts floating-point image buffers. However, in many scenarios one is starting with an integer buffer. Furthermore, the first operation often performed is a 1d-LUT. So rather than accepting an integer buffer and doing a fast look-up, OCIO currently converts to float and does interpolation rather than a simple look-up.

Some ground works were already done to ease the bit-depth support integration and reduce the final pull request size. Please refer to #699 and #722 .

When a 1d-LUT is the first operator in the chain, it will be interpolated to the needed bit-depth and then a look-up will be done on the pixels. This gives a big performance boost with no loss of precision.

The implemented approach is now using the first (or last) op in the color transform (i.e. processor instance) to perform the input and output bit-depth conversions when needed.
Public API:
To remove the existing limitation of the OCIOv1 API, the new design provides several improvements:

Ability to specify an input and output ImageDesc supporting bit-depths:
void CPUProcessor::apply(ImageDesc & imgDesc) const;
void CPUProcessor::apply(const ImageDesc & srcImgDesc, ImageDesc & dstImgDesc) const;
Ability to control the optimization (what optimization(s) to use or not), and finalization (fast or exact inverse LUT support):
ConstCPUProcessorRcPtr Processor::getDefaultCPUProcessor() const;
ConstCPUProcessorRcPtr Processor::getOptimizedCPUProcessor(OptimizationFlags oFlags, FinalizationFlags fFlags)  const;
Ability to process diffferent input and output bit-depths:
ConstCPUProcessorRcPtr Processor::getOptimizedCPUProcessor(BitDepth inBitDepth, BitDepth outBitDepth, OptimizationFlags oFlags,  FinalizationFlags fFlags) const;

Reviewers:
I would suggest you start from the public API before digging in the implementation. One could start with the OpenColorIO.h file, the Config class (header + implementation), the Processor class (header + implementation), and the new CPUProcessor class (header + implementation and associated unit tests).
Pseudo-code (from ocioconvert):
OIIO::ImageSpec spec;
OCIO::ImgBuffer img;
...
img.allocate(spec);
...
OCIO::ConstConfigRcPtr config = OCIO::GetCurrentConfig();
OCIO::ConstProcessorRcPtr processor = config->getProcessor(...);
const OCIO::BitDepth bitDepth = OCIO::GetBitDepth(spec);
OCIO::ConstCPUProcessorRcPtr cpuProcessor = processor->getOptimizedCPUProcessor(...);
OCIO::ImageDescRcPtr imgDesc = OCIO::CreateImageDesc(spec, img);
cpuProcessor->apply(*imgDesc);
Measures (RB-1.1 vs branch) using spi-vfx:

Conversion
bit-depths
Improvement

lnf to lgf
F32 to F32
3x

lgf to lnf
F32 to F32
4x

lnf to lgf
UI16 to UI16
9.7x

lgf to lnf
UI16 to UI16
7x

lnf to xyz16
UI16 to UI16
3.4x

The finalization flag provides a way to select the fast or exact inversion mode (refer to #621).

The pull request fixes #401 .

In order to help everyone investigating and/or tracking performance, the pull request adds:

A new app named ocioperf to perform some predefined benchmarks i.e. full image, line-by-line and pixel-per-pixel,
A CPU processing measure in verbose mode to ocioconvert.

This is terribly old, sorry Patrick.
What is currently blocking this? Just reviews?

@scoopxyz  Thanks, only waiting reviews. You could start from the public API to have an overview.

Following the two-weeks rule, here is a friendly reminder that the branch could now be merged this week unless there are some objections.


Is it compatible with nanopore reads?

Dear,
First of all, sorry for asking about nanopore in the first issue when it is clearly indicated that this pipeline is ment for PacBio HiFi reads.
BUT, since it takes as input the assembly and the BAM file, I guess it could run also with an assembly and alignment file created using nanopore reads. So, my question is, do you expect similar efficiency at separating haplotypes within MAGs? Because, a priori, the biggest difference is the lower accuracy of nanopore reads.
As you can imagine, I have some MAGs in my dataset generated using nanopore's long read sequencing and it would be nice to test your approach on these.
Regards,
Cristian

Hello Cristian,
I can't answer on behalf of the developer, but I believe that read quality is the most important attribute here. Basically, the MAGPhase algorithm works when long reads have sufficiently low error rates (<= 1%) to enable accurate SNP calling.
I highly encourage you to test this with your dataset! The great thing about MAGPhase is that it was developed to enable the visualization of reads using IGV to confirm haplotype attribution. This makes it so much easier to validate results and you don't have to rely on "black-box metrics" to see if your dataset generated useable results.
Thanks,
Derek

Dear Derek,
Thanks for your answer!
I will definetly give it a try then. If I observe something worth mentioning, I will come back to you.
Regards,
Cristian


Fix a path handling issue of ProjectConverter3To4

Fixed a filepath handling issue #63377
Bugsquad edit:

Fixes #63377

Thanks! And congrats for your first merged Godot contribution :tada:


⚠ Bofrost.de has degraded performance

In b623fa9, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1382 ms

Resolved: Bofrost.de performance has improved in 92b8acc after 13 minutes.


Automator: update<EMAIL_ADDRESS>in<EMAIL_ADDRESS>

Generated by Automator - 2021-04-20T15:12:15+00:00

@istio-testing: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-security-multicluster-tests_istio_release-1.10
6fad519e6fc368e231225e4fd4ebde2873b0341d
link
/test integ-security-multicluster-tests_istio_release-1.10

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest


Adding dependency of macos-fuse-t

due to deprecation of osxfuse, macos-fuse-t would be good alternative. Cloud you add dependency on formula?
Actually I have no idea about differences btw osxfuse and fuse-t. So, some more tests and checks are needed before adding dependency.

I opened same issue on https://github.com/macos-fuse-t/fuse-t/issues/6 . Please check it out.


V3.8 test (Test for uChat built from MM master)

Chris this is the branch that we can use to test a uChat build from MM master.
It has 4 commits that differ from master. The first two need to be committed back to MM, they're white-labeling thru configs.
The other two are the config files we would pull from our own repository along with all the images files from this PR:
https://github.com/csduarte/uchat-desktop/commit/bea108e
We also need a uChat version of this gif that is displayed while loading. It's a new file:
src/assets/loading.gif

This should be closed.


Running gulp on cloned repo give error "jekyll-sitemap"

Configuration file: C:/Git/ucsb-wsg.github.io/_config.yml

jekyll 3.1.1 | Error:  jekyll-sitemap

I think this might be because jekyll-sitemap isn't in the Gemfile (https://github.com/ucsb-wsg/ucsb-wsg.github.io/blob/master/Gemfile)?

@loganfranken yes and no. I added but still error. I discovered I needed to install "bundler" and run the "bundle" command. It finally installed it and I no longer get the error, but then running gulp does not stand up a server at localhost:4000, which I will open in a new issue.

Ahh, nice, looks like you added Bundler to the list of required tools in that PR too.

Merging #108 should address this one, right?


BLE Security parameter "AuthOpt_u8" doesn't get reset after the BLE was disabled/re-enabled. (IDFGH-12909)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
v4.3
Espressif SoC revision.
ESP32
Operating System used.
Windows
How did you build your project?
VS Code IDE
If you are using Windows, please specify command line type.
None
Development Kit.
custom board
Power Supply used.
USB
What is the expected behavior?
the value of the BLE security parameter "AuthOpt_u8" should be cleared after cycles of BLE enable/disable.
What is the actual behavior?
the value of the BLE security parameter "AuthOpt_u8" is stored between cycles of BLE enabling /disabling.
Steps to reproduce.

Enable BLE
esp_bt_controller_config_t NPI_BT_ControllerConfig_st = BT_CONTROLLER_INIT_CONFIG_DEFAULT();
ErrorCode_s32 = esp_bt_controller_init(&NPI_BT_ControllerConfig_st);
if (ErrorCode_s32 != ESP_OK) {
//error message
}
ErrorCode_s32 = esp_bt_controller_enable(ESP_BT_MODE_BLE);
if (ErrorCode_s32 != ESP_OK) {
//error message
}
ErrorCode_s32 = esp_ble_config_file_path_update((char_t*)NPI_BT_Filename_pac);
if (ErrorCode_s32 != ESP_OK) {
//error message
}
ErrorCode_s32 = esp_bluedroid_init();
if (ErrorCode_s32 != ESP_OK) {
//error message
}
ErrorCode_s32 = esp_bluedroid_enable();
if (ErrorCode_s32 != ESP_OK) {
//error message
}

Set AuthOpt_u8 = ESP_BLE_ONLY_ACCEPT_SPECIFIED_AUTH_ENABLE;

Disable BLE

esp_err_t ErrorCode_s32;

// Before deinit, disable ble first
ErrorCode_s32 = esp_bluedroid_disable();
if (ErrorCode_s32 != ESP_OK) {
//error message
}
// Deinit and free the stack for ble
ErrorCode_s32 = esp_bluedroid_deinit();
if (ErrorCode_s32 != ESP_OK) {
//error message    }
// Disable the bt controller
ErrorCode_s32 = esp_bt_controller_disable();
if (ErrorCode_s32 != ESP_OK) {
//error message    }
// Deinit the bt controller
ErrorCode_s32 = esp_bt_controller_deinit();
if (ErrorCode_s32 != ESP_OK) {
//error message    }

Enable BLE again like in step 1.

The value of  AuthOpt_u8 = ESP_BLE_ONLY_ACCEPT_SPECIFIED_AUTH_ENABLE is still as set before.

This is not the case if you reboot the board.

In order to fix this you need to explicit clear this settings.
...

Debug Logs.
No response
More Information.
No response

The BLE security parameter "AuthOpt_u8" does not get reset after the BLE is disabled and then re-enabled.
Could you please explain what impact this might have on usage if the security parameter "AuthOpt_u8" is not cleared?

Well actually it is a problem.  Now the AuthOpt_u8 from the server are applied to the Client also. And also i am no longer confident that like disabling and enabling again the BLE stack is actually reseting to initial configuration. This is clearly software bug since it disapears if you do a hardware reset.
I do not consider that this issue is done

Thank you for your report. We have now fixed the issue.
The fix has been committed to the master branch with commit ID: 42ca04b5866dbe59a0cbe1f725708db6a75608e6.
Once testing is complete, we will sync it to GitHub and other branch.

0001-fix-ble-bluedroid-Fixed-BLE-set-adv-param-check.zip

Closing the issue as the user did not reply. If you will encounter any problems, feel free to reopen this one or create a new issues.
Cheers!


please ditch the .rvmrc

unless the only supported platform is jruby-1.6.1 with a gemset torquebox

Just pushed the change. I've been meaning to do that for a long time now.


Some Laravel Love

Dear Amazing Team of LlamaCoder,
First let me salut your magic with this great work, but from my personal experience the "Nextjs cold boot time", of course coming from a PHP world.. but the community is massive.
So, it would really be very much appreciated if you do consider supporting the Laravel framework.
Thank you. #php

Could be interesting to add in the future! I included this in my future TODOs in the README so closing for now


Use rbush instead of quadtree

Better performance with rbush library #15

Looks much faster in my project!

Tested it in my project and on dome page. Looks good. :+1:


mechx interact throws error for agent 1

Following the README, I tried fetching data from agent_id 1 on Gnosis
https://aimechs.autonolas.network/mech/0x3504fb5053ec12f748017248a395b4ed31739705/0x714ef7dafa358c7152f6703dd764a1df40d369dcf53275dd2543b0fdbf207298
however I get an error
:~/code/gnosis/mech-client-test$ mechx interact  "Tell me a joke" 1
Chain configuration: gnosis
Traceback (most recent call last):
  File "/home/gabrielfior/miniconda3/envs/mech-client/bin/mechx", line 8, in <module>
[...]
  File "/home/gabrielfior/miniconda3/envs/mech-client/lib/python3.10/site-packages/mech_client/interact.py", line 298, in verify_or_retrieve_tool
    available_tools = fetch_tools(
  File "/home/gabrielfior/miniconda3/envs/mech-client/lib/python3.10/site-packages/mech_client/interact.py", line 327, in fetch_tools
    return response["tools"]
KeyError: 'tools'

I debugged this a bit and found a possible cause - if you look at the agent definition for agent_id 6 (link), you see a tools key, whereas agent_id 1 doesn't have tools in its definition (link).
When I run the code above for agent_id 6, the prompt gets uploaded to IPFS (link below).
Relevant links:
-> Prompt generated for interaction with agent 6 - https://gateway.autonolas.tech/ipfs/f01701220a3901882dc60880c2e8538626f91f57125fdbbe44d6f6ec1c626801b82874512
-> Transaction created for fulfilling the request (failed because it ran out of gas) - https://gnosisscan.io/tx/0x465cddd037733073a1cb4c14abd9c54ac80695f970f53ffd5a47db9fa01a65ea

correct the agent_id 1 doesn't have any tools, you can pass the agent_id(3 or 6 in this case) that has list of tools, thanks!
cc @0xArdi


Cannot load underlying module for 'XCTest'? And How to use pod with one branch?

Version of protoc (protoc --version)
3.0
Version of ProtocolBuffers.framework
ProtoBuf3.0-Swift3.0
.proto file to reproduce

Description
protobuf-swift/plugin/ProtocolBuffers/ProtocolBuffersTests/ErrorHandlingTest.swift:9:8: Cannot load underlying module for 'XCTest'
another question:
How to use pod with branch? like this?
pod 'ProtocolBuffers-Swift', :git => 'https://github.com/alexeyxo/protobuf-swift', :branch => 'ProtoBuf3.0-Swift3.0'
Thanks.

pod 'ProtocolBuffers-Swift'


CLI[Cluster]: Implement cluster setup command

This command needs to install fullstack-cluster-setup chart. We need to ensure we don't attempt to install if it is already installed.

:tada: This issue has been resolved in version 0.12.0 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


small fixes

overloaded Base.oneunit
fixed typo in ScalarOperator
overloaded \(A, B) = inv(A) * B, and same for /

@ChrisRackauckas merge?


make only builds the static library

Running make builds libzeep.a (static library) but not libzeep.so.3.0.3 (dynamic library). The dynamic library is only created when make install is run.
Likewise, if you run make install on a clean repository, it will fail because it can't find libzeep.a, meaning that make install doesn't trigger a static library build.

I've updated the code to include a autoconf configure script along with a new makefile capable of creating whatever was selected by configure.


populate db script

Does this only generate data for rated prompts? Also would be helpful to add active, inactive (upcoming), and inactive (expired) prompts


Specify listen option with none type value

This will avoid slotting a spice connection.

We have to test that with older releases of libvirt (< 2.0). Do not merge

On libvirt 1.2.18 this causes the VM to not have a display: from virt-manager it's not possible to see what the machine is doing or log into it.
It turns out this is caused by the removal of the Autoport attribute. I changed your PR in the following mode:
diff --git a/libvirt/domain_def.go b/libvirt/domain_def.go
index 97485a6..24d2f5a 100644
--- a/libvirt/domain_def.go
+++ b/libvirt/domain_def.go
@@ -23,9 +23,10 @@ type defDomain struct {
        Devices struct {
                Disks             []defDisk             `xml:"disk"`
                NetworkInterfaces []defNetworkInterface `xml:"interface"`
-               Graphics struct {
-                       Type   string `xml:"type,attr"`
-                       Listen struct {
+               Graphics          struct {
+                       Type     string `xml:"type,attr"`
+                       Autoport string `xml:"autoport,attr"`
+                       Listen   struct {
                                Type string `xml:"type,attr"`
                        } `xml:"listen"`
                } `xml:"graphics"`
@@ -107,6 +108,7 @@ func newDomainDef() defDomain {
        domainDef.VCpu.Amount = 1

        domainDef.Devices.Graphics.Type = "spice"
+       domainDef.Devices.Graphics.Autoport = "yes"
    diff --git a/libvirt/domain_def.go b/libvirt/domain_def.go
index 97485a6..24d2f5a 100644
--- a/libvirt/domain_def.go
+++ b/libvirt/domain_def.go
@@ -23,9 +23,10 @@ type defDomain struct {
        Devices struct {
                Disks             []defDisk             `xml:"disk"`
                NetworkInterfaces []defNetworkInterface `xml:"interface"`
-               Graphics struct {
-                       Type   string `xml:"type,attr"`
-                       Listen struct {
+               Graphics          struct {
+                       Type     string `xml:"type,attr"`
+                       Autoport string `xml:"autoport,attr"`
+                       Listen   struct {
                                Type string `xml:"type,attr"`
                        } `xml:"listen"`
                } `xml:"graphics"`
@@ -107,6 +108,7 @@ func newDomainDef() defDomain {
        domainDef.VCpu.Amount = 1

        domainDef.Devices.Graphics.Type = "spice"
+       domainDef.Devices.Graphics.Autoport = "yes"
        domainDef.Devices.Graphics.Listen.Type = "none"

        domainDef.Devices.QemuGAChannel.Type = "unix"
    domainDef.Devices.Graphics.Listen.Type = "none"

        domainDef.Devices.QemuGAChannel.Type = "unix"

This works correctly. libvirt silently drops the <graphics><listen> which is not supported and keeps working as expected.
Can you double check what happens on libvirt 2.0?

Works perfectly under libvirt-2.4.0. Updated pull request to reflect changes.


setup on localhost

Thanks for supplying the code for this, and the work you put into it! I've used JPsychoMorph in our lab for years, so naturally I was/am excited to try this version.
I am trying to setup an instance of webmorph on my local machine (macOS Mojave) and ran into a bit of an issue. I followed the instructions on your README and everything appeared to work, but when I go to webmorph.test or localhost my browser takes me to a webpage that simply states "It works!". When I add :8080 it takes me to the tomcat webpage stating it was successfully installed. I'm not a web developer, so this is fairly new to me. I'm just curious if I overlooked a step, or missed something that might be an easy fix since you went though this before.
I did notice that I am running an newer version of php than what you mention in your docs (7.x vs 5.x), though I'm not sure that should matter?

Oh no, I just saw this!
Local server problems are really difficult to debug remotely. Did you sort it or give up?
Are you not able to use the free web version on my server (webmorph.org)? If you sign up with an academic email you get an account immediately.


A suspected bug of CreateThread API usage

First of all, I am very happy to read the source code of this project.
When I see the implementation of VtInputThread::Start() in /src/host/VtInputThread.cpp (https://github.com/Microsoft/Terminal/blob/35229a775d9e9908a8dfb102e01f49b494b29194/src/host/VtInputThread.cpp#L160) and found:
hThread = CreateThread(nullptr,
                           0,
                           (LPTHREAD_START_ROUTINE)VtInputThread::StaticVtInputThreadProc,
                           this,
                           0,
                           &dwThreadId);

RETURN_LAST_ERROR_IF(hThread == INVALID_HANDLE_VALUE);

And I also read the documentation of the CreateThread API (https://docs.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-createthread) and found:

If the function fails, the return value is NULL. To get extended error information, call GetLastError.

I am curious about that. Maybe this is a bug.
Kenji Mouri

Oh, yea that might be a bug.

There are some implementations which have the same suspected bug.
PtySignalInputThread::Start() in src/host/PtySignalInputThread.cpp
https://github.com/Microsoft/Terminal/blob/35229a775d9e9908a8dfb102e01f49b494b29194/src/host/PtySignalInputThread.cpp#L168


Release version 1.0.0

Description

Version 1.0 major release
Checklist:

[x] I agree with the terms of the Contributor Agreement.
[x] My name is on the list of contributors.
[ ] The changes introduced by this pull request are covered by existing or newly introduced tests.
[ ] I have updated the changelog and added any related issue and pull request numbers for future reference.

More information on the changes to the publish workflow's authentication, see https://docs.pypi.org/trusted-publishers/.

Would also recommend that reviewers have a look at https://github.com/glotzerlab/synced_collections/pull/7, which sets up automated Github Releases. I merged that quickly so that I could verify that the deployment workflows for PyPI (or at least TestPyPI) worked correctly in this PR, but I am happy to edit the workflow as well in this PR.

Opened #9. I'd like to merge that before this PR is merged.

Opened #9. I'd like to merge that before this PR is merged.

Thanks @bdice! Your PR is merged now. Let me know if you'd like to see anything else. No rush, I would like at least one more approval before I merge this PR in any case.

I opened these PRs. Looks like things are passing against the main branch here, so we can merge this PR and release 1.0.0 before or after merging the downstream PRs.

https://github.com/glotzerlab/signac/pull/924
https://github.com/glotzerlab/signac-docs/pull/189


Expansion of unit tests for utils/utils.go

This commit expands the test coverage for the utils/utils.go module.
The utils module uses the 'github.com/spf13/jwalterweatherman' (aka jww)
package for logging. The tests take the approach of examining the log
file that is produced by this module to verify correct behaviour. This
avoids refactoring the utils module.
The log file messages written by the jww module are of the form:
: yyyy/mm/dd <string|error message>
The checkLogFile function checks each of these parts in turn except for
the date string, which is currently ignored. The final part of the log
file format can either be a single error message, or a series of
strings followed by an error message. Both the error message and the
series of strings can be empty strings.
The log file is checked using a combination of the regex package,
along with the bufio scanner type. Each test logs to its own temporary
log file. This is achieved with standard test setup and teardown
functions.
One consequence of these tests is that StopOnErr has been refactored
into call a new unexported function doStopOnErr which contains the bulk
of the original logic. This allows the same testing approach to be used
with StopOnErr as with CheckErr and cutUsageMessage, rather than look at
the exit status code of the test itself.
An unfortunate side effect of this is that the author of the tests must
now know if a log file is expected or not. If doStopOnErr determines
that an empty error message would be written to the log file then
nothing is written. In the context of the tests this means that the log
file created by the test would have no contents. Consequently there
would be nothing for the test to examine. This situation is indicated by
the boolean flag logFileExoected in the testData struct, and processed
by the logFileIsExpectedAndValid function.
Although not ideal this was deemed a reasonable compromise.

Could you do a push -f to trigger new builds?

Sorry all, I merged this prematurely.  Should have waited @owenwaller to git push -f it to trigger a test build, as this is failing on Windows, and I will need to revert this PR for now:
--- FAIL: TestCheckErr (0.01 seconds)
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_773934895: The process cannot access the file because it is being used by another process.
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_063973136: The process cannot access the file because it is being used by another process.
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_960236341: The process cannot access the file because it is being used by another process.
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_764374446: The process cannot access the file because it is being used by another process.
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_580651211: The process cannot access the file because it is being used by another process.
=== RUN TestDoStopOnErr
Logging to C:\Users\appveyor\AppData\Local\Temp\utils_test_504241858
Logging to C:\Users\appveyor\AppData\Local\Temp\utils_test_057641273
Logging to C:\Users\appveyor\AppData\Local\Temp\utils_test_913919812
Logging to C:\Users\appveyor\AppData\Local\Temp\utils_test_001469907
Logging to C:\Users\appveyor\AppData\Local\Temp\utils_test_622529814
Logging to C:\Users\appveyor\AppData\Local\Temp\utils_test_131851645
Logging to C:\Users\appveyor\AppData\Local\Temp\utils_test_839439544
Logging to C:\Users\appveyor\AppData\Local\Temp\utils_test_964376247
Logging to C:\Users\appveyor\AppData\Local\Temp\utils_test_747965610
--- FAIL: TestDoStopOnErr (0.01 seconds)
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_747965610: The process cannot access the file because it is being used by another process.
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_964376247: The process cannot access the file because it is being used by another process.
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_839439544: The process cannot access the file because it is being used by another process.
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_131851645: The process cannot access the file because it is being used by another process.
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_622529814: The process cannot access the file because it is being used by another process.
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_001469907: The process cannot access the file because it is being used by another process.
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_913919812: The process cannot access the file because it is being used by another process.
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_057641273: The process cannot access the file because it is being used by another process.
	utils_test.go:177: Error: Could not remove file "f". Error: remove C:\Users\appveyor\AppData\Local\Temp\utils_test_504241858: The process cannot access the file because it is being used by another process.
FAIL
coverage: 84.2% of statements
FAIL	github.com/spf13/hugo/utils	0.160s

Hi @owenwaller, still here?  Could you please look into it?  Thanks!

@anthonyfok @bep
Hi Guys, Sorry I've just seen this. I've been extremely busy with other things for the last month.
So, I've had a quick look and fairly obviously I don't see this failure on Linux. So my first question is do either of you two have a Windows box we can try something on, as I don't?
But... the file "f" is the temporary file that is passed to the jww logger instance so we can later parse it to check the results. It's created with:
	const logfilename = "utils_test_"
	f, err := ioutil.TempFile(os.TempDir(), logfilename)

That's lines 163, 164 in utils/utils_test.go
The error is suggesting that the file is locked. As you can see from the ci log the file names are all unique - the jww package (helpfully in this case) spits out the filename, but the only process that should ever touch this file is the one that created it which is the test itself.
But, even if every TestXxxx in the set was run in its own process, the file that is used for logging is local to each TestXxxx function. So again is should be ok as only one process is accessing the file.
So I'm kinda stumped at this point. Can someone give me some details of this appveyor thing and what their set up might be? I can see that it's only the appveyor build that falls over with this.
My one short in the dark would be this.... If you look at the filename, it's all rooted on the same path and that path doesn't look to be very unique - i.e .it doesn't appear to have anything like a build number or string that relates it uniquely to hugo. So assuming a large enough scale and a fast enough rate, I guess it's possible for another process/ci run in the appveyor environment to create a file with the same name at the same path. I admit that seeing this it does make my prefix of "utils_test_" look less than ideal. So that can certainly be improved. But as I said this is a shot in the dark at this point.
Owen

@anthonyfok @bep @spf13
Folks in order to take a look at this, I've set-up an account on AppVeyor and am trying building the current master branch via AppVeyor.
Having not used AppVeyor before this seems the simplest test I could do, since the master branch should be golden...
But it's not building - See:https://ci.appveyor.com/project/owenwaller/hugo/build/1.0.23 for my latest attempt.
Also AppVoyer is using go 1.3 wndows/amd64 as you can see from the build output, though I am not sure this is any effect. But I'm using go 1.4.1 locally on Linux and I don't have a problem building this branch.
So can anyone spot anything that I might have missed?
@spf13 Steve can you send me a copy of you AppVeyor settings so I can be sure that I have the same settings as a the build node you are using? Or create an Appveyor.yml and share that.
But at the minute until I get the current master branch to build I'm a bit stuck :(. My plan is then to rebase my original pull request on the latest master and see if I can reproduce the issue.
(I see similar behaviour is I try to build my original utils-test-code that contained the original pull request. So this looks like a Windows/AppVeyor only thing at the minute)
Any help at this stage would be much appreciated.
Owen

I'm looking into this now. I'll try to share with you what I have.
Best,
Steve
--
Steve Francia
http://stevefrancia.com
http://spf13.com
http://twitter.com/spf13
On February 19, 2015 at 8:22:14 AM, Owen Waller<EMAIL_ADDRESS>wrote:
@anthonyfok @bep @spf13
Folks in order to take a look at this, I've set-up an account on AppVeyor and am trying building the current master branch via AppVeyor. So my last commit is: a389268.
Having not used AppVeyor before this seems the simplest test I could do, since the master branch should be golden...
But it's not building - See:https://ci.appveyor.com/project/owenwaller/hugo/build/1.0.23 for my latest attempt.
Also AppVoyer is using go 1.3 wndows/amd64 as you can see from the build output, though I am not sure this is any effect. But I'm using go 1.4.1 locally on Linux and I don't have a problem building this branch.
So can anyone spot anything that I might have missed?
@spf13 Steve can you send me a copy of you AppVeyor settings so I can be sure that I have the same settings as a the build node you are using? Or create an Appveyor.yml and share that.
But at the minute until I get the current master branch to build I'm a bit stuck :(. My plan is then to rebase my original pull request on the latest master and see if I can reproduce the issue.
(I see similar behaviour is I try to build my original utils-test-code that contained the original pull request. So this looks like a Windows/AppVeyor only thing at the minute)
Any help at this stage would be much appreciated.
Owen
—
Reply to this email directly or view it on GitHub.

Ok, @owenwaller here's the YAML file. It's quite short.
version: 1.0.{build}
clone_folder: C:\GOPATH\src\github.com\spf13\hugo
environment:
  GOPATH: C:\GOPATH
build_script:
- go get -v -d -t ./...
test_script:
- go test -v -cover ./...

There may still be problems if appveyor is still using 1.3 since some of the path issues are fixed by bug fixes that were part of 1.4.
I'll try to take a look at this on Windows today.

None of the path issues I fixed was fixed by 1.4 stuff.
@owenwaller if you just push to your utils-test-cod branch, it will trigger a build with Hugo's build setup.

Joel,
Assuming this is the case - and I half suspect you may be correct - then
I see to approaches.
A) ask Appveyor to see if they will officially add support for go 1.4.X
and commit to keeping it up to date, so that's the go v 1.4.X series and
then go 1.5 and onwards.
or
b) Try this as a workaround:
http://help.appveyor.com/discussions/problems/1215-go-14-support
It seems we can install go v1.4 locally - I've not tried this myself
yet. The issue I have with this is that it means we are going to be
dragging a full go install tarball over the net for every build. Which
seems just a little excessive, and that can't be good for appveyor in
terms of bandwidth or costs. And obviously for us it slows the build
down considerably
Owen
On Thu, 2015-02-19 at 08:34 -0800, Joel Scoble wrote:

There may still be problems if appveyor is still using 1.3 since some
of the path issues are fixed by bug fixes that were part of 1.4.
I'll try to take a look at this on Windows today.
—
Reply to this email directly or view it on GitHub.

--
Dr. Owen Waller
Head of Development
Kulawe Limited
Forsyth House
Cormac Square
Belfast
Northern Ireland
BT2 8LA
+44 (0)757 882 9933
<EMAIL_ADDRESS>www.kulawe.com
Company Number: NI604010
VAT Number : GB 122 6319 42

@spf13 - ta for the yml file. I've attached the YML file I'm using - the
only change is that it's pulling from my repo and the default branch (as
specified in the project settings) is my appveyor branch - as you can
see form the log.
OK I've just tried it again and no dice. Results here:
https://ci.appveyor.com/project/owenwaller/hugo/build/1.0.24
Now we are missing packages... and at least "github.com/kr/pty" should
be linux only, as that's the Linux Pseudo TY emulation layer. It's being
dragging in as part of the test code for something else though. I'm
trying to track down just what at the minute.
@bep - ok pushing to my repo public auto triggered the build on my
apveyor account. So that's fine. If you want me to push to the upstream
spf13.hugo repo then I think I have permissions for that, but I'd rather
not push anything to the master branch till we get this sorted out. I'm
happy to push what I currently have to a new branch in Steve's root
repo, if that will also trigger a build based on his settings.
Owen
On Thu, 2015-02-19 at 09:51 -0800, Bjørn Erik Pedersen wrote:

None of the path issues I fixed was fixed by 1.4 stuff.
@owenwaller if you just push to your utils-test-cod branch, it will
trigger a build with Hugo's build setup.
—
Reply to this email directly or view it on GitHub.

--
Dr. Owen Waller
Head of Development
Kulawe Limited
Forsyth House
Cormac Square
Belfast
Northern Ireland
BT2 8LA
+44 (0)757 882 9933
<EMAIL_ADDRESS>www.kulawe.com
Company Number: NI604010
VAT Number : GB 122 6319 42

The build above fails for an old reason that is fixed:
page_permalink_test.go:68: Test 0: Expected abs url: /x/y/z/boofar/, got: x/y/z/boofar/

@owenwaller - as I asked you a couple time already, could you just trigger a new build by doing a push -f?

@bep thanks for the info. I had been thinking about #699, problems with FindCWD() on Windows. and things affected by Go issue 8090
@owenwaller I guess, for now, using the suggested method is the way that we need to do testing if 1.4 is actually needed for test passing. @bep's comment make me wonder if my assumption that it was is true.

@mohae @owenwaller the failure in the current failure above was caused by #878 - there may be other issues, but do a push first.

@bep.
Has a fix for #878 been merged into the hugo master line as of about
09:00 this morning UK time?
Because, I'm just trying to build hugo's master branch at this point. My
pull request #818 shouldn't be in this code (yet) @anthonyfok tried and
reverted it.
So, I'm happy to do a push -f(*) but to the upstream repo (spf13/hugo?)
and do you want this in a new branch? As I've said I'd rather not stuff
up the master branch any more than it currently appears to be. And
besides my current appveyor branch would be identical to the
spf13/hugo/master branch in any case - bar my addition of the
appveyor.yml file
I'm am quite happy to add a dummy commit onto the spf13/hugo.master
branch if that's all it takes to trigger an appveyor build on spf13's
repo, if that's what is required here.
(*)Not that the -f switch should be necessary. My push should not be
overwriting anything and everyone's local branches should be an ancestor
of remote master branch in any case.
Owen
On Thu, 2015-02-19 at 10:18 -0800, Bjørn Erik Pedersen wrote:

@mohae @owenwaller the failure in the current failure above was caused
by #878 - there may be other issues, but do a push first.
—
Reply to this email directly or view it on GitHub.

--
Dr. Owen Waller
Head of Development
Kulawe Limited
Forsyth House
Cormac Square
Belfast
Northern Ireland
BT2 8LA
+44 (0)757 882 9933
<EMAIL_ADDRESS>www.kulawe.com
Company Number: NI604010
VAT Number : GB 122 6319 42

@owenwaller do a rebase against master, then push -f

@owenwaller git rebase master + git push -f. I'm asking you to do something that takes 20 keystrokes. Do it, or don't.

@owenwaller we use a rebase approach, not merge. I believe it keeps things much cleaner and writes history in the order the commits are applied to the trunk, not in the order they are committed.
I'm sorry this issue has been such a prickly one. How can I help? I don't have access to a windows box at the moment (I'm traveling), but would be happy to help in another way.

I added a request for AppVeyor to upgrade to 1.4 on http://help.appveyor.com/discussions/problems/1215-go-14-support. They said they would update this week.
But that won't address the failures within these tests on Windows.
First I added a sleep to teardown() before removing the file, which didn't work.
I then:

Created a tempDir using init() and placed all the testing logfiles in there.
Changed teardown() do an os.RemoveAll() on the temp directory.
Changed setup() to return the *os.File in addition to the filename.
Added a f.Close() to the returned file handle in the testing code.
Added jww.DiscardLogging() before calling `f.Close().
Created a main test function which ran the tests and after all tests were run called teardown().

These changes, which were made incrementally, also failed due to the files still being in use.
The root of the problem is that jww never closes any files. It relies on the program exiting to close the filies. It doesn't appear that these tests will pass on Windows until there is a way of telling jww to close its log files.
For jww, I think that calling DiscardLogging() might be an appropriate place to close files, or any other function which allows for the Handle to be changed.
Regardless, I don't think there is a way for these tests, as currently written, to pass until there are changes on the jww side.

We can add ways to close things on jww. Do you want to send a PR?

I've made some local changes to support closing things on jww, but I'm still getting inconsistent and strange results when running the tests on Windows. I'm trying to figure out the cause/if my assumption about the root cause of the failure is correct.

@mohae Good man. I had a very quick glance at jww yesterday when I
first became aware that my new tests failed on Windows. I did wonder
about how the log file was closed but I had thought my call to
os.Remove() in my teardown() method would have been sufficient. But
having just looked at the src for os.Remove that's a nope.
So I've been relying on the program exit behaviour on Linux to close the
file implicitly. Which is why I've not seen this problem. So the code is
defiantly missing a call to f.Close somewhere. So that’s my goof.
So I agree with you, the tests as written won't work. So I need to fix
these.
Now the question is where does the Close() go? My current feeling
would be in jww somewhere as that keeps the "just use it" spirit of
jww.
What does everyone else think?
Also @mohae, I'm assuming you are running this on Windows? ;) If this
the case can you do me a favour. Take a fresh clone of the master repo
and see if the current master branch builds. I'm also trying to work out
why Appveyor is apparently failing at the build stage - see my latest
Appveyor build result above. As that should (in theory work). At the
minute I can't work out if the build failure is a Windows or an Appveyor
thing.
Owen
On Thu, 2015-02-19 at 13:32 -0800, Joel Scoble wrote:

I added a request for AppVeyor to upgrade to 1.4 on
http://help.appveyor.com/discussions/problems/1215-go-14-support. They
said they would update this week.
But that won't address the failures within these tests on Windows.
First I added a sleep to teardown() before removing the file, which
didn't work.
I then:
  * Created a tempDir using init() and placed all the testing
    logfiles in there.
  * Changed teardown() do an os.RemoveAll() on the temp directory.
  * Changed setup() to return the *os.File in addition to the
    filename.
  * Added a f.Close() to the returned file handle in the testing
    code.
  * Added jww.DiscardLogging() before calling `f.Close().
  * Created a main test function which ran the tests and after all
    tests were run called teardown().

These changes, which were made incrementally, also failed due to the
files still being in use.
The root of the problem is that jww never closes any files. It relies
on the program exiting to close the filies. It doesn't appear that
these tests will pass on Windows until there is a way of telling jww
to close its log files.
For jww, I think that calling DiscardLogging() might be an appropriate
place to close files, or any other function which allows for the
Handle to be changed.
Regardless, I don't think there is a way for these tests, as currently
written, to pass until there are changes on the jww side.
—
Reply to this email directly or view it on GitHub.

--
Dr. Owen Waller
Head of Development
Kulawe Limited
Forsyth House
Cormac Square
Belfast
Northern Ireland
BT2 8LA
+44 (0)757 882 9933
<EMAIL_ADDRESS>www.kulawe.com
Company Number: NI604010
VAT Number : GB 122 6319 42

@owenwaller Windows does not allow open files to be deleted.
I added a CloseLogFile() function that to jww that checks to see if the io.Writer is an *os.File and closes the file if it is. The LogHandle is set to ioutil.Discard prior to file.Close().
With this change, I was still getting "the process cannot access the file because it is being used by another process" error. This error persisted regardless of what hoops I had jww jump through. Also, some files would be removeable. Defering the Close seemed to affect the success rate, but numerours files were still being flagged as in use.
I added a test to jww to test the new close function and the test was able to remove the closed LogFile.
There is something going on with the code in utils_test.go that is contributing to this issue. I'm not sure what it is yet.

@owenwaller I think you have some extraneous packages in your repo. Notice the "cannot find package..." errors. I don't think those are in hugo. Also, Hugo builds fine for me on Windows.

Had a quick look at this (hate puzzles).
@mohae - you can do jww.LogHandle.(*os.File).Close() - and you don't have to modify the jww code.
But it will not fix this issue.
My best guess is some Windows antivirus scanner issue.
Maybe add this for now?
if runtime.GOOS == "windows" {
		t.Skip("No Windows")
}

@bep, good point about closing the file from the test. I overlooked the fact that LogHandle is exported.
Disabling my antivirus did not affect the outcome. In addition, my test in jww properly removes the test log file after calling CloseLogFile() so I don't think that's the issue.
Using runtime to separate out behavior for windows will work for now, but then either this functionality doesn't get tested for windows or the files remain in the temp directory. While I would prefer not to leave files in temp, it wouldn't be the first process to do so; even chrome does so.
I will take another look at utils_test.go later today, when I have some additional time. I'm thinking that all of the function calls along with the various go routines that test creates still being in flight is contributing to this since creating simpler test situations work as expected.

FWIW if you've not noticed yet, Appveyor has upgraded the installed go
to the current v1.4.2 windows/amd64
I have to say I'm impressed, The ticket suggested next week :)
Owen
On Thu, 2015-02-19 at 13:32 -0800, Joel Scoble wrote:

I added a request for AppVeyor to upgrade to 1.4 on
http://help.appveyor.com/discussions/problems/1215-go-14-support. They
said they would update this week.
But that won't address the failures within these tests on Windows.
First I added a sleep to teardown() before removing the file, which
didn't work.
I then:
  * Created a tempDir using init() and placed all the testing
    logfiles in there.
  * Changed teardown() do an os.RemoveAll() on the temp directory.
  * Changed setup() to return the *os.File in addition to the
    filename.
  * Added a f.Close() to the returned file handle in the testing
    code.
  * Added jww.DiscardLogging() before calling `f.Close().
  * Created a main test function which ran the tests and after all
    tests were run called teardown().

These changes, which were made incrementally, also failed due to the
files still being in use.
The root of the problem is that jww never closes any files. It relies
on the program exiting to close the filies. It doesn't appear that
these tests will pass on Windows until there is a way of telling jww
to close its log files.
For jww, I think that calling DiscardLogging() might be an appropriate
place to close files, or any other function which allows for the
Handle to be changed.
Regardless, I don't think there is a way for these tests, as currently
written, to pass until there are changes on the jww side.
—
Reply to this email directly or view it on GitHub.

--
Dr. Owen Waller
Head of Development
Kulawe Limited
Forsyth House
Cormac Square
Belfast
Northern Ireland
BT2 8LA
+44 (0)757 882 9933
<EMAIL_ADDRESS>www.kulawe.com
Company Number: NI604010
VAT Number : GB 122 6319 42

Folks,
Ok so something s still up with my Appveyor build or my repo is somehow
completely corrupt at this point.
So if you look here:
https://ci.appveyor.com/project/owenwaller/hugo/build/1.0.36
That's my latest build and up that still fails with "cannot find package
"github.com/kr/pty""
This is after both a rebase of the upstream master onto my local master,
and a push -f master to my github repo. The rebase was completely
clean. So I am now upto date with the master branch - as my github repo
proves.
Currently my last commit is @BEP's
dc7b7ef865f5ba568702e867a29408e7f1a25be7
Now, if I repeat what appveyor is doing locally on my Linux box I
actually see different behaviour.
owen@thinkpad:/tmp/go-src$ rm -rf src/
owen@thinkpad:/tmp/go-src$ mkdir src
owen@thinkpad:/tmp/go-src$ cd src/
owen@thinkpad:/tmp/go-src/src$ go get -v -t -d
github.com/owenwaller/hugo
github.com/owenwaller/hugo (download)
github.com/spf13/hugo (download)
bitbucket.org/kardianos/osext (download)
bitbucket.org/pkg/inflect (download)
github.com/BurntSushi/toml (download)
github.com/PuerkitoBio/purell (download)
github.com/opennota/urlesc (download)
github.com/dchest/cssmin (download)
github.com/eknkc/amber (download)
github.com/gorilla/websocket (download)
github.com/mitchellh/mapstructure (download)
github.com/russross/blackfriday (download)
github.com/shurcooL/sanitized_anchor_name (download)
github.com/spf13/afero (download)
github.com/spf13/cast (download)
github.com/spf13/jwalterweatherman (download)
github.com/spf13/cobra (download)
github.com/spf13/pflag (download)
github.com/spf13/fsync (download)
github.com/spf13/viper (download)
github.com/kr/pretty (download)
github.com/kr/text (download)
github.com/xordataexchange/crypt (download)
github.com/armon/consul-api (download)
github.com/coreos/go-etcd (download)
github.com/coreos/etcd (download)
Fetching https://golang.org/x/crypto/openpgp?go-get=1
Parsing meta tags from https://golang.org/x/crypto/openpgp?go-get=1
(status code 200)
get "golang.org/x/crypto/openpgp": found meta tag
main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git",
RepoRoot:"https://go.googlesource.com/crypto"} at
https://golang.org/x/crypto/openpgp?go-get=1
get "golang.org/x/crypto/openpgp": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code
200)
golang.org/x/crypto (download)
Fetching https://gopkg.in/yaml.v2?go-get=1
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code
200)
get "gopkg.in/yaml.v2": found meta tag
main.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git",
RepoRoot:"https://gopkg.in/yaml.v2"} at
https://gopkg.in/yaml.v2?go-get=1
gopkg.in/yaml.v2 (download)
github.com/yosssi/ace (download)
github.com/spf13/nitro (download)
Fetching https://gopkg.in/fsnotify.v0?go-get=1
Parsing meta tags from https://gopkg.in/fsnotify.v0?go-get=1 (status
code 200)
get "gopkg.in/fsnotify.v0": found meta tag
main.metaImport{Prefix:"gopkg.in/fsnotify.v0", VCS:"git",
RepoRoot:"https://gopkg.in/fsnotify.v0"} at
https://gopkg.in/fsnotify.v0?go-get=1
gopkg.in/fsnotify.v0 (download)
owen@thinkpad:/tmp/go-src/src$ cd github.com/owenwaller/hugo/
owen@thinkpad:/tmp/go-src/src/github.com/owenwaller/hugo$ go test -cover
-v ./...
# github.com/owenwaller/hugo/helpers
helpers/content_test.go:4:2: cannot find package
"github.com/stretchr/testify/assert" in any of:
	/home/owen/golang/go/src/github.com/stretchr/testify/assert (from
$GOROOT)
	/tmp/go-src/src/github.com/stretchr/testify/assert (from $GOPATH)
FAIL	github.com/owenwaller/hugo/helpers [setup failed]
# github.com/owenwaller/hugo/hugolib
helpers/content_test.go:4:2: cannot find package
"github.com/stretchr/testify/assert" in any of:
	/home/owen/golang/go/src/github.com/stretchr/testify/assert (from
$GOROOT)
	/tmp/go-src/src/github.com/stretchr/testify/assert (from $GOPATH)
FAIL	github.com/owenwaller/hugo/hugolib [setup failed]
# github.com/owenwaller/hugo/tpl
helpers/content_test.go:4:2: cannot find package
"github.com/stretchr/testify/assert" in any of:
	/home/owen/golang/go/src/github.com/stretchr/testify/assert (from
$GOROOT)
	/tmp/go-src/src/github.com/stretchr/testify/assert (from $GOPATH)
FAIL	github.com/owenwaller/hugo/tpl [setup failed]
?   	github.com/owenwaller/hugo	[no test files]
?   	github.com/owenwaller/hugo/bufferpool	[no test files]
=== RUN TestFixUrl
--- PASS: TestFixUrl (0.00s)
PASS
coverage: 11.7% of statements
ok  	github.com/owenwaller/hugo/commands	0.009s	coverage: 11.7% of
statements
?   	github.com/owenwaller/hugo/create	[no test files]
?   	github.com/owenwaller/hugo/hugofs	[no test files]
?   	github.com/owenwaller/hugo/livereload	[no test files]
=== RUN TestDegenerateCreatePageFrom
--- PASS: TestDegenerateCreatePageFrom (0.00s)
=== RUN TestStandaloneCreatePageFrom
--- PASS: TestStandaloneCreatePageFrom (0.00s)
=== RUN TestPageShouldRender
--- PASS: TestPageShouldRender (0.00s)
=== RUN TestPageHasFrontMatter
--- PASS: TestPageHasFrontMatter (0.00s)
=== RUN TestExtractFrontMatter
--- PASS: TestExtractFrontMatter (0.00s)
=== RUN TestExtractFrontMatterDelim
--- PASS: TestExtractFrontMatterDelim (0.00s)
PASS
coverage: 41.1% of statements
ok  	github.com/owenwaller/hugo/parser	0.007s	coverage: 41.1% of
statements
=== RUN TestIgnoreDotFilesAndDirectories
--- PASS: TestIgnoreDotFilesAndDirectories (0.00s)
=== RUN TestEmptySourceFilesystem
--- PASS: TestEmptySourceFilesystem (0.00s)
=== RUN TestAddFile
--- PASS: TestAddFile (0.00s)
PASS
coverage: 39.4% of statements
ok  	github.com/owenwaller/hugo/source	0.040s	coverage: 39.4% of
statements
=== RUN TestHTMLRedirectAlias
--- PASS: TestHTMLRedirectAlias (0.00s)
=== RUN TestPageTranslator
--- PASS: TestPageTranslator (0.00s)
=== RUN TestPageTranslatorBase
--- PASS: TestPageTranslatorBase (0.00s)
=== RUN TestTranslateUglyUrls
--- PASS: TestTranslateUglyUrls (0.00s)
=== RUN TestTranslateDefaultExtension
--- PASS: TestTranslateDefaultExtension (0.00s)
PASS
coverage: 49.2% of statements
ok  	github.com/owenwaller/hugo/target	0.018s	coverage: 49.2% of
statements
=== RUN TestChainZeroTransformers
--- PASS: TestChainZeroTransformers (0.00s)
=== RUN TestAbsUrl
--- PASS: TestAbsUrl (0.00s)
=== RUN TestXMLAbsUrl
--- PASS: TestXMLAbsUrl (0.00s)
PASS
coverage: 87.6% of statements
ok  	github.com/owenwaller/hugo/transform	0.019s	coverage: 87.6% of
statements
=== RUN TestCutUsageMessage
--- PASS: TestCutUsageMessage (0.00s)
PASS
coverage: 11.1% of statements
ok  	github.com/owenwaller/hugo/utils	0.034s	coverage: 11.1% of
statements
?   	github.com/owenwaller/hugo/watcher	[no test files]
owen@thinkpad:/tmp/go-src/src/github.com/owenwaller/hugo$ git branch
* master

Note: Line one of this nukes my my entire src tree under $GOPATH/src
Also as you can see this is all on the master branch in my repo - which
matches the current HEAD in the upstream repo.
Either way I still don't have a build. But this time I don't have a
build on linux or Windows.
So has anyone any idea just what is going on here and how to fix this?
Right now I am stumped.
I am tempted to nuke my github repo and fork again, but I really don't
think this will fix anything - and it'll killl of the pull request we
are all discussing along with all of these comments....
If someone wants to take 10 minutes and clone my hugo repo to see what
they get that would be very much appreciated at this point.
Thanks
Owen
On Thu, 2015-02-19 at 20:38 -0800, Joel Scoble wrote:

@owenwaller I think you have some extraneous packages in your repo.
Notice the "cannot find package..." errors. I don't think those are in
hugo. Also, Hugo builds fine for me on Windows.
—
Reply to this email directly or view it on GitHub.

--
Dr. Owen Waller
Head of Development
Kulawe Limited
Forsyth House
Cormac Square
Belfast
Northern Ireland
BT2 8LA
+44 (0)757 882 9933
<EMAIL_ADDRESS>www.kulawe.com
Company Number: NI604010
VAT Number : GB 122 6319 42

@owenwaller the tests now pass on both Windows and Linux. The updated tests can be viewed at https://github.com/mohae/hugo/tree/utils_test. You can grab the updated code from my branch, or I can use that branch to submit a pull request. Please let me know if you want me to do so.
The problem was related to the setup/teardown functions and the way the testing library works. Eliminating those functions and moving the file creation to the test functions that were using them, and closing those files there resolved the issue.
To keep things cleaner, I created a tempDir on init() that both of the functions that create log files used. I also added a TestMain() function to run the tests from. After all of the tests are run, the tempDir is removed. Errors in TestMain and init() result in a panic.
I'm not sure that error checking is needed on the removal of the temporary directory and its files, but since that was being checked in the original teardown function, and that error was the source of these test changes, I left it in.
I also changed the name of the checkForExpectedStingOrFail() function to checkForExpectedStringOrFail() since I assumed that the Sting was erroneous.

I forgot to mention that these changes did not require any changes to jww due to @bep's suggestion.

B

@mohae thanks, well done. Give me a day - I'm away from my laptop for
most of the day - to look a the changes and get back to you.
But you seem to be suggesting that this is some sort of interaction
between my setup/teardown functions and the go testing library.
Correct?? Can you be more specific as to why this would be an issue?
Owen
On Fri, 2015-02-20 at 15:29 -0800, Joel Scoble wrote:

@owenwaller the tests now pass on both Windows and Linux. The updated
tests can be viewed at https://github.com/mohae/hugo/tree/utils_test.
You can grab the updated code from my branch, or I can use that branch
to submit a pull request. Please let me know if you want me to do so.
The problem was related to the setup/teardown functions and the way
the testing library works. Eliminating those functions and moving the
file creation to the test functions that were using them, and closing
those files there resolved the issue.
To keep things cleaner, I created a tempDir on init() that both of the
functions that create log files used. I also added a TestMain()
function to run the tests from. After all of the tests are run, the
tempDir is removed. Errors in TestMain and init() result in a panic.
I'm not sure that error checking is needed on the removal of the
temporary directory and its files, but since that was being checked in
the original teardown function, and that error was the source of these
test changes, I left it in.
I also changed the name of the checkForExpectedStingOrFail() function
to checkForExpectedStringOrFail() since I assumed that the Sting was
erroneous.
—
Reply to this email directly or view it on GitHub.

--
Dr. Owen Waller
Head of Development
Kulawe Limited
Forsyth House
Cormac Square
Belfast
Northern Ireland
BT2 8LA
+44 (0)757 882 9933
<EMAIL_ADDRESS>www.kulawe.com
Company Number: NI604010
VAT Number : GB 122 6319 42

@owenwaller:
One should not write tests the same way one would write the application code. Imo, writing a lot of helper functions to do work for you in tests is probably a sign that the tests are being written wrong. KISS
iirc, tests spawn go routines. The problems this code was exhibiting seemed to indicate that some go routines were still in flight while the teardown function was being executed, which is why some files could not be deleted. Moving around where the close occurred, e.g. defer vs not deferring, also affected how many files had this error. Also, runs could exhibit very minor variations.
I had thought that sleeping some before the close and removal of the test file might allow whatever process that had a handle on the file to finish and the close/removal to succeed. That was not the case.
Since I had already spent too much time trying to work out the issue, changing the way the tests were structured seemed more productive than doing further investigations to determine why those routines were still in flight and how to avoid that situation with the existing test code base.
For tests, in general, keep the code the test is running as local to that test as possible, keep things as simple as possible to achieve your testing goals, and minimize the use of helper functions within the tests.
Helper functions do have their place, and I do use them, mainly to help evaluation of results, but they should be used when necessary and not necessarily when convenient. You will notice that the functions that did remain in the updated tests were the ones that evaluated the results.
The same is true with testing libraries. I went from using GoConvey, because of all of its "conveniences" baked in, to really disliking it. Then started using stretchr/testify, but now I mainly just use the standard testing package.
The above is my opinion. It may or may not be useful. As you do more testing in Go, you will probably form your own opinions.
Also, I noticed in you tended towards very verbose function names, Go tends to lean towards less verbose naming.
I hope some of the above helps.


Issuing Commands

Is your feature request related to a problem? Please describe.
None.
Describe the solution you'd like
The ability to speak and issue commands
Describe alternatives you've considered
Typing commands.
Additional context
None.

We just need to listen to the mic and interpret what is being said. We could use speechrecognition python package to do this.


Build fails on initial repo clone

The initial yarn build on a freshly cloned repo fails since the .generated files in the basic example are missing. To reproduce, clone the repo in a new directory and run yarn build.
You will get
$ tsc
lerna ERR! yarn run build exited 1 in '@examples/basic'
lerna ERR! yarn run build stdout:
$ tsc
index.ts(5,8): error TS2307: Cannot find module './.generated/ContainerInstance'.
index.ts(8,8): error TS2307: Cannot find module './.generated/ContainerRegistry'.

To resolve this, one can change the build script for the example to armkit generate && tsc, or add a new script, e.g. bootstrap that will be used only once, is the same as build for the other packages, and will run armkit generate && tsc for the example. First approach is simpler, but I am not sure what the overhead of running generate each time would be.

I agree.  I willl have a look.

resolves in #81 that  updates yarn task deps

@CodeRunRepeat @tomconte @pgollucci  Does th current changes in main resolves this issue ?

Feel free to repopen if there should be still issues arise