Improve API

Few options should be pass to the reconstruction:

number of iterations
scale to GeV
print

The tool should return the top quark, but also the status of the reconstruction and the number of iteration with actual soluions.

This is now implemented, expect the scale to GeV ... this can be done before the algo starts!


Add MIN_PNG_SIZE and MIN_JPEG_SIZE to allow for images below 128 bytes

See https://github.com/praw-dev/praw/issues/561

Merged in as #562. Thanks!


Where is Alabarza?

I re-installed the game a few days ago and skipped some mods; for instance I believe
the old stronghold NPC mod is no longer necessary, as another mod handles that
now (I think; at the least I can take over all strongholds and rule them with an iron
fist \o/). I also omitted a few MODs and this time I am running through quite
nicely. I also skipped installing that specific lich that occupies the area where the
leprechaun etc... have their new hideout, and this allowed me to do the quest
fine. I just defeated Poki or Peski (whatever the name was of the demon).
One oddity I have is that the main actor spasms about suddenly in the docks
area, as if he were to talk about something or someone but never does. I don't
think this is related to your mod at all, but I mention it - I think things bugged
out when a quest with Paris (of another mod) didn't continue. Some interjections
of other mods can cause issues, oddly enough. Anyway.
Long comment here, but I am now wondering where Alabarza is. It reads
"we will find him at the troll mound", in the journal. However had, when I
go there, after whacking down the Poki demon, Alabarza is not there. Perhaps
some condition is not met. But, at any rate, perhaps the journal entry could
clarify this a bit more? When I first read it, I thought an area west of the
druid grove was meant on the world map. When I re-read it, the troll mound
was meant (but ... outside, right?). This probably does not fit into the FAQ
entry, but it's just feedback - as allows please feel free to proceed as you
deem fit. I like that the mod focuses primarily on one NPC, rather than the
strange mega-mods that modify so many things (unless you pay close
attention to what they install; some mods create really huge issues to
the game overall).
(It's also quite difficult to find high quality information on google search
these days, so in-game documentation may be quite important as a
consequence. I found tons of information on the beamdog forum but
almost nothing when I tried different search strategies on google search,
which I am 100% sure worked fine in the past. Google really worsened
their search engine ... probably riding the AI wave ... but I digress)

Aha - found the ferret IN the mound. Perhaps the description can be
adapted a bit to give a hint that we should also look into the mound.
From the description right now I thought it would be west of the
druid region on the world map OR west of the troll mound entrance,
both which didn't lead to anything. But now as I entered the troll
mound I saw that ferret. Anyway this is a small issue, I will
close this - don't want to spam you with issues on github. :)


how can i know the meaning of shadow qpos parameters?

thanks for your work.
when i was looking the handmodel.py script, there is a function get_meshes_from_q(self, q=None, i=0).
the expected input of q is 33 dim, however, other project use 30-dim qpos to represent the pose of shadow hand.
how can i know what those 33-dim params mean? Many thanks.

The shadow's qpos consists of a 9-dimensional q_global and a 24-dimensional q_local. Similar to other projects, the 24-dimensional q_local represents the angles of each joint. For q_global, the first 3 dimensions correspond to the global pose's xyz-position, while the last 6 dimensions represent the global pose's 6D rotation.

I have found the function get_rot6d_from_rpy(). thanks, I will close this issue.

Exactly, considering that the forearm link is the base link of the shadowhand URDF, q_global refers to the transformation of this URDF in the world coordinate frame.

OK thanks so much.


ListMailings -yields error on MailingId being int with error JSON integer<PHONE_NUMBER>8 is too large or small for an Int32. Path '[0].mailing_id', line 1, position 117.

Hi Kyle.
I got error JSON integer<PHONE_NUMBER>8 is too large or small for an Int32. Path '[0].mailing_id', line 1, position 117. on mailing_id when try to call emma api listMailings. Should mailing_id be a long integer instead. Not sure if it was emma bug or just from emmasharp. Can you help?
Thanks
MilSix

Hi Kyle,
Thank you for a quick update all IDs to long.
Sorry to took so long to test it out. Anyhow, I'm still get the same error after download get the latest version. Since you update all the IDs to long but still for those defaults (to list range of ID still in int such as start , end parameters in ListMailings method.
    public List<MailingInfo> ListMailings(bool archived = false, List<MailingType> mailingType = null, List<MailingStatus> mailingStatus = null, bool isScheduled = false, bool withHtmlBody = false, bool withPlaintext = false, _**int** start = -1, **int** end = -1_)
    {
        var request = new RestRequest();
        request.Resource = "/{accountId}/mailings";

		if (archived)
            request.AddParameter("include_archived", archived);

        if (mailingType != null)
            request.AddParameter("mailing_types", string.Join(",", Array.ConvertAll(mailingType.ToArray(), i => i.ToEnumString<MailingType>())));

        if (mailingStatus != null)
            request.AddParameter("mailing_statuses", string.Join(",", Array.ConvertAll(mailingStatus.ToArray(), i => i.ToEnumString<MailingStatus>())));

        if (isScheduled)
            request.AddParameter("is_scheduled", isScheduled);

        if (withHtmlBody)
            request.AddParameter("with_html_body", withHtmlBody);

        if (withPlaintext)
            request.AddParameter("with_plaintext", withPlaintext);

       ###  **return Execute<List<MailingInfo>>(request, start, end);**
    }

@MilSix Right, but int start and int end are the "pages" in the pagination. I wouldn't expect you to have that many mailing records, correct? Must be something else. Also my latest mailing ID is way under the 2.1 billion for int32, so I don't think that was it, though probably not wrong to plan ahead there.

agree. Our main account has 37 sub-accounts, where each of 37 (colleges-entities) managing their own campaigns.  it's only about 6 or 7 of sub-accounts are currently active.
I don't think we have that many campaigns. Don't know how emma store ID, may be they stored all customers email in same table. That would make ID possible with long values. I was hoping to call ListMailings for inventory purposes.

@MilSix might be good if you can download and reference the project manually so you can set some breakpoints and investigate. Did the exception thrown change since you updated?

Exception thrown only when I reach 37th loop.  Might be I just going to force it to just use for one sub account and see if I get the correct list.  Here is current code. Not sure if I use it correctly, that might be the case too.
    public int ListMailingsCountBySubAccount(List<USR_USP_EMMA_SUBACCOUNT_SP_Result> subAccounts, List<EmmaSharp.Models.Mailings.MailingInfo> outList)
    {
        int mailCount = 0;
        foreach (var subAccount in subAccounts)
        {
            EmmaApi emmasharpRead = new EmmaApi(publicKey, privateKey, subAccount.AccountId.ToString());
            mailCount += emmasharpRead.ListMailingsCount(true, null, null, false, false, false);
            List<EmmaSharp.Models.Mailings.MailingInfo> MasterMailingInfosList = emmasharpRead.ListMailings(true, null, null, false, false, false, -1, -1);
        }
        return (mailCount);
    }

@MilSix Ah, I see. Doing that many calls (each emmasharpRead being a call, 2 times a loop) could be getting you rate limited. I ran this code on my end and that's what happened. Emma has a 180 calls/minute limit. Seeing as it's getting to about the same point each time, I'm thinking that might be why.
Might think about splitting the work into separate calls, holding on the count then doing your Listing later.  Also if you use each accounts API key rather than your personal, you don't get flagged for as many call hits. Since each account has unique keys, you only get flagged for the calls within the loop. For larger work you may need to consider going this route.
Let me know if that gets you further along.

Thank you Kyle, that makes sense. Il try to get all sub accounts keys  Thought we only have one set keys. for all sub account.


AttributeError: 'NoneType' object has no attribute 'notify'

This happens in a VM (Ubuntu 20.04.3 LTS) in a platform provided by Google Cloud. This is a very fresh implementation after new git clone on von-network and aries-cloudagent-python. Then, running ./run_demo faber, this happens:
eduardo@eduardoito:~/aries/aries-cloudagent-python/demo$ ./run_demo faber
Preparing agent image...
sha256:739adae5f285d5e9ed031fd83efe7d97df17d08588bb2ec99bf572d069903ea3
Trying to detect ngrok service endpoint
jq not found
x.x.x.x
Starting [faber] agent with args [--port 8020]
Initializing demo agent faber with AIP 20 and credential type indy
Setting use_did_exchange: True
#1 Provision an agent and wallet, get back configuration details
Started webhook listener on port: 8022
Faber      | Registering faber.agent ...
Faber      | nym_info: {'did': 'ELoxNnY1VbJLcTcYWs3cmL', 'seed': 'd_000000000000000000000000392955', 'verkey': '8GkeTXkVFYfxZKDcX9RzFakQSksVeS8swBZd2rUcaMKG'}
Faber      | Registered DID: ELoxNnY1VbJLcTcYWs3cmL
Created public DID
Faber      | ['/home/indy/.pyenv/versions/3.6.13/bin/python', '-m', 'aries_cloudagent', 'start', '--endpoint', 'http://x.x.x.x:8020', '--label', 'faber.agent', '--auto-ping-connection', '--auto-respond-messages', '--inbound-transport', 'http', '<IP_ADDRESS>', '8020', '--outbound-transport', 'http', '--admin', '<IP_ADDRESS>', '8021', '--admin-insecure-mode', '--wallet-type', 'indy', '--wallet-name', 'faber.agent392955', '--wallet-key', 'faber.agent392955', '--preserve-exchange-records', '--auto-provision', '--emit-new-didcomm-prefix', '--genesis-transactions', '{"reqSignature":{},"txn":{"data":{"data":{"alias":"Node1","blskey":"4N8aUNHSgjQVgkpm8nhNEfDf6txHznoYREg9kirmJrkivgL4oSEimFF6nsQ6M41QvhM2Z33nves5vfSn9n1UwNFJBYtWVnHYMATn76vLuL3zU88KyeAYcHfsih3He6UHcXDxcaecHVz6jhCYz1P2UZn2bDVruL5wXpehgBfBaLKm3Ba","blskey_pop":"RahHYiCvoNCtPTrVtP7nMC5eTYrsUA8WjXbdhNc8debh1agE9bGiJxWBXYNFbnJXoXhWFMvyqhqhRoq737YQemH5ik9oL7R4NTTCz2LEZhkgLJzB3QRQqJyBNyv7acbdHrAT8nQ9UkLbaVL9NBpnWXBTw4LEMePaSHEw66RzPNdAX1","client_ip":"x.x.x.x","client_port":9702,"node_ip":"<IP_ADDRESS>","node_port":9701,"services":["VALIDATOR"]},"dest":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv"},"metadata":{"from":"Th7MpTaRZVRYnPiabds81Y"},"type":"0"},"txnMetadata":{"seqNo":1,"txnId":"fea82e10e894419fe2bea7d96296a6d46f50f93f9eeda954ec461b2ed2950b62"},"ver":"1"}\n{"reqSignature":{},"txn":{"data":{"data":{"alias":"Node2","blskey":"37rAPpXVoxzKhz7d9gkUe52XuXryuLXoM6P6LbWDB7LSbG62Lsb33sfG7zqS8TK1MXwuCHj1FKNzVpsnafmqLG1vXN88rt38mNFs9TENzm4QHdBzsvCuoBnPH7rpYYDo9DZNJePaDvRvqJKByCabubJz3XXKbEeshzpz4Ma5QYpJqjk","blskey_pop":"Qr658mWZ2YC8JXGXwMDQTzuZCWF7NK9EwxphGmcBvCh6ybUuLxbG65nsX4JvD4SPNtkJ2w9ug1yLTj6fgmuDg41TgECXjLCij3RMsV8CwewBVgVN67wsA45DFWvqvLtu4rjNnE9JbdFTc1Z4WCPA3Xan44K1HoHAq9EVeaRYs8zoF5","client_ip":"<IP_ADDRESS>","client_port":9704,"node_ip":"<IP_ADDRESS>","node_port":9703,"services":["VALIDATOR"]},"dest":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb"},"metadata":{"from":"EbP4aYNeTHL6q385GuVpRV"},"type":"0"},"txnMetadata":{"seqNo":2,"txnId":"1ac8aece2a18ced660fef8694b61aac3af08ba875ce3026a160acbc3a3af35fc"},"ver":"1"}\n{"reqSignature":{},"txn":{"data":{"data":{"alias":"Node3","blskey":"3WFpdbg7C5cnLYZwFZevJqhubkFALBfCBBok15GdrKMUhUjGsk3jV6QKj6MZgEubF7oqCafxNdkm7eswgA4sdKTRc82tLGzZBd6vNqU8dupzup6uYUf32KTHTPQbuUM8Yk4QFXjEf2Usu2TJcNkdgpyeUSX42u5LqdDDpNSWUK5deC5","blskey_pop":"QwDeb2CkNSx6r8QC8vGQK3GRv7Yndn84TGNijX8YXHPiagXajyfTjoR87rXUu4G4QLk2cF8NNyqWiYMus1623dELWwx57rLCFqGh7N4ZRbGDRP4fnVcaKg1BcUxQ866Ven4gw8y4N56S5HzxXNBZtLYmhGHvDtk6PFkFwCvxYrNYjh","client_ip":"<IP_ADDRESS>","client_port":9706,"node_ip":"x.x.x.x","node_port":9705,"services":["VALIDATOR"]},"dest":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya"},"metadata":{"from":"4cU41vWW82ArfxJxHkzXPG"},"type":"0"},"txnMetadata":{"seqNo":3,"txnId":"7e9f355dffa78ed24668f0e0e369fd8c224076571c51e2ea8be5f26479edebe4"},"ver":"1"}\n{"reqSignature":{},"txn":{"data":{"data":{"alias":"Node4","blskey":"2zN3bHM1m4rLz54MJHYSwvqzPchYp8jkHswveCLAEJVcX6Mm1wHQD1SkPYMzUDTZvWvhuE6VNAkK3KxVeEmsanSmvjVkReDeBEMxeDaayjcZjFGPydyey1qxBHmTvAnBKoPydvuTAqx5f7YNNRAdeLmUi99gERUU7TD8KfAa6MpQ9bw","blskey_pop":"RPLagxaR5xdimFzwmzYnz4ZhWtYQEj8iR5ZU53T2gitPCyCHQneUn2Huc4oeLd2B2HzkGnjAff4hWTJT6C7qHYB1Mv2wU5iHHGFWkhnTX9WsEAbunJCV2qcaXScKj4tTfvdDKfLiVuU2av6hbsMztirRze7LvYBkRHV3tGwyCptsrP","client_ip":"<IP_ADDRESS>","client_port":9708,"node_ip":"<IP_ADDRESS>","node_port":9707,"services":["VALIDATOR"]},"dest":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA"},"metadata":{"from":"TWwCRQRZ2ZHMJFn9TzLp7W"},"type":"0"},"txnMetadata":{"seqNo":4,"txnId":"aa5e817d7cc626170eca175822029339a444eb0ee8f0bd20d3b0b76e566fb008"},"ver":"1"}\n', '--seed', 'd_000000000000000000000000392955', '--webhook-url', 'http://x.x.x.x:8022/webhooks', '--trace-target', 'log', '--trace-tag', 'acapy.events', '--trace-label', 'faber.agent.trace', '--auto-accept-invites', '--auto-accept-requests', '--auto-store-credential']
Shutting down agent ...
Faber      |
Faber      | Shutting down
Faber      | 2021-11-26 15:45:10,366 asyncio ERROR Task exception was never retrieved
Faber      | future: <Task finished coro=<run_loop..done() done, defined at /home/indy/aries_cloudagent/commands/start.py:77> exception=AttributeError("'NoneType' object has no attribute 'notify'",)>
Faber      | Traceback (most recent call last):
Faber      |   File "/home/indy/aries_cloudagent/commands/start.py", line 79, in done
Faber      |     await shutdown
Faber      |   File "/home/indy/aries_cloudagent/commands/start.py", line 35, in shutdown_app
Faber      |     await conductor.stop()
Faber      |   File "/home/indy/aries_cloudagent/core/conductor.py", line 373, in stop
Faber      | Process did not terminate in time
Faber      |     await self.root_profile.notify(SHUTDOWN_EVENT_TOPIC, {})
ERROR:runners.agent_container:Error terminating agent:
Traceback (most recent call last):
File "/home/indy/demo/runners/faber.py", line 417, in main
else False,
File "/home/indy/demo/runners/agent_container.py", line 678, in initialize
await self.agent.start_process()
File "/home/indy/demo/runners/support/agent.py", line 657, in start_process
await self.detect_process()
File "/home/indy/demo/runners/support/agent.py", line 996, in detect_process
+ f"Admin URL: {status_url}"
Exception: Timed out waiting for agent process to start (status=None). Admin URL: http://<IP_ADDRESS>:8021/status
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/indy/demo/runners/agent_container.py", line 881, in terminate
await self.agent.terminate()
File "/home/indy/demo/runners/support/agent.py", line 681, in terminate
result = await asyncio.wait_for(future, 10, loop=loop)
File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
return fut.result()
File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/indy/demo/runners/support/agent.py", line 668, in _terminate
raise Exception(msg)
Exception: Process did not terminate in time
Faber      | AttributeError: 'NoneType' object has no attribute 'notify'
Traceback (most recent call last):
File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/indy/demo/runners/faber.py", line 765, in
asyncio.get_event_loop().run_until_complete(main(args))
File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete
return future.result()
File "/home/indy/demo/runners/faber.py", line 417, in main
else False,
File "/home/indy/demo/runners/agent_container.py", line 678, in initialize
await self.agent.start_process()
File "/home/indy/demo/runners/support/agent.py", line 657, in start_process
await self.detect_process()
File "/home/indy/demo/runners/support/agent.py", line 996, in detect_process
+ f"Admin URL: {status_url}"
Exception: Timed out waiting for agent process to start (status=None). Admin URL: http://xx.xx.x.x:8021/status
Faber      |
Faber      | ::::::::::::::::::::::::::::::::::::::::::::::
Faber      | :: faber.agent                              ::
Faber      | ::                                          ::
Faber      | ::                                          ::
Faber      | :: Inbound Transports:                      ::
Faber      | ::                                          ::
Faber      | ::   - http://<IP_ADDRESS>:8020                  ::
Faber      | ::                                          ::
Faber      | :: Outbound Transports:                     ::
Faber      | ::                                          ::
Faber      | ::   - http                                 ::
Faber      | ::   - https                                ::
Faber      | ::                                          ::
Faber      | :: Public DID Information:                  ::
Faber      | ::                                          ::
Faber      | ::   - DID: ELoxNnY1VbJLcTcYWs3cmL          ::
Faber      | ::                                          ::
Faber      | :: Administration API:                      ::
Faber      | ::                                          ::
Faber      | ::   - http://<IP_ADDRESS>:8021                  ::
Faber      | ::                                          ::
Faber      | ::                               ver: 0.7.2 ::
Faber      | ::::::::::::::::::::::::::::::::::::::::::::::
Faber      |
Faber      | Listening...
Faber      |

Are you using "x.x.x.x" as an IP or are you redacting your log file?

yes. It was due to  security reason, just in case.

This issue was found due to a fault in SSL/TLS as reported wrongly at https://github.com/bcgov/von-network/issues/187, indeed it should be reported herein, as it is related to aries-cloudagent-python. Briefly describing what happened is that I have an VM created in university cloud environment, where I installed aries-cloudagent, when executing run_demo faber results "OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to github.com:443". Wade Barnes made a correct assumption, I think, "proxy setting should be set to git", as when I do wireshark on interface docker0, it displays clearly that TLS header comes empty. So, I had to request an independent VM on Google cloud, that Wade´s assumption is true. So, I have wireshark trace displaying clearly that TLS header is not empty, what shows that assumption is true. Nonetheless, appears this NoneType error.

The root cause is that Faber isn't detecting that the aca-py agent has started up.  Could be a timing issue, or a network-related issue (although the aca-py agent is started as a sub-process, so not sure why there would be a network issue).
When faber starts their agent they poll the http://<IP_ADDRESS>:8021/status endpoint to confirm that aca-py has finished its startup.
You can try extending the timeout for this step, or logging what is getting returned from this call.

as it may take time to analyze, I would like to share .pcap file. As it is 79MB, I would like to request you an repository to upload.

Hi,
I´ve doubled the START_TIMEOUT as below:
START_TIMEOUT = float(os.getenv("START_TIMEOUT", 60.0)) # 30.0
at ~/aries-cloudagent-python/demo/runners/support/agent.py
and the things improved a lot, now it is possible to connect faber2alice,
send credential, and make proof. It delays a little more to bring the invitation data, so
the start_time might be in between of 30..60s.
Nevertheless, when exiting faber and alice, it comes new timeout problem. But, I think it is now manageable.
.......
22/webhooks', '--trace-target', 'log', '--trace-tag', 'acapy.events', '--trace-label', 'faber.agent.trace', '--auto-accept-invites', '--auto-accept-requests', '--auto-store-credential']' timed out after 0.5 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/indy/demo/runners/agent_container.py", line 881, in terminate
await self.agent.terminate()
File "/home/indy/demo/runners/support/agent.py", line 681, in terminate
result = await asyncio.wait_for(future, 10, loop=loop)  # 10
File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
return fut.result()
File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/indy/demo/runners/support/agent.py", line 668, in _terminate
raise Exception(msg)
Exception: Process did not terminate in time
eduardo@eduardoito:~/aries-cloudagent-python/demo$

after some more simulations, START_TIMEOUT needs to be set to 60s.
at coroutine _terminate, timeout should be set to 2.0 instead of 0.5s.

Regarding to TLS header empty in University Environment, that it could be proxy/firewall setting. Indeed, it happens because when during dockerfile.demo at docker run --rm bcgovimages/von-image36-1.15-1 /bin/bash, it may happen that MTU size may not fit VxLAN policy of local IT environment. In my case, it was limited to 1450 bytes. The procedure made was to change /etc/docker/daemon.json by introducing { "mtu": 1450}


script/server works but script/install doesn't (org.itunes-api.plist: Path had bad ownership/permissions)

Hi all,
I am able to run itunes-api using script/server, but then I haven't found a way to put it into the background so that I can exit my terminal. I tried with CTRL-Z, bg, exit, and also with nohup, but neither worked. And I'd really rather have it run as a service anyway. But script/install isn't working for me. I have forever installed, and itunes API works, but it's giving me a permissions error as shown below..  Any ideas what's causing this? Forever is installed, I'm not sure if I need to upgrade to a certain version or something?
Here are a few of the many things I've tried.
$ script/install
Installing itunes-api...
/Volumes/Users/scstraus/Library/LaunchAgents/org.itunes-api.plist: Path had bad ownership/permissions
$ ls -la /Volumes/Users/scstraus/Library/LaunchAgents/org.itunes-api.plist
-rwxr-x---  1 scstraus  staff  1022 Nov 15 22:19 /Volumes/Users/scstraus/Library/LaunchAgents/org.itunes-api.plist
$ npm install
audited 169 packages in 8.576s
found 0 vulnerabilities
$ script/install
Installing itunes-api...
/Volumes/Users/scstraus/Library/LaunchAgents/org.itunes-api.plist: Path had bad ownership/permissions
$ chmod 777 /Volumes/Users/scstraus/Library/LaunchAgents/org.itunes-api.plist
$ ls -la /Volumes/Users/scstraus/Library/LaunchAgents/org.itunes-api.plist
-rwxrwxrwx  1 scstraus  staff  1022 Nov 15 22:26 /Volumes/Users/scstraus/Library/LaunchAgents/org.itunes-api.plist
$ script/install
Installing itunes-api...
/Volumes/Users/scstraus/Library/LaunchAgents/org.itunes-api.plist: Path had bad ownership/permissions

did you ever get this figured out

Nope. Waiting for a new Mac, will try again. One other guy got it working so may be version of OS X I was on.

So I'm getting the same error on MacOS 10.14.1. Anyone know what I'm doing wrong? Full install log below:
Xanadu:itunes-api scstraus$ script/bootstrap
!!!!
You don't have forever installed. You need to install it first.
Just install it with this command:
sudo npm install forever -g
Xanadu:itunes-api scstraus$ sudo npm install forever -g
Password:
sudo: npm: command not found
Xanadu:itunes-api scstraus$ brew update
-bash: brew: command not found
Xanadu:itunes-api scstraus$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
==> This script will install:
/usr/local/bin/brew
/usr/local/share/doc/homebrew
/usr/local/share/man/man1/brew.1
/usr/local/share/zsh/site-functions/_brew
/usr/local/etc/bash_completion.d/brew
/usr/local/Homebrew
==> The following new directories will be created:
/usr/local/bin
/usr/local/etc
/usr/local/include
/usr/local/lib
/usr/local/sbin
/usr/local/share
/usr/local/var
/usr/local/opt
/usr/local/share/zsh
/usr/local/share/zsh/site-functions
/usr/local/var/homebrew
/usr/local/var/homebrew/linked
/usr/local/Cellar
/usr/local/Caskroom
/usr/local/Homebrew
/usr/local/Frameworks
Press RETURN to continue or any other key to abort
Xanadu:itunes-api scstraus$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
==> This script will install:
/usr/local/bin/brew
/usr/local/share/doc/homebrew
/usr/local/share/man/man1/brew.1
/usr/local/share/zsh/site-functions/_brew
/usr/local/etc/bash_completion.d/brew
/usr/local/Homebrew
==> The following new directories will be created:
/usr/local/bin
/usr/local/etc
/usr/local/include
/usr/local/lib
/usr/local/sbin
/usr/local/share
/usr/local/var
/usr/local/opt
/usr/local/share/zsh
/usr/local/share/zsh/site-functions
/usr/local/var/homebrew
/usr/local/var/homebrew/linked
/usr/local/Cellar
/usr/local/Caskroom
/usr/local/Homebrew
/usr/local/Frameworks
Press RETURN to continue or any other key to abort
==> /usr/bin/sudo /bin/mkdir -p /usr/local/bin /usr/local/etc /usr/local/include /usr/local/lib /usr/local/sbin /usr/local/share /usr/local/var /usr/local/opt /usr/local/share/zsh /usr/local/share/zsh/site-functions /usr/local/var/homebrew /usr/local/var/homebrew/linked /usr/local/Cellar /usr/local/Caskroom /usr/local/Homebrew /usr/local/Frameworks
==> /usr/bin/sudo /bin/chmod g+rwx /usr/local/bin /usr/local/etc /usr/local/include /usr/local/lib /usr/local/sbin /usr/local/share /usr/local/var /usr/local/opt /usr/local/share/zsh /usr/local/share/zsh/site-functions /usr/local/var/homebrew /usr/local/var/homebrew/linked /usr/local/Cellar /usr/local/Caskroom /usr/local/Homebrew /usr/local/Frameworks
==> /usr/bin/sudo /bin/chmod 755 /usr/local/share/zsh /usr/local/share/zsh/site-functions
==> /usr/bin/sudo /usr/sbin/chown scstraus /usr/local/bin /usr/local/etc /usr/local/include /usr/local/lib /usr/local/sbin /usr/local/share /usr/local/var /usr/local/opt /usr/local/share/zsh /usr/local/share/zsh/site-functions /usr/local/var/homebrew /usr/local/var/homebrew/linked /usr/local/Cellar /usr/local/Caskroom /usr/local/Homebrew /usr/local/Frameworks
==> /usr/bin/sudo /usr/bin/chgrp admin /usr/local/bin /usr/local/etc /usr/local/include /usr/local/lib /usr/local/sbin /usr/local/share /usr/local/var /usr/local/opt /usr/local/share/zsh /usr/local/share/zsh/site-functions /usr/local/var/homebrew /usr/local/var/homebrew/linked /usr/local/Cellar /usr/local/Caskroom /usr/local/Homebrew /usr/local/Frameworks
==> Downloading and installing Homebrew...
remote: Enumerating objects: 126245, done.
remote: Total 126245 (delta 0), reused 0 (delta 0), pack-reused 126245
Receiving objects: 100% (126245/126245), 29.99 MiB | 3.14 MiB/s, done.
Resolving deltas: 100% (92542/92542), done.
From https://github.com/Homebrew/brew

[new branch]      master     -> origin/master
[new tag]         0.1        -> 0.1
[new tag]         0.2        -> 0.2
[new tag]         0.3        -> 0.3
[new tag]         0.4        -> 0.4
[new tag]         0.5        -> 0.5
[new tag]         0.6        -> 0.6
[new tag]         0.7        -> 0.7
[new tag]         0.7.1      -> 0.7.1
[new tag]         0.8        -> 0.8
[new tag]         0.8.1      -> 0.8.1
[new tag]         0.9        -> 0.9
[new tag]         0.9.1      -> 0.9.1
[new tag]         0.9.2      -> 0.9.2
[new tag]         0.9.3      -> 0.9.3
[new tag]         0.9.4      -> 0.9.4
[new tag]         0.9.5      -> 0.9.5
[new tag]         0.9.8      -> 0.9.8
[new tag]         0.9.9      -> 0.9.9
[new tag]         1.0.0      -> 1.0.0
[new tag]         1.0.1      -> 1.0.1
[new tag]         1.0.2      -> 1.0.2
[new tag]         1.0.3      -> 1.0.3
[new tag]         1.0.4      -> 1.0.4
[new tag]         1.0.5      -> 1.0.5
[new tag]         1.0.6      -> 1.0.6
[new tag]         1.0.7      -> 1.0.7
[new tag]         1.0.8      -> 1.0.8
[new tag]         1.0.9      -> 1.0.9
[new tag]         1.1.0      -> 1.1.0
[new tag]         1.1.1      -> 1.1.1
[new tag]         1.1.10     -> 1.1.10
[new tag]         1.1.11     -> 1.1.11
[new tag]         1.1.12     -> 1.1.12
[new tag]         1.1.13     -> 1.1.13
[new tag]         1.1.2      -> 1.1.2
[new tag]         1.1.3      -> 1.1.3
[new tag]         1.1.4      -> 1.1.4
[new tag]         1.1.5      -> 1.1.5
[new tag]         1.1.6      -> 1.1.6
[new tag]         1.1.7      -> 1.1.7
[new tag]         1.1.8      -> 1.1.8
[new tag]         1.1.9      -> 1.1.9
[new tag]         1.2.0      -> 1.2.0
[new tag]         1.2.1      -> 1.2.1
[new tag]         1.2.2      -> 1.2.2
[new tag]         1.2.3      -> 1.2.3
[new tag]         1.2.4      -> 1.2.4
[new tag]         1.2.5      -> 1.2.5
[new tag]         1.2.6      -> 1.2.6
[new tag]         1.3.0      -> 1.3.0
[new tag]         1.3.1      -> 1.3.1
[new tag]         1.3.2      -> 1.3.2
[new tag]         1.3.3      -> 1.3.3
[new tag]         1.3.4      -> 1.3.4
[new tag]         1.3.5      -> 1.3.5
[new tag]         1.3.6      -> 1.3.6
[new tag]         1.3.7      -> 1.3.7
[new tag]         1.3.8      -> 1.3.8
[new tag]         1.3.9      -> 1.3.9
[new tag]         1.4.0      -> 1.4.0
[new tag]         1.4.1      -> 1.4.1
[new tag]         1.4.2      -> 1.4.2
[new tag]         1.4.3      -> 1.4.3
[new tag]         1.5.0      -> 1.5.0
[new tag]         1.5.1      -> 1.5.1
[new tag]         1.5.10     -> 1.5.10
[new tag]         1.5.11     -> 1.5.11
[new tag]         1.5.12     -> 1.5.12
[new tag]         1.5.13     -> 1.5.13
[new tag]         1.5.14     -> 1.5.14
[new tag]         1.5.2      -> 1.5.2
[new tag]         1.5.3      -> 1.5.3
[new tag]         1.5.4      -> 1.5.4
[new tag]         1.5.5      -> 1.5.5
[new tag]         1.5.6      -> 1.5.6
[new tag]         1.5.7      -> 1.5.7
[new tag]         1.5.8      -> 1.5.8
[new tag]         1.5.9      -> 1.5.9
[new tag]         1.6.0      -> 1.6.0
[new tag]         1.6.1      -> 1.6.1
[new tag]         1.6.10     -> 1.6.10
[new tag]         1.6.11     -> 1.6.11
[new tag]         1.6.12     -> 1.6.12
[new tag]         1.6.13     -> 1.6.13
[new tag]         1.6.14     -> 1.6.14
[new tag]         1.6.15     -> 1.6.15
[new tag]         1.6.16     -> 1.6.16
[new tag]         1.6.17     -> 1.6.17
[new tag]         1.6.2      -> 1.6.2
[new tag]         1.6.3      -> 1.6.3
[new tag]         1.6.4      -> 1.6.4
[new tag]         1.6.5      -> 1.6.5
[new tag]         1.6.6      -> 1.6.6
[new tag]         1.6.7      -> 1.6.7
[new tag]         1.6.8      -> 1.6.8
[new tag]         1.6.9      -> 1.6.9
[new tag]         1.7.0      -> 1.7.0
[new tag]         1.7.1      -> 1.7.1
[new tag]         1.7.2      -> 1.7.2
[new tag]         1.7.3      -> 1.7.3
[new tag]         1.7.4      -> 1.7.4
[new tag]         1.7.5      -> 1.7.5
[new tag]         1.7.6      -> 1.7.6
[new tag]         1.7.7      -> 1.7.7
[new tag]         1.8.0      -> 1.8.0
[new tag]         1.8.1      -> 1.8.1
[new tag]         1.8.2      -> 1.8.2
[new tag]         1.8.3      -> 1.8.3
[new tag]         1.8.4      -> 1.8.4
[new tag]         1.8.5      -> 1.8.5
[new tag]         1.8.6      -> 1.8.6
[new tag]         1.9.0      -> 1.9.0
[new tag]         1.9.1      -> 1.9.1
[new tag]         1.9.2      -> 1.9.2
[new tag]         1.9.3      -> 1.9.3
[new tag]         2.0.0      -> 2.0.0
[new tag]         2.0.1      -> 2.0.1
[new tag]         2.0.2      -> 2.0.2
[new tag]         2.0.3      -> 2.0.3
[new tag]         2.0.4      -> 2.0.4
[new tag]         2.0.5      -> 2.0.5
[new tag]         2.0.6      -> 2.0.6
[new tag]         2.1.0      -> 2.1.0
[new tag]         2.1.1      -> 2.1.1
[new tag]         2.1.10     -> 2.1.10
[new tag]         2.1.11     -> 2.1.11
[new tag]         2.1.2      -> 2.1.2
[new tag]         2.1.3      -> 2.1.3
[new tag]         2.1.4      -> 2.1.4
[new tag]         2.1.5      -> 2.1.5
[new tag]         2.1.6      -> 2.1.6
[new tag]         2.1.7      -> 2.1.7
[new tag]         2.1.8      -> 2.1.8
[new tag]         2.1.9      -> 2.1.9
Checking out files: 100% (1819/1819), done.
HEAD is now at e6e569fd1 Merge pull request #6489 from dawidd6/create-rust
==> Homebrew is run entirely by unpaid volunteers. Please consider donating:
https://github.com/Homebrew/brew#donations
==> Tapping homebrew/core
Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core'...
remote: Enumerating objects: 5039, done.
remote: Counting objects: 100% (5039/5039), done.
remote: Compressing objects: 100% (4839/4839), done.
remote: Total 5039 (delta 46), reused 580 (delta 9), pack-reused 0
Receiving objects: 100% (5039/5039), 4.02 MiB | 3.45 MiB/s, done.
Resolving deltas: 100% (46/46), done.
Checking out files: 100% (5055/5055), done.
Tapped 2 commands and 4827 formulae (5,080 files, 12.5MB).
Already up-to-date.
==> Installation successful!

==> Homebrew has enabled anonymous aggregate formulae and cask analytics.
Read the analytics documentation (and how to opt-out) here:
https://docs.brew.sh/Analytics
==> Homebrew is run entirely by unpaid volunteers. Please consider donating:
https://github.com/Homebrew/brew#donations
==> Next steps:

Run brew help to get started
Further documentation:
https://docs.brew.sh
Xanadu:itunes-api scstraus$ brew update
Already up-to-date.
Xanadu:itunes-api scstraus$ brew install node
==> Installing dependencies for node: icu4c
==> Installing node dependency: icu4c
==> Downloading https://homebrew.bintray.com/bottles/icu4c-64.2.mojave.bottle.ta
==> Downloading from https://akamai.bintray.com/e8/e858556120acc0c2d52b8fb572b67
######################################################################## 100.0%
==> Pouring icu4c-64.2.mojave.bottle.tar.gz
==> Caveats
icu4c is keg-only, which means it was not symlinked into /usr/local,
because macOS provides libicucore.dylib (but nothing else).

If you need to have icu4c first in your PATH run:
echo 'export PATH="/usr/local/opt/icu4c/bin:$PATH"' >> ~/.bash_profile
echo 'export PATH="/usr/local/opt/icu4c/sbin:$PATH"' >> ~/.bash_profile
For compilers to find icu4c you may need to set:
export LDFLAGS="-L/usr/local/opt/icu4c/lib"
export CPPFLAGS="-I/usr/local/opt/icu4c/include"
==> Summary
🍺  /usr/local/Cellar/icu4c/64.2: 257 files, 69.2MB
==> Installing node
==> Downloading https://homebrew.bintray.com/bottles/node-12.10.0.mojave.bottle.
==> Downloading from https://akamai.bintray.com/e2/e230187b694d98b4419de13f66b56
######################################################################## 100.0%
==> Pouring node-12.10.0.mojave.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
/usr/local/etc/bash_completion.d
==> Summary
🍺  /usr/local/Cellar/node/12.10.0: 4,590 files, 53.4MB
Removing: /Volumes/Users/scstraus/Library/Caches/Homebrew/node--11.0.0.tar.gz... (34.6MB)
Removing: /Volumes/Users/scstraus/Library/Caches/Homebrew/node--npm--6.4.1.tgz... (4.5MB)
==> Caveats
==> icu4c
icu4c is keg-only, which means it was not symlinked into /usr/local,
because macOS provides libicucore.dylib (but nothing else).
If you need to have icu4c first in your PATH run:
echo 'export PATH="/usr/local/opt/icu4c/bin:$PATH"' >> ~/.bash_profile
echo 'export PATH="/usr/local/opt/icu4c/sbin:$PATH"' >> ~/.bash_profile
For compilers to find icu4c you may need to set:
export LDFLAGS="-L/usr/local/opt/icu4c/lib"
export CPPFLAGS="-I/usr/local/opt/icu4c/include"
==> node
Bash completion has been installed to:
/usr/local/etc/bash_completion.d
Xanadu:itunes-api scstraus$ node -v
v12.10.0
Xanadu:itunes-api scstraus$ npm -v
6.11.3
Xanadu:itunes-api scstraus$
Xanadu:itunes-api scstraus$ sudo npm install forever -g
/usr/local/bin/forever -> /usr/local/lib/node_modules/forever/bin/forever

<EMAIL_ADDRESS>install /usr/local/lib/node_modules/forever/node_modules/fsevents
node install

node-pre-gyp WARN Using needle for node-pre-gyp https download
node-pre-gyp WARN Pre-built binaries not installable for<EMAIL_ADDRESS>and<EMAIL_ADDRESS>(node-v72 ABI, unknown) (falling back to source compile with node-gyp)
node-pre-gyp WARN Hit error EACCES: permission denied, mkdir '/usr/local/lib/node_modules/forever/node_modules/fsevents/lib'
gyp ERR! configure error
gyp ERR! stack Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/forever/node_modules/fsevents/build'
gyp ERR! System Darwin 18.2.0
gyp ERR! command "/usr/local/Cellar/node/12.10.0/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--module=/usr/local/lib/node_modules/forever/node_modules/fsevents/lib/binding/Release/node-v72-darwin-x64/fse.node" "--module_name=fse" "--module_path=/usr/local/lib/node_modules/forever/node_modules/fsevents/lib/binding/Release/node-v72-darwin-x64" "--napi_version=4" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v72"
gyp ERR! cwd /usr/local/lib/node_modules/forever/node_modules/fsevents
gyp ERR! node -v v12.10.0
gyp ERR! node-gyp -v v5.0.3
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute '/usr/local/Cellar/node/12.10.0/bin/node /usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/usr/local/lib/node_modules/forever/node_modules/fsevents/lib/binding/Release/node-v72-darwin-x64/fse.node --module_name=fse --module_path=/usr/local/lib/node_modules/forever/node_modules/fsevents/lib/binding/Release/node-v72-darwin-x64 --napi_version=4 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v72' (1)
node-pre-gyp ERR! stack     at ChildProcess. (/usr/local/lib/node_modules/forever/node_modules/fsevents/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:209:13)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:1021:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
node-pre-gyp ERR! System Darwin 18.2.0
node-pre-gyp ERR! command "/usr/local/Cellar/node/12.10.0/bin/node" "/usr/local/lib/node_modules/forever/node_modules/fsevents/node_modules/node-pre-gyp/bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /usr/local/lib/node_modules/forever/node_modules/fsevents
node-pre-gyp ERR! node -v v12.10.0
node-pre-gyp ERR! node-pre-gyp -v v0.12.0
node-pre-gyp ERR! not ok
Failed to execute '/usr/local/Cellar/node/12.10.0/bin/node /usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/usr/local/lib/node_modules/forever/node_modules/fsevents/lib/binding/Release/node-v72-darwin-x64/fse.node --module_name=fse --module_path=/usr/local/lib/node_modules/forever/node_modules/fsevents/lib/binding/Release/node-v72-darwin-x64 --napi_version=4 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v72' (1)
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/forever/node_modules/fsevents):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>install: node install
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Exit status 1

<EMAIL_ADDRESS>added 254 packages from 160 contributors in 27.899s
Xanadu:itunes-api scstraus$ forever --help
help:    usage: forever [action] [options] SCRIPT [script-options]
help:
help:    Monitors the script specified in the current process or as a daemon
help:
help:    actions:
help:      start               Start SCRIPT as a daemon
help:      stop                Stop the daemon SCRIPT by Id|Uid|Pid|Index|Script
help:      stopall             Stop all running forever scripts
help:      restart             Restart the daemon SCRIPT
help:      restartall          Restart all running forever scripts
help:      list                List all running forever scripts
help:      config              Lists all forever user configuration
help:      set       Sets the specified forever config
help:      clear          Clears the specified forever config
help:      logs                Lists log files for all forever processes
help:      logs <script|index> Tails the logs for <script|index>
help:      columns add    Adds the specified column to the output in forever list
help:      columns rm     Removed the specified column from the output in forever list
help:      columns set   Set all columns for the output in forever list
help:      columns reset       Resets all columns to defaults for the output in forever list
help:      cleanlogs           [CAREFUL] Deletes all historical forever log files
help:
help:    options:
help:      -m  MAX          Only run the specified script MAX times
help:      -l  LOGFILE      Logs the forever output to LOGFILE
help:      -o  OUTFILE      Logs stdout from child script to OUTFILE
help:      -e  ERRFILE      Logs stderr from child script to ERRFILE
help:      -p  PATH         Base path for all forever related files (pid files, etc.)
help:      -c  COMMAND      COMMAND to execute (defaults to node)
help:      -a, --append     Append logs
help:      -f, --fifo       Stream logs to stdout
help:      -n, --number     Number of log lines to print
help:      --pidFile        The pid file
help:      --uid            Process uid, useful as a namespace for processes (must wrap in a string)
help:                       e.g. forever start --uid "production" app.js
help:                           forever stop production
help:      --sourceDir      The source directory for which SCRIPT is relative to
help:      --workingDir     The working directory in which SCRIPT will execute
help:      --minUptime      Minimum uptime (millis) for a script to not be considered "spinning"
help:      --spinSleepTime  Time to wait (millis) between launches of a spinning script.
help:      --colors         --no-colors will disable output coloring
help:      --plain          alias of --no-colors
help:      -d, --debug      Forces forever to log debug output
help:      -v, --verbose    Turns on the verbose messages from Forever
help:      -s, --silent     Run the child script silencing stdout and stderr
help:      -w, --watch      Watch for file changes
help:      --watchDirectory Top-level directory to watch from
help:      --watchIgnore    To ignore pattern when watch is enabled (multiple option is allowed)
help:      -t, --killTree   Kills the entire child process tree on stop
help:      --killSignal     Support exit signal customization (default is SIGKILL)
help:                       used for restarting script gracefully e.g. --killSignal=SIGTERM
help:      -h, --help       You're staring at it
help:
help:    [Long Running Process]
help:      The forever process will continue to run outputting log messages to the console.
help:      ex. forever -o out.log -e err.log my-script.js
help:
help:    [Daemon]
help:      The forever process will run as a daemon which will make the target process start
help:      in the background. This is extremely useful for remote starting simple node.js scripts
help:      without using nohup. It is recommended to run start with -o -l, & -e.
help:      ex. forever start -l forever.log -o out.log -e err.log my-daemon.js
help:          forever stop my-daemon.js
help:
Xanadu:itunes-api scstraus$ script/bootstrap
npm notice created a lockfile as package-lock.json. You should commit this file.
added 60 packages from 42 contributors and audited 176 packages in 7.799s
found 0 vulnerabilities

Finished setting up itunes-api! run it with script/server or install it with script/install.
Xanadu:itunes-api scstraus$ script/install
Installing itunes-api...
/Volumes/Users/scstraus/Library/LaunchAgents/org.itunes-api.plist: Path had bad ownership/permissions

I'm going to assume that I was simply misunderstanding what forever's role in this was. I think it's actually meant to restart iTunes rather than to restart the server itself. From my new install, it does appear that this happens.. Though I would like a way for the script to start automatically when iTunes starts.. Anyhow I will close this for now.


Error during WebSocket handshake: Unexpected response code: 302

Hi, i want to use this websocket method, but still i have a issue with connection, the connection state is stucked, all of the websocket connection link is correct, while i am trying to send a message to another browser user, it will return error on the console, error is "Websocket: still in CONNECTING state", i don't know why it happening

Did you release your port?


could not get porteiris to block any images

What commit ID of Portieris did you experience the problem with?
recent master, commit 46e9cd5ef06f98853ae3674a3f023108eeafad53
What went wrong?
deployed in the following environment:
kubernetes v1.9.1
helm v2.11
harbor v1.6.0 which provides registry and notary in question

command 'helm install -n portieris helm/portieris' would fail to deploy porteiris because namespace ibm-system does not exist.  It is very difficult to delete the failed helm release for re-install -- may need to keep deleting the failed jobs (due to missing namespace or service account) before helm delete times out.  The installation document may want to add a step to create namespace ibm-system and how to delete a failed porteiris release.

under the default clusterimagepolicy, unsigned images from the registry of our harbor installation should be blocked but they are not.  The test pod runs in the default namespace.  Adding a ImagePolicy below does not help;

kind: ImagePolicy
metadata:
  name: test-image-policy
  namespace: default
spec:
  repositories:
  - name: '*'
    policy:
      trust:
        enabled: true

Adding trustServer: https://notary.harbor.XYZ.com to the above polilcy did not help either, where https://notary.harbor.XYZ.com is the notary url tested.  The registry url is https://core.harbor.XYZ.com
What should have happened differently?
unsigned images should be blocked per image policy
How can it be reproduced?
follow the description above, push some unsigned images into the registry, run a pod using the unsigned images in the default namespace.
Any other relevant information
Would appreciate trouble shooting instructions.

Hi - thanks for your report. I've spun out your installation issues into different issues, so that we can keep this one to do with your issue of your ImagePolicy not working.
First of all, I understand your frustration with the jobs lying around if the Helm chart fails to install. The jobs are a workaround for a limitation with Helm on initial release, in that it wasn't able to create custom resource definitions. I've raised an issue to revisit those jobs (#41), because it might be possible to properly integrate them into the chart.
I thought we created the ibm-system namespace as part of the chart, but evidently not. Serves us right for only testing on IKS! #42
As for your ImagePolicy issue, that looks like it should work. Please could you confirm that the ImagePolicy you created has the apiVersion field set correctly? I'd expect it to be right at the top of the yaml. It should be:
apiVersion: securityenforcement.admission.cloud.ibm.com/v1beta1

For further debugging, Portieris's logs should help identify what's going on. If you have stern, you can run stern portieris -n ibm-system to get all the logs for the multiple Portieris instances. If not, it might be easier to scale the Portieris deployment to one instance so that you only need to do kubectl logs on one pod.

The image policy has the verion string.  Here are the full image policy, which has disable_tofu: true added added later but did not help either:
apiVersion: securityenforcement.admission.cloud.ibm.com/v1beta1
kind: ImagePolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"securityenforcement.admission.cloud.ibm.com/v1beta1","kind":"ImagePolicy","metadata":{"annotations":{},"name":"test-image-policy","namespace":"default"},"spec":{"repositories":[{"name":"*","policy":{"trust":{"disable_tofu":true,"enabled":true}}}]}}
  clusterName: ""
  creationTimestamp: 2018-12-01T05:58:39Z
  name: test-image-policy
  namespace: default
  resourceVersion: "25713101"
  selfLink: /apis/securityenforcement.admission.cloud.ibm.com/v1beta1/namespaces/default/imagepolicies/test-image-policy
  uid: 266a3ff7-f52e-11e8-9891-fa163eb8d59c
spec:
  repositories:
  - name: '*'
    policy:
      trust:
        disable_tofu: true
        enabled: true

The portieris pod logs are all empty, containing only something like  I1201 05:34:44.599868       1 webhook.go:109] Starting notary Webhook on port 8000...

Since you've got no messages about having received a request in the logs, I suspect that your admission webhook isn't working. Please could you print the result of:
kubectl get MutatingWebhookConfiguration image-admission-config -o yaml

Make sure to redact the caBundle field.

Here is the image-addmission-config:
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"admissionregistration.k8s.io/v1beta1","kind":"MutatingWebhookConfiguration","metadata":{"annotations":{},"labels":{"app":"portieris","chart":"portieris-0.2.0","heritage":"Tiller","release":"portieris"},"name":"image-admission-config","namespace":""},"webhooks":[{"clientConfig":{"caBundle":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNyRENDQVpRQ0NRQzFBU2FwMWl0NTBEQU5CZ2txaGtpRzl3MEJBUXNGQURBWE1SVXdFd1lEVlFRRERBeHcKYjNKMGFXVnlhWE5mWTJFd0lCY05NVGd3TkRJeU1UTXlOak01V2hnUE1qSTVNakF5TURVeE16STJNemxhTUJjeApGVEFUQmdOVkJBTU1ESEJ2Y25ScFpYSnBjMTlqWVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDCkFRb0NnZ0VCQU9JNitUVlYxZHlKNVlqbk9VZjNlKzlRd0wyMWEvSHlyNCtZdTBuZUhKMFViaGNHa3VLa0NudXQKUTBVNWVic2YvY1VNT2ZYVzVsMXlReTgyM3Fad0M5czdZWlZzdjd5R0dKSnVXTEhiRW82R2x0eDR3Mm9uMkdtbgpJVDJrTEZNaHRLTGh4anM4RjBzU1V3Zm5nQ3FLVFRZZ1l4OTNOVGM2WlFYZlNpenBpNDNqSTdLQWtkZWlDUEJwCkhzMVJ2ei9mVys1TUw4OTMwajRqL1RqVWNnTnk2cHlsbEdtYVdPemcxbkYrcjYyVDZuRzZNSkxMWlRLNWFhOC8KOUVBWlhRczdYOUJURmlLeSt4TjBCVlk5TWZKNDJNY3A3NndzT21mR29TUm9JbGxBRWtNaGRwRkZvTGR0YWp2agp5OW1DVTVFZHB3VUY1RkZtOHk2a0FZWFpRYmhteElrQ0F3RUFBVEFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBCjJSYkpqYTNUSzBlRjRLUUw5T2hNTHVVYkFZNzc1WFBTT3VXMkVjR05XRUs1OXA1UG1VdjhzeWplcEUwM2dYZ0EKSkxINENQcE1POGt4VStmdCtwTjFaRmNwT1NmSnFuMkx6WU95eTBQVjVtM3NPWEpzRHVWUGRXWmN4VUZVWGtqYgp4QjU4NVZURTRiSU1nT1Vyc3I3eVoraHU0SFlzRFBzeXZNbHBHcDQrSDBBOU9HZmwwaXVGL1BjTG5keVdWV0NPCkI5Z3dYMUdlNEdGbms3cWJGdUplVWwwN3BoL3VnOE5xTTdVOC9HcG1uMlVrbFNtTlRhd09kQW9kR2VUSTlCaTUKQmNyaWpVVERhRVhlNHhvTTNTR3NIaFVUTUhOTk9XdzMyR0tzeGEyZU1YL0RrdXhWN1pTYktLdUQvU1RhWkdFNgpIcDFBY2thajl5eDE1YWYzL0hldnlnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=","service":{"name":"portieris","namespace":"ibm-system","path":"/admit"}},"failurePolicy":"Fail","name":"trust.hooks.securityenforcement.admission.cloud.ibm.com","rules":[{"apiGroups":["*"],"apiVersions":["*"],"operations":["CREATE","UPDATE"],"resources":["pods","deployments","replicationcontrollers","replicasets","daemonsets","statefulsets","jobs","cronjobs"]}]}]}
  creationTimestamp: 2018-12-01T05:35:00Z
  generation: 1
  labels:
    app: portieris
    chart: portieris-0.2.0
    heritage: Tiller
    release: portieris
  name: image-admission-config
  resourceVersion: "25711031"
  selfLink: /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/image-admission-config
  uid: d8b2b2b6-f52a-11e8-9891-fa163eb8d59c
webhooks:
- clientConfig:
    caBundle: ...redacted...
    service:
      name: portieris
      namespace: ibm-system
      path: /admit
  failurePolicy: Fail
  name: trust.hooks.securityenforcement.admission.cloud.ibm.com
  namespaceSelector: {}
  rules:
  - apiGroups:
    - '*'
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - pods
    - deployments
    - replicationcontrollers
    - replicasets
    - daemonsets
    - statefulsets
    - jobs
    - cronjobs

Have you got any other mutating admission controllers running? It may be that your Kubernetes API server isn't configured to call them. https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#is-there-a-recommended-set-of-admission-controllers-to-use

Thanks for the information.  We are using k8s 1.9.1, and the apiserver had the following parameter (default):
--admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota
I replaced it in the /etc/kubernetes/manifests/kube-apiserver.yaml with the following according to the above document:
--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
The net change seems to replace Initializers and NodeRestriction with MutatingAdmissionWebhook and ValidatingAdmissionWebhook.
After deleted the apiserver pod and verified that the restarted apiserver had the intended parameter, an new problem developed: k8s could not find the image pulling secret when starting a pod until I changed the image policy to disable trust.
The error message when creating a pod (with minor obfuscations):
Error from server (InternalError): error when creating "pod-signed.yaml": Internal error occurred: admission webhook "trust.hooks.securityenforcement.admission.cloud.ibm.com" denied the request:
Deny "core.harbor.domain.com/library/alpine:3.6", no valid ImagePullSecret defined for core.harbor.domain.com

The above error would go away after I added the following image policy:
cat <<EOF | kubectl apply -f -
apiVersion: securityenforcement.admission.cloud.ibm.com/v1beta1
kind: ImagePolicy
metadata:
  name: open-image-policy
  namespace: default
spec:
   repositories:
    - name: "core.harbor.domain.com/*/*"
      policy:
        trust:
          enabled: false
EOF

In  a portieris pod log, I found the following errors:
E1208 01:10:31.454882       1 oauth.go:73] Error sending request to registry-oauth: Post https://core.harbor.domain.com/oauth/token: x509: certificate signed by unknown authority
E1208 01:10:31.454916       1 controller.go:139] Post https://core.harbor.domain.com/oauth/token: x509: certificate signed by unknown authority

Our registry is using basic authentication, and the k8s cluster is otherwise configured to work with it ok when trust is disabled, i.e., no x509 certificate issue, and can use the existing image pulling secret.
Before adding the open-image-policy described above, the default cluster image policy was in effect.  Creating an image policy with the proper trustServer did not help:
cat <<EOF | kubectl apply -f -
apiVersion: securityenforcement.admission.cloud.ibm.com/v1beta1
kind: ImagePolicy
metadata:
  name: private-image-policy
  namespace: default
spec:
   repositories:
    - name: "core.harbor.domaiin.com/*/*"
      policy:
        trust:
          enabled: true
          trustServer: "https://notary.harbor.domain.com"
EOF

I think you need to put Harbor's CA into the Portieris container's trusted CAs.

I think @molepigeon is correct. The Notary client requires TLS and certs signed by a trusted CA. We added a feature in 0.5.1 that allows you to mount the Harbour CA's cert as part of the Kube secret as well. See https://github.com/IBM/portieris/blob/master/cmd/trust/main.go#L38.
apiVersion: v1
kind: Secret
metadata:
  name: portieris-certs
  namespace: ibm-system
type: Opaque
data:
  serverCert.pem: {{ .Files.Get "certs/serverCert.pem" | b64enc }}
  serverKey.pem: {{ .Files.Get "certs/serverKey.pem" | b64enc }}
  ca.pem: <base64-encodeed-harbor-self-signed-CA>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: portieris
  namespace: ibm-system
  labels:
    app: portieris
spec:
  replicas: 1
  selector:
    matchLabels:
      app: portieris
  template:
    metadata:
      labels:
        app: portieris
    spec:
      serviceAccountName: portieris
      containers:
        - name: portieris
          image: "docker.io/ibmcom/portieris:0.5.1"
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          volumeMounts:
          - name: portieris-certs
            readOnly: true
            mountPath: "/etc/certs"
      volumes:
      - name: portieris-certs
        secret:
          secretName: portieris-certs

These are used by Portieris to communicate with Kube API server, they should be generated using this script. They will be automatically populated by helm if you run that script.
If you're just playing around you won't need to generate your own certs, but these certs are in GitHub so don't use them in production!

After adding the ca.pem, I got the same errors.   I tried to override only the /etc/certs/ca.pem, but seems to hit a k8s bug.  Whatelse is supposed to be in /etc/certs?  cmd/trust/certs.go seems to suggest there are at least two more files.  It might be worth trying to create a full /etc/certs then mount to the pods.
Here are the details.

I used the gencerts.sh to generate the serverCert and serverKey files, and supplied the ca file, added the secret to the deployment file, then updated its volumes, and the following is the result as printed by helm with the --debug parameter:

# Source: portieris/templates/deployment.yaml
### Secret and related volume added per https://github.com/IBM/portieris/issues/40
### deployment/upgrade process
### need yq
### sudo pip install  --trusted-host pypi.org --trusted-host files.pythonhosted.org --upgrade pip
### sudo pip install  --trusted-host pypi.org --trusted-host files.pythonhosted.org yq
### setup env for running helm, especially, define env KUBECONFIG
### cd .../portieris/helm
### ./gencerts.sh
### cleanpem.sh .../the.ca # the.ca is the root ca cert used by our notary, this step strips comments
### cp .../the.ca.pm certs/harborca.pem
### mv certs portieris
### helm upgrade portieris helm/portieris
apiVersion: v1
kind: Secret
metadata:
  name: portieris-certs
  namespace: ibm-system
type: Opaque
data:
  serverCert.pem: ...
  serverKey.pem: ...
  ca.pem: ...
---
# Source: portieris/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: portieris
  namespace: ibm-system
  labels:
    app: portieris
    chart: portieris-0.2.0
    release: portieris
    heritage: Tiller
spec:
  replicas: 3
  selector:
    matchLabels:
      app: portieris
      release: portieris
  template:
    metadata:
      labels:
        app: portieris
        release: portieris
    spec:
      serviceAccountName: portieris
      volumes:
      - name: portieris-certs
        secret:
          secretName: portieris-certs
      containers:
        - name: portieris
          image: "docker.io/ibmcom/portieris:0.4.0"
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          env:
          volumeMounts:
          - name: portieris-certs
            readOnly: true
            mountPath: "/etc/certs"
          resources:
            {}

after getting the same error with the above update, tried to override only /etc/certs/ca.pem:

# Source: portieris/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: portieris
  namespace: ibm-system
  labels:
    app: portieris
    chart: portieris-0.2.0
    release: portieris
    heritage: Tiller
spec:
  replicas: 3
  selector:
    matchLabels:
      app: portieris
      release: portieris
  template:
    metadata:
      labels:
        app: portieris
        release: portieris
    spec:
      serviceAccountName: portieris
      volumes:
      - name: portieris-certs
        secret:
          secretName: portieris-certs
      containers:
        - name: portieris
          image: "docker.io/ibmcom/portieris:0.4.0"
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          env:
          volumeMounts:
          - name: portieris-certs
            readOnly: true
            mountPath: "/etc/certs/ca.pem"
            subPath: "ca.pem"
          resources:
            {}

the portieris pod wont start, with the following errors (from kubectl describe pod), might be related to https://github.com/kubernetes/kubernetes/issues/61563:
Warning  Failed                 <invalid> (x8 over <invalid>)  kubelet, sladovmap67  Error: failed to prepare subPath for volumeMount "portieris-certs" of container "portieris"

You need to use the most recent charts with Portieris 0.5.1.

I seems to get the same error.   At the command line:
kubectl create -f .../kube/yaml/pod-signedalpine.yaml
Error from server (InternalError): error when creating ".../kube/yaml/pod-signedalpine.yaml": Internal error occurred: admission webhook "trust.hooks.securityenforcement.admission.cloud.ibm.com" denied the request:
Deny "core.harbor.domain.com/library/alpine:3.6", no valid ImagePullSecret defined for core.harbor.domain.com

in the portieris log:
I1215 03:05:53.138338       1 controller.go:65] Processing Trust Admission Request for CREATE on
I1215 03:05:53.143057       1 controller.go:109] Container Image: core.harbor.domain.com/library/alpine:3.6   Namespace: default
I1215 03:05:53.156502       1 controller.go:119] Trust is enforced
E1215 03:05:53.388337       1 oauth.go:73] Error sending request to registry-oauth: Post https://core.harbor.domain.com/oauth/token: x509: certificate signed by unknown authority
E1215 03:05:53.388563       1 controller.go:139] Post https://core.harbor.domain.com/oauth/token: x509: certificate signed by unknown authority

we are interacting with the latest Harbor registry deployed today, master branch.
portieris was deployed using the latest master branch a moment ago with the following modifications:

Added harborca.pem to helm/portieris/certs/
verified using openssl that the certs presented by https://core.harbor.avaya.com/ and helm/portieris/certs/harborca.pem form a good certificate chain of three certificates.  Is there a command that we can run in the portieris pod to print out its /etc/certs/ca.pem?
helm/portieris/templates/secret.yaml reads like the  following (adding last line):

apiVersion: v1
kind: Secret
metadata:
  name: portieris-certs
  namespace: ibm-system
type: Opaque
data:
  serverCert.pem: {{ .Files.Get "certs/serverCert.pem" | b64enc }}
  serverKey.pem: {{ .Files.Get "certs/serverKey.pem" | b64enc }}
  ca.pem: {{ .Files.Get "certs/harborca.pem" | b64enc }}

helm/portieris/templates/deployment.yaml reads like the following (kept our comments at the  beginning but otherwise should be the same as the current master version):

---
### Secret and related volume added per https://github.com/IBM/portieris/issues/40
### deployment/upgrade process
### need yq
### sudo pip install  --trusted-host pypi.org --trusted-host files.pythonhosted.org --upgrade pip
### sudo pip install  --trusted-host pypi.org --trusted-host files.pythonhosted.org yq
### setup env for running helm, especially, define env KUBECONFIG
### cd .../portieris/helm
### ./gencerts.sh
### cleanpem.sh /app/pwu/certs/snodes.ca
### cp /app/pwu/certs/snodes.ca.pm certs/harborca.pem
### mv certs portieris
### helm upgrade portieris helm/portieris
### Secret portieris-certs in 0.5.1 is defined in secret.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ template "portieris.fullname" . }}
  namespace: ibm-system
  labels:
    app: {{ template "portieris.name" . }}
    chart: {{ template "portieris.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ template "portieris.name" . }}
      release: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ template "portieris.name" . }}
        release: {{ .Release.Name }}
    spec:
      serviceAccountName: portieris
      volumes:
      - name: portieris-certs
        secret:
          secretName: portieris-certs
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.host }}/{{ .Values.image.image }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          volumeMounts:
          - name: portieris-certs
            readOnly: true
            mountPath: "/etc/certs"
          env:
          resources:
{{ toYaml .Values.resources | indent 12 }}
    {{- with .Values.nodeSelector }}
      nodeSelector:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.affinity }}
      affinity:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
{{ toYaml . | indent 8 }}
    {{- end }}
      volumes:
      - name: portieris-certs
        secret:
          secretName: portieris-certs

You will need to add an image pull secret to use with Notary.

Hi, I am seeing the "no valid ImagePullSecret" as well when launching a Pod and providing imagePullSecrets in the pod template. Does Portieris not need a separate ImagePullSecret of its own? There seems to be no information about this online.

This might be the same issue as reported in #51. That issue describes Portieris wrongly assuming that the oauth endpoint is always /oauth/token on the registry's DNS server. You can confirm whether or not this is the case for your registry by making an unauthenticated GET request to /v2/ on the Notary server and looking at the realm field in the Www-Authenticate header of the response. If the realm says that you should go anywhere other than /oauth/token, you're affected by the same issue.

closing due to lack of activity, also #51 is closed


Gnosis DAO Phase 1 Doc

follow format of previous samples.

added issue to Gnosis milestone

Writing Aide
Follow examples here for format of Phase 1.


Cleave

The cleave affix attacks neutral mobs, most importantly villagers.
Minecraft version 1.16.5 and Apotheosis 4.8.99F
(Also in case the updates for 1.16.5 have stopped this could also maybe pass on to the other version when the deadly module gets added)

Fixed in 1.18

(1.18.2) Cleave will no longer hit villagers or animals unless the primary target was one.


add int-to-separator-string method

add toStringWithSeparated method.
This method returns a string of numbers separated by national delimiters.

conflicts 🤔

Looks good to me except the difference of .idea/misc.xml is included.
It's my fault that it is old, but I think the difference should be removed from this PR :)

@okuzawats Sorry, I have restored.


Google Assistant - "my test app" unable to login to HA for integration

Home Assistant release with the issue:
.101.2
Last working Home Assistant release (if known):
.99
Operating environment (Hass.io/Docker/Windows/etc.):
Windows
Integration:
Google Assistant
Description of problem:
Google Assistant unable to log into HA via API or username for HA integration.  HA login screen comes up on phone to grant access to Google Assistant for "my test app", but the login fails to allow access for Google Assistant.
Additional information:
Refresh Tokens are being added to the legacy API login when attempts are made to add GA to HA, but the request fails on the Google Assistant side saying "Couldn't update the setting. Check your connection".
Could this be related to the Breaking Change in .101 regarding the legacy API?  It does have this in the configuration:  https://[YOUR HOME ASSISTANT URL:PORT]/api/google_assistant
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

google_assistant:
  project_id: Home-Assistant-21406
  api_key: secret-api-key
  expose_by_default: false
  exposed_domains:
  entity_config:
    switch.beep_beep:
      expose: true
    switch.outdoor_lamp_post:
      expose: true
    switch.outdoor_front_lights:
      expose: true
    switch.marks_office_salt_lamp_switch:
      expose: true
    switch.marks_office_m_light_switch:
      expose: true
    switch.alexs_room_light:
      expose: true
    light.marks_office_light:
      expose: true
    script.random_colors_on:
      expose: true
    script.random_colors_off:
      expose: true
    scene.daylight:
      expose: true
    sensor.outdoor_temperature:
      expose: true
    sensor.data_center_temperature:
      expose: true
    binary_sensor.front_door:
      expose: true
    input_boolean.whitenoise_mark_candice_room:
      expose: true
    input_boolean.whitenoise_alex_room:
      expose: true

http:
  base_url: http://my-ha-url:port#

homeassistant:
  auth_providers:
    - type: homeassistant
    - type: legacy_api_password
      api_password: !secret http_password
    - type: trusted_networks
      trusted_networks:
        - <IP_ADDRESS>/32
        - <IP_ADDRESS>/32
  # Name of the location where Home Assistant is running
  name: Home
  # Location required to calculate the time the sun rises and sets
  latitude: 39.568669
  longitude: -76.337701
  # Impacts weather/sunrise data (altitude above sea level in meters)
  elevation: 485
  # metric for Metric, imperial for Imperial
  unit_system: imperial
  # Pick yours from here: http://en.wikipedia.org/wiki/List_of_tz_database_time_zones
  time_zone: America/New_York
  # Customization file
  customize: !include customize.yaml
  whitelist_external_dirs:
    - d:\camera_capture

Refresh Tokens are being added to the legacy API login when attempts are made to add GA to HA, but the request fails on the Google Assistant side saying "Couldn't update the setting.  Check your connection".

Deleted my entire HA<->GA integration on both HA and GA, and re-setup everything from scratch.  For whatever reason, that corrected the problem.  Closing the issue.


layers.Label parameter "color_dict" should be renamed

📚 Documentation
#1339 fixed a longstanding issue of directly mapping labels to colors. However, the parameter name "color_dict" is too restrictive in my opinion, since many color schemes can be conceived for labels. My suggestion is colormap which can be a callable mapping labels to colors, or a dict, or floats to colors, and could be in the future expanded to include e.g. an object that includes a GL shader to pass to vispy so that the full colormapping (labels -> graphics) can be done on the GPU, addressing #204 .
This issue should be addressed before the next release so that we don't have to deprecate an API keyword shortly after introducing it.

Ooo, sounds very intriguing @jni. Picking up on https://github.com/napari/napari/pull/1339#issuecomment-643723379

I think this PR should be reworked to match more closely what @kevinyamauchi did with point colouring, but with the additional "random" mode to match existing behaviour. Specifically, there could be a "direct" mode that grabs the colour from a single property (it does not need to be named "color" @sofroniewn, it could have any name and be settable), "colormapped" that uses a colormap, or "auto"/"random" that uses the current behaviour.

The next plan for this chunk of work was going to be to support coloring based on properties, see some comments from @kevinyamauchi here https://github.com/napari/napari/pull/1339#issuecomment-639650482. If we could think from an API perspective what we'd like that to look like and then rename this kwarg appropriately so that we don't have to rename it again, I think that would be best.
I'm not so sure about colormap as in the Image layer which this subclasses colormap means one very specific thing and now we'd have it mean something more general. I will think on it more too.

This issue should be addressed before the next release so that we don't have to deprecate an API keyword shortly after introducing it.

Agreed

Looking at Points https://github.com/napari/napari/blob/master/napari/layers/points/_points_constants.py#L6-L15
class ColorMode(StringEnum):
    """
    ColorMode: Color setting mode.
    DIRECT (default mode) allows each point to be set arbitrarily
    CYCLE allows the color to be set via a color cycle over an attribute
    COLORMAP allows color to be set via a color map over an attribute
    """

I read DIRECT mode as setting arbitrarily with the rgba array - which is why it is similar to the color_dict in Labels and that COLORMAP would be more akin to grabbing the color from a single property.
I guess stepping back what do I think we need:

A place to store what color mode we're in
Our autogenerated colormap
A color dictionary based on labels
A color dictionary based on properties.
If we're using properties the name of the property that is being mapped to the dictionary if it is categorical (right now we don't really support this for points, we just support the cycle, but the dictionary approach might be more natural for string labeled categories too see comment in #1348

In general, I find specifying the color_cycles via lists and then the mapping via zip(np.unique(values), color_cycle) not intuitive and think maps would make a lot more sense here.

Stepping back I think we should create a small dedicated Color object, thinking along the lines of #1343 that is completely reusable across Points, Shapes, Vectors, and Labels. For shapes we'd have two instantiations, one for edge_color and one for face_color. We'd then be able to access attributes like color.map, color.mode, color.dict, color.contrast_limits etc as needed. Note right now we have attributes like face_color, face_color_cycle, face_colormap, face_contrast_limits and it is all quite confusing and many things are duplicated or out of sync between our layers.
Thoughts @ziyangczi @jni @kevinyamauchi?

 It's great seeing you all over github again :)
@sofroniewn, thank you for the proposal. I totally agree that the "flat" approach of attaching all properties to the layer model class is getting out of hand. When using these Color objects to store the color-related properties, would we still expose the individual properties in the __init__ method? I would prefer that we avoid asking the user to instantiate and pass a Color object to the init method. I suppose an alternative would be to allow the user to pass a dictionary:

# make a points layer where all points have the same color
layer = viewer.add_points(data, face_color='blue')

# make a points layer where each point has a different color
color_array = ['blue', 'green', 'magenta']
layer = viewer.add_points(data, face_color=color_array)

# make a points layer with the color properties individually set
edge_color = {
    'color': 'point_type',
    'mode': 'cycle',
    'cycle': ['magenta', 'blue', 'green']
}
properties = {'point_type': ['A', 'B', 'C']}
layer = viewer.add_points(data, properties=properties, edge_color=edge_color)

With this approach, it would also be pretty easy to keep the current kwargs in parallel (with a deprecation warning) so that people have time to change over to the new API.
I also agree with @jni that we probably want to make all of these changes at once, as it's a pretty big breaking change. From my side, it would be helpful to plan out when we want to do this a bit so that I can make sure I can carve out some time.
Also super minor and we can decide this later, but I just wanted to note that we might want to choose a different name, as there is a Color class in vispy.

I'm not so sure about colormap as in the Image layer which this subclasses colormap means one very specific thing and now we'd have it mean something more general. I will think on it more too.

I'm just spitballing, but maybe label_map or label_colormap?

I would prefer that we avoid asking the user to instantiate and pass a Color object to the init method.

We are all on the same page. The Color object, which should be renamed to ColorManager or ColorMapper or similar rather than just Color, should remain private, at least in the beginning. As I've said many times before, I would like to keep the number low of new classes that people have to learn to use napari effectively.
I'm also hoping that we don't even have to deprecate the current approach, but let's see how these ideas evolve.

I agree with the spirit of the proposal: having a unified approach to manage colors, when I implemented the feature in question: adding a dictionary of colors, I was looking at Points layer and found it approaching colors differently. It might be due to them having a different characteristic of data requirements. Labeling requires a large area of data and using a color map with float would allow the representation of color in the least amount of space. while Pointing has less concern of data usage, and can use RGBA array to store the color information. This does create some difficulties when developers move from layer to layer as their need to relearn the coloring mechanism that is used in a specific layer.

As #1362 has now merged, and this issue has a good discussion about color refactoring more broadly I am going to leave it open but rename the issue to something more clear.
I really like the approach that @kevinyamauchi proposes where we "overload" the public API like that, but internally have an object that can handle the full range of possibilities with everything neatly organized. As @ziyangczi said, a bit more thought is probably needed in the design phase before jumping into the coding

Regarding the favored "Color" or "ColorManager" approach, I have a slightly related question: how many colors are really needed? by virtual I mean sure, there is practically an infinite amount of colors that a user can input with an RGBA array, but do we ever have a use case for that? given that when the colors are no longer distinguishable on screen. there is practically no point separating them since coloring is solely for display purposes. Wouldn't it make sense to restrict the color manager to a subset of colors, with names defined in a color dictionary? (Don't even get me to the mathematical 4 coloring theorm 😆 )

Wouldn't it make sense to restrict the color manager to a subset of colors, with names defined in a color dictionary?

We do support a couple named color dictionaries internally, see here for the XKCD colors that we vendor https://github.com/napari/napari/blob/master/napari/utils/colormaps/vendored/_color_data.py#L55 but we're also need to support hex values and people setting rgb values directly - I personally do that all the time for things like shades of gray where it is much more natural for me to think about the rgb values then remember what light grey or dark grey is.
In general our public API should allow most of https://matplotlib.org/3.1.0/gallery/color/color_demo.html. At that point having the internal representation as rgba seemed the most natural as you can always map into that space.
I see the ColorManager as being more about controlling the relationship between the data and how it is colored (i.e. colormap, color cycle, color dict, with properties/ random etc) rather than just about how we store color.


Retry.Multiplier - no way to know if its undefined

in the struct Retry.Multiplier which is a floatstr.Float32OrString, there's no way to distinguish this field as not defined.  In the spec, this is supposed to be an optional field.  The default struct of Float32OrString will be Type Float and the zero value of FloatVal will be a 0.0 - which will be value if the workflow json/yaml left this out.
Unless i take Multiplier of 0 to be "not defined"?  in which case, the specs need to be updated to say zero value is not allowed or treated as not defined

@globalflea agree, can you open issue in spec repo to add that multiplier smallest value should be 1. It makes no sense imo to have a 0 multiplier.

@globalflea do you mind opening a PR to fix this?

@globalflea do you mind opening a PR to fix this?

I am new to open source and not too sure what you mean.. Are you asking me to open a Pull Request - and I fix it?

@globalflea yeah we are community-driven project and community help on reporting and contributing to fixing bugs is something we rely on.
The level of contributions can of course vary depending on the person, but all help is more than welcome.
If you don't feel comfortable with contributing code, that's perfectly fine, just note that in some cases contributing a fix yourself might get you unblocked faster.

No problems, don't think I have any issues fixing the code. Since I am fairly new to this, patience on your part may be required.

You did great :)


ERROR:  value "2148732281" is out of range for type integer

Got this error when importing php/php-src repo.

I got this using posgresql as well for database_id for both issues and pull requests. According to the REST api, the database_id is an int64. However, the database schema has the database_id as a postgres integer which is an int32.
This field needs to be converted to a postgresql bigint which is an int64. I think this might be an expensive migration as the entire table will need to be rewritten on the disc due to the change in block size. I am unsure if this will be a problem or not

This should be resolved by this PR: https://github.com/mergestat/syncs/pull/97 which integrates your changes @RobinCombrink (thank you for the contribution!)
It will require a newly intialized (empty) database however, as the changes do not handle migrating from the previous table schema

I am currently working on a migration for this repository as sqlc will generate int32 for the Go code.
The UI code interprets this as a JavaScript number, which can represent integers up to 2^53. This is smaller than SQL’s bigint, which can represent integers up to 2^64. Unfortunately, my experience with the UI is limited, I don’t foresee this causing any immediate issues as 2^53 is significantly larger than 2^32.


Removed twig version restriction

It seems to be working fine with symfony 3.2 and twig 2.1
Fixes #75

👍

Would be great if you could tag a new version with this fix in it. It currently keeps me from getting to Twig 2.0.

@hvt done


Fix of process locking functionality on Windows

This is the re-submission of PR #1547.
I restarted from the master branch due to the merge problem.
Below is the original PR message:
The current version of
luigi.lock.getpcmd does not work properly on Windows, because Windows does not have ps command which is used to get command in the function. As a result, locking functionality does not work on Windows.
Fix the function to use wmic command instead of ps when luigi is run on Windows.

thanks


Can it draw in a item of recycle view

Can it draw in a item of Recycle view? I tried but it did not show anything when recording.

Sorry sir,
It is my coding mistake. I found it.
Thanks


Bugfix/issue 2052

Both of these had to be made together to pass our tests

always pass absolute url to json-schema-ref-parser
update json-schema-ref-parser to v11.1.0

Should fix: https://github.com/ferdikoomen/openapi-typescript-codegen/issues/2052
(Original PR: https://github.com/ferdikoomen/openapi-typescript-codegen/pull/2053#issue-2164840140)

Thanks @tajnymag, this has been released! 🚀

@tajnymag the upstream package says it also fixed this issue if I'm reading the changelog correctly. I upgraded to the latest version yesterday. Are you able to check if it works on Windows if you revert your changes?

@mrlubos Went back to the state of the repository before this PR. Tests were failing. Upgraded to the latest schema parser and the tests seem to be succeeding again. Good 👍

@tajnymag should we revert our fix then?

@mrlubos Sure. I don't really mind.


Needs updated with cryptography crack

They cracked the unknown6 cryptography in the original pgoapi. Needs updated here.

Already an issue for this. Mod please close this.

Duplicate of #23.


[WIP] Refactor to stateless validators

This work-in-progress is exploring how we might approach stateless validators.
Essentially, instead of an isValid() method returning a boolean, and a subsequent call on the validator to retrieve validation error messages, we would instead:

Define a Result interface modeling the results of validation; it would compose the value validated, validation status, and, if present, any validation error messages.
Define validators would define a single validate() method, accepting both a value and optional context, and return a Result instance.
Define a ResultAggregate interface for aggregating several results, as is necessary in a ValidatorChain; Result instances would be pushed upon an aggregate.

Translation, value obscuration, and message truncation then become presentation issues, and can be achieved by decorating Result instances.
Additionally, we'd remove the concept of $options for creating individual validator instances; they would instead have concrete constructor arguments. This simplifies a ton of logic internally, but means that each validator would require its own factory class. On the flip side, it also means developers can write factories for specific options combinations, and have shared instances without worrying about state.
Migration concerns
We could create a new minor release that adds the new Validator, Result, and ResultAggregate interfaces, and various Result implementations. Validator would define validate instead of isValid(), allowing devs to implement both in order to forward-proof their validators. We could also provide a wrapper for Validator implementations to make them work as ValidatorInterface instances; in such a case, it would pull info from the result in order to populate its members.
The bigger issue is existing validators, and extensions to them. Developers extending existing validators may want to copy/paste implementations and begin migration of those to make them forwards-compatible with version 3. Since we would have version 3 released simultaneously to a v2 with the new interface additions, they could even copy those from v3 to aid their migration.
Integration concerns
I have not yet investigated impact on zend-inputfilter; however, that version will require a similar refactor towards stateless inputs/input filters as well.

Originally posted by @weierophinney at https://github.com/zendframework/zend-validator/pull/181

The bigger issue is existing validators, and extensions to them. Developers extending existing validators may want to copy/paste implementations and begin migration of those to make them forwards-compatible with version 3. Since we would have version 3 released simultaneously to a v2 with the new interface additions, they could even copy those from v3 to aid their migration.

Every single custom validator and every extension to a supplied validator will need to be rewritten?

Originally posted by @akrabat at https://github.com/zendframework/zend-validator/pull/181#issuecomment-316092263

Every single custom validator and every extension to a supplied validator will need to be rewritten?

Yes - as isValid() goes away, and is replaced by validate(), and the return values would differ. This is, of course, why it's targeted for a new major version.
My hope is that we can provide some tooling and guidance around this. For instance, something like the following trait would allow adapting an existing validator to implement the new Validator interface:
trait LegacyValidator
{
    public function validate($value, array $context = null) : Result
    {
        if ($this->isValid($value, $context)) {
            return ValidatorResult::createValidResult($value);
        }

        return ValidatorResult::createInvalidResult(
            $value,
            $this->getMessageTemplates(),
            $this->abstractOptions['messageVariables']
        );
    }
}

class ExistingValidator extends AbstractValidator implements Validator
{
    use LegacyValidator;

    /* ... */
}

If we provide the legacy interface in v3, but mark it deprecated, this approach would allow mixing and matching existing v2-capable validators with v3, providing a longer upgrade path.

Originally posted by @weierophinney at https://github.com/zendframework/zend-validator/pull/181#issuecomment-316098432

I think there's going to have to be a compelling advantage to v3 or upgrading is going to have to be quite easy :)  A quick check of one of my projects shows that I have 30 or so Validators that would need upgrading and that won't be an easy sell to the client if there's no tangible benefit if it's more than a day's work.
I assume that the trait won't work for validators that extend current validators, so they'll have to be re-done.

Originally posted by @akrabat at https://github.com/zendframework/zend-validator/pull/181#issuecomment-316100173

I think there's going to have to be a compelling advantage to v3

One of the issues currently is that stateful validators can lead to hard to debug issues, particularly if you share an instance.
As an example:
$validator->isValid($value1);
$validator->isValid($value2);
$messages = $validator->getMessages(); // $value1 messages are missing
$value = $validator->getValue(); // $value2

The other issue I've been seeing is maintenance of the AbstractValidator, as it contains a ton of logic around gathering options, formatting messages, etc:

Due to the options based approach, we have to do a lot of internal testing to see if the validator is in a valid state before we validate — does it have the necessary options? are they of the correct type? etc.
Moving presentation logic (value obscuration, translation, message truncation, etc.) into results, result decorators, or DTOs greatly reduces the logic, and thus maintenance, of that base class.

One nice benefit of the approach is that we can finally use callables for validation more cleanly. If they return a Result instance, this is no different than using a Validator for a consumer. The Callback validator can be used when you want to return a boolean result instead; you would compose your message templates and variables in that, so it can return a Result. This simplifies the creation and usage of one-off validators; you no longer need to have classes. (That said, PHP 7's anonymous classes also makes this easier.)

Originally posted by @weierophinney at https://github.com/zendframework/zend-validator/pull/181#issuecomment-316102841


colheaders.tsv format

For the munge -C colheaders.tsv option, what is the format of the colheaders.tsv file?  The -c options does not include ourput from GENESIS (https://github.com/UW-GAC/GENESIS).

Did you check the Column Headers Mappings section in the documentation? It shows how to create that file. If it does not have the column headers from GENESIS let me know

Thanks!  I found that section.  The column headers for GENESIS were not there. Their columns had a lot of small caps. So I made a genesis.colheaders.tsv.  It looks like this:
rsid    SNP
P       P
n.obs   N
chr     CHR
pos     BP
Est     BETA
alt     A1
ref     A2
Est.SE  SE
eff.frq FRQ
Z       Z
mac     AC
I assigned the `mac column (minor allele count) to AC which is probably not ideal.  A lot of our users and TOPMed researchers use GENESIS so it would be useful to have a -c genesis option.

I ran metal on the 4 summary files (3 with SAIGE output and 1 with this GENESIS output). With just the 3 SAIGE results, I got this output for the APOE variant:
chr19   44908684        rs429358        T       C       .       .       .       NS:NC:ES:SE:LP:AF:AC:NE:I2:CQ:ED        725785:81165:1.04249:0.0155521:977.642:0.176641:223431:268674:98.6529:32.2391:+++
When I added the GENESIS summary data, I ended up with 3 lines for some reason.
chr19   44908684        rs429358;chr19:44908684:T:C     T       C       .       .       .       NS:ES:SE:LP:AF:AC:I2:CQ:ED      729318:1.03617:0.0150746:1027.89:0.178736:224924:98.0159:31.8383:++++
chr19   44908684        chr19:44908684:T:C      T       C       .       .       .       NS:ES:SE:LP:AF:AC:I2:CQ:ED      3533:0.937911:0.0613045:52.1114:0.211294:1493:.:.:???+
chr19   44908684        chr19:44908684:T:C      T       C       .       .       .       NS:ES:SE:LP:AF:AC:I2:CQ:ED      3533:0.937911:0.0613045:52.1114:0.211294:1493:.:.:???+
Any ideas or suggestions on this? The first line looks correct, not sure why the other 2 lines are in there.

@freeseek I will try go create some smaller gwas_vcfs  to try to reproduce the issue with the APOE variant and get back to you. It may have to do with the number of threads being used which I will test out.
For the GENESIS bindings:
https://github.com/UW-GAC/GENESIS/blob/devel/man/assocTestSingle.Rd
\item{variant.id}{The variant ID}
\item{chr}{The chromosome value}
\item{pos}{The base pair position}
\item{allele.index}{The index of the alternate allele. For biallelic variants, this will always be 1.}
\item{n.obs}{The number of samples with non-missing genotypes}
\item{freq}{The estimated effect allele frequency}
\item{MAC}{The minor allele count. For multiallelic variants, "minor" is determined by comparing the count of the allele specified by \code{allele.index} with the sum of all other alleles.}
If \code{geno.coding} is \code{"recessive"}:
\item{n.hom.eff}{The number of samples homozygous for the effect allele.}
If \code{geno.coding} is \code{"dominant"}:
\item{n.any.eff}{The number of samples with any copies of the effect allele.}
If \code{test} is \code{"Score"}:
\item{Score}{The value of the score function}
\item{Score.SE}{The estimated standard error of the Score}
\item{Score.Stat}{The score Z test statistic}
\item{Score.pval}{The score p-value}
\item{Est}{An approximation of the effect size estimate for each additional copy of the effect allele}
\item{Est.SE}{An approximation of the standard error of the effect size estimate}
\item{PVE}{An approximation of the proportion of phenotype variance explained}
% If \code{test} is \code{"Wald"} and \code{GxE} is \code{NULL}:
% \item{Est}{The effect size estimate for each additional copy of the effect allele}
% \item{Est.SE}{The estimated standard error of the effect size estimate}
% \item{Wald.Stat}{The Wald Z test statistic}
% \item{Wald.pval}{The Wald p-value}
If \code{test} is \code{"Score.SPA"}:
\item{SPA.pval}{The score p-value after applying the saddle point approximation (SPA)}
\item{SPA.converged}{logical indiactor of whether the SPA converged; \code{NA} indicates that the SPA was not applied and the original Score.pval was returned}
If \code{GxE} is not \code{NULL}:
\item{Est.G}{The effect size estimate for the genotype term}
\item{Est.G:env}{The effect size estimate for the genotype*env interaction term. There will be as many of these terms as there are interaction variables, and "env" will be replaced with the variable name.}
\item{SE.G}{The estimated standard error of the genotype term effect size estimate}
\item{SE.G:env}{The estimated standard error of the genotype*env effect size estimate. There will be as many of these terms as there are interaction variables, and "env" will be replaced with the variable name.}
\item{GxE.Stat}{The Wald Z test statistic for the test of all genotype interaction terms.  When there is only one genotype interaction term, this is the test statistic for that term.}
\item{GxE.pval}{The Wald p-value for the test of all genotype interaction terms; i.e. the test of any genotype interaction effect}
\item{Joint.Stat}{The Wald Z test statistic for the joint test of the genotype term and all of the genotype interaction terms}
\item{Joint.pval}{The Wald p-value for the joint test of the genotype term and all of the genotype interaction terms; i.e. the test of any genotype effect}
If \code{test} is \code{"BinomiRare" or "CMP"}:
\item{n.carrier}{Number of individuals with at least one copy of the effect allele}
\item{n.D.carrier}{Number of cases with at least one copy of the effect allele}
\item{pval}{p-value}
\item{mid.pval}{mid-p-value}

I don't see rsid, alt, ref, eff.frq, or Z among the GENESIS bindings. Maybe it would be simply better to add some of the bindings you are working with in colheaders.tsv rather than add a -c option for a non-conventional summary statistics format